In this article I will demonstrate how you can install Container Service Extension 3.1.1 (CSE 3.1.1) with VMware Cloud Director 10.3.1.
CSE is a VMware Cloud Director extension that helps tenants create and work with Kubernetes clusters.
CSE brings Kubernetes as a Service to VCD, by allowing providers to:
- Create customized VM templates (Kubernetes templates)
- Import standard VMware Tanzu Kubernetes Grid OVA (TKG templates)
And enabling tenant users to deploy fully functional Kubernetes clusters as self-contained vApps.
Official VMware Documentation can be found here:
CSE 3.1.1 – What’s New?
- Support for importing VMware Tanzu Kubernetes Grid OVAs and deploying Kubernetes clusters.
- Kubernetes External Cloud Provider for VCD. Learn more about CPI for VCD
- Kubernetes Container Storage Interface for VCD. Learn more about CSI for VCD
- Kubernetes Container Clusters plugin is updated to version 3.1.0, which includes support for Tanzu Kubernetes Grid. The plugin ships with VCD 10.3.1.
- Deploy externally accessible TKG clusters on NSX-T based Routed OrgVDC Networks from Kubernetes Container Clusters UI plugin v3.1.0.
- Deprecation of Photon OS 2.0 based native templates, they will be removed in a future CSE release.
So much for official documentation, let’s get started!
Deploy CSE VM
Deploy a new Photon VM using this Photon .OVA.
Do not use Photon OS 4, as CSE requires Python 3.7.x
I gave my VM 2vCPU, 4 GB of RAM and deployed it in the same network as my vCenter Server, NSX Manager and Cloud Director cells.
Deployment will take a minute or two
Configure CSE VM
After deployment, connect to the VM Console and log on for the first time
Update the hostname to the name of your choice
# Update host name on Photon OS vi /etc/hostname # Check host name on Photon OS cat /etc/hostname
Configure a static IP on the CSE Server with the following command:
cat > /etc/systemd/network/10-static-en.network << "EOF" [Match] Name=eth0 [Network] Address=192.168.200.107/24 Gateway=192.168.200.254 DNS=192.168.200.101 192.168.200.102 Domains=yourdomain.local EOF
Update the file’s rights and restart the services
chmod 644 /etc/systemd/network/10-static-en.network systemctl restart systemd-networkd systemdctl restart systemd-resolved ping vblog.nl
Update Photon OS
cd /etc/yum.repos.d/ sed -i 's/dl.bintray.com\/vmware/packages.vmware.com\/photon\/$releasever/g' photon.repo photon-updates.repo photon-extras.repo photon-debuginfo.repo # Update Photon OS tdnf --assumeyes update # Install CSE dependencies tdnf --assumeyes install build-essential python3-devel python3-pip git # Create a new CSE user and application directories mkdir -p /opt/vmware/cse chmod 775 -R /opt chmod 777 / groupadd cse useradd cse -g cse -m -p PASSWORD -d /opt/vmware/cse chown cse:cse -R /opt
# Run as CSE user su - cse # Add your public SSH key to CSE server mkdir -p ~/.ssh cat >> ~/.ssh/authorized_keys << EOF ssh-rsa AAAAB:q3NzaC1yc2EAAAADAQABAAABAQCnfTDN/JM3JPBeh6JkS/AeBRXWmSk3Q8NajE4gVWQMZTW9wQGLwjNP6jmuw62qm1KAyyPVlZFSm9iaefTwA9UGd+SlWDi76yH48oGLQAtXUvE0CseMHOUNideHLv/wDMNKCuGk8enngph2JiqOGKexgdec1EpGR7BBQ3Q0kX1ZasP1vzKA38vUtMkE+r/6XO99eAMU2oU8gdNHB6kkU8rgceQPdETyqQbAs3ZhxqT2wf+FEkv0QsrtL9l+GHETMsfx5GSv2FKhlewmuD6WdTXijBZffn2y9aImwFJ+7i2A+jG2KvYiQRwxykB6IAAdUOyHbZEMvgyUD+f+DKHn9j29 rsa-key-20211101 EOF cat >> ~/.bash_profile << EOF # For Container Service Extension export CSE_CONFIG=/opt/vmware/cse/config/config.yaml export CSE_CONFIG_PASSWORD=PASSWORD source /opt/vmware/cse/python/bin/activate EOF
Install the Container Service Extension
To install the CSE software in the Virtual Environment, please follow the commands below.
# Install CSE in the virtual environment python3 -m venv /opt/vmware/cse/python source /opt/vmware/cse/python/bin/activate pip3 install container-service-extension==3.1.1
Associate the CSE extension to VCD-CLI
# Prepare vcd-cli cat > ~/.vcd-cli/profiles.yaml << EOF extensions: - container_service_extension.client.cse EOF # Verify if the extension is configured correctly vcd cse version
# Use the following VCD-CLI command to log on to your VMware Cloud Director environment vcd login tkg.vblog.nl system administrator -p PASSWORD # Create a new CSE Service Role on your VMware Cloud Director environment cse create-service-role tkg.vblog.nl # Create a new CSE Service account on your VMware Cloud Director environment vcd user create --enabled _svc_cse PASSWORD "CSE Service Role"
Prepare VMware Cloud Director
Before we can proceed with the installation of the Container Service Extension, we need to prepare VMware Cloud Director first.
The following must be in place:
- Organization VDC
- VCD Catalog
- Edge device with internet access (mine is an NSX-V Edge)
- Firewall & NAT rules for VM traffic
- VM Network with an IP pool or a DHCP server
- DNS must work in this environment!
To satisfy these requirements, I have created a new Organization “TKGM”.
With an OrgVDC “OrgVDC-TKGM-01”.
Which include a new catalog “TKG-Catalog”.
And a new NSX Edge “Edge-TKGM-01”.
On the NSX Edge you can configure grouping objects for your subnets, and allow them internet access as per the following example.
Make sure you have a NAT rule configured for your subnet.
Have a VM network created.
With an IP pool or a DHCP server configured.
Make sure DNS works in your VM network environment!
In my POC I’ve configured an Active Directory server with DNS services configured.
Now, back to the CSE server!
In this file, you need to define your Cloud Director instance, vCenter Server, Storage Policy, VCD Organization, and more.
Run the following commands to setup your config.yaml file.
# Create config file mkdir -p /opt/vmware/cse/config cat > /opt/vmware/cse/config/config-not-encrypted.conf << EOF mqtt: verify_ssl: false vcd: host: tkg.vblog.nl log: true password: PASSWORD port: 443 username: administrator verify: true vcs: - name: vcsa01.tkginfra.local password: PASSWORD username: email@example.com verify: true service: enforce_authorization: false legacy_mode: false log_wire: false no_vc_communication_mode: false processors: 15 telemetry: enable: true broker: catalog: TKG-Catalog ip_allocation_mode: pool network: Network-02 org: TKGM remote_template_cookbook_url: https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml storage_profile: 'vSAN Default Storage Policy' vdc: OrgVDC-TKGM-01 EOF
Passwords are stored in plain-text in the config.yaml file, which is not very secure.
Encrypt the file with the following commands
cse encrypt /opt/vmware/cse/config/config-not-encrypted.conf --output /opt/vmware/cse/config/config.yaml chmod 600 /opt/vmware/cse/config/config.yaml cse check /opt/vmware/cse/config/config.yaml
When the config.yaml file is ‘valid’, you can request the available templates with the following command:
cse template list
Now, we’re ready to install CSE
# Install CSE cse install -c /opt/vmware/cse/config/config.yaml # This process can take up to 3 hours, depending on your internet connectivity and storage performance. During the installation process, CSE templates are being downloaded and customzied.
As mentioned earlier, the update process can take up to 3 hours. If you desire to babysit the process, it will look like the following screenshot .
When you see “Waiting for guest tools status: vm=vim.VirtualMachine:vm-144′, status=GuestToolsNotRunning” a few times, don’t get worried. In time the status changes to “Waiting for guest tools status: vm=vim.VirtualMachine:vm-144′, status=GuestToolsRunning”
The installation of CSE server finished succesfully and all the templates have been downloaded and customized.
The templates can be found in the VCD Library as a vApp Template
Add CSE extension to VMware Cloud Director
Next step to do now is to register the extension in VMware Cloud Director by running the following commands:
# Log on to VMware Cloud Director using the VCD-CLI vcd login tkg.vblog.nl system administrator -p PASSWORD # Registreer de extension vcd system extension create cse cse cse vcdext '/api/cse, /api/cse/.*, /api/cse/.*/.*'
In the Cloud Director UI, onder More > Customize Portal you can see the Container UI Plugin with status “Enabled”.
# Setup cse.sh cat > /opt/vmware/cse/cse.sh << EOF #!/usr/bin/env bash source /opt/vmware/cse/python/bin/activate export CSE_CONFIG=/opt/vmware/cse/config/config.yaml export CSE_CONFIG_PASSWORD=PASSWORD cse run EOF # Make cse.sh executable chmod +x /opt/vmware/cse/cse.sh # Deactivate the python virtual environment and go back to root deactivate exit # Setup cse.service, use MQTT and not RabbitMQ cat > /etc/systemd/system/cse.service << EOF [Unit] Description=Container Service Extension for VMware Cloud Director [Service] ExecStart=/opt/vmware/cse/cse.sh User=cse WorkingDirectory=/opt/vmware/cse Type=simple Restart=always [Install] WantedBy=default.target EOF systemctl enable cse.service systemctl start cse.service systemctl status cse.service
In VMware Cloud Director navigate to Administration > Rights Bundles > Publish the following rights bundle to the tenants you setup for CSE:
- cse:nativeCluster Entitlement
- cse:tkgCluster Entitlement
In VMware Cloud Director navigate to Administration > Global Roles and edit the Global Role for Organization Administrator.
In my POC environment, I’ve scrolled down to “Other”, in the bottom and added all rights listed there:
Now, enable Native Kubernetes for a specific OrgVDC by running the following commands:
# Log on to VMware Cloud Director using the VCD-CLI vcd login tkg.vblog.nl system administrator -p PASSWORD # Enable Native Kubernetes for a specific OrgVDC vcd cse ovdc enable --native --org TKGM OrgVDC-TKGM-01
Add more recente TKG templates to your environment by running the following commands:
- Download .OVA template van my.vmware.com > ubuntu-2004-kube-v1.21.2+vmware.1-tkg.1-7832907791984498322.ova
- Upload to the /tmp folder of your vcd-cse01 server with WinSCP.
- Update the rights to 0644
cse template import -c /opt/vmware/cse/config/config.yaml -F /tmp/ubuntu-2004-kube-v1.21.2+vmware.1-tkg.1-7832907791984498322.ova
On the CLI, it will look like this: