Install CSE 3.1.1 with Cloud Director 10.3.1

Posted on January 7, 2022

In this article I will demonstrate how you can install Container Service Extension 3.1.1 (CSE 3.1.1) with VMware Cloud Director 10.3.1.

CSE is a VMware Cloud Director extension that helps tenants create and work with Kubernetes clusters.

CSE brings Kubernetes as a Service to VCD, by allowing providers to:

  • Create customized VM templates (Kubernetes templates)
  • Import standard VMware Tanzu Kubernetes Grid OVA (TKG templates)

And enabling tenant users to deploy fully functional Kubernetes clusters as self-contained vApps.

Documentation

Official VMware Documentation can be found here:
https://vmware.github.io/container-service-extension/cse3_1/CSE31.html

CSE 3.1.1 – What’s New?

  • Support for importing VMware Tanzu Kubernetes Grid OVAs and deploying Kubernetes clusters.
  • Kubernetes External Cloud Provider for VCD. Learn more about CPI for VCD
  • Kubernetes Container Storage Interface for VCD. Learn more about CSI for VCD
  • Kubernetes Container Clusters plugin is updated to version 3.1.0, which includes support for Tanzu Kubernetes Grid. The plugin ships with VCD 10.3.1.
  • Deploy externally accessible TKG clusters on NSX-T based Routed OrgVDC Networks from Kubernetes Container Clusters UI plugin v3.1.0.
  • Deprecation of Photon OS 2.0 based native templates, they will be removed in a future CSE release.


So much for official documentation, let’s get started!

Deploy CSE VM

Deploy a new Photon VM using this Photon .OVA.
Do not use Photon OS 4, as CSE requires Python 3.7.x

I gave my VM 2vCPU, 4 GB of RAM and deployed it in the same network as my vCenter Server, NSX Manager and Cloud Director cells.

Deployment will take a minute or two

Configure CSE VM

After deployment, connect to the VM Console and log on for the first time

  • root
  • changeme

Update the hostname to the name of your choice

# Update host name on Photon OS
vi /etc/hostname
# Check host name on Photon OS
cat /etc/hostname

Configure a static IP on the CSE Server with the following command:

cat > /etc/systemd/network/10-static-en.network << "EOF"
 
[Match]
Name=eth0
 
[Network]
Address=192.168.200.107/24
Gateway=192.168.200.254
DNS=192.168.200.101 192.168.200.102
Domains=yourdomain.local
EOF

Update the file’s rights and restart the services

chmod 644 /etc/systemd/network/10-static-en.network
systemctl restart systemd-networkd
systemdctl restart systemd-resolved
ping vblog.nl

Update Photon OS

cd /etc/yum.repos.d/
sed  -i 's/dl.bintray.com\/vmware/packages.vmware.com\/photon\/$releasever/g' photon.repo photon-updates.repo photon-extras.repo photon-debuginfo.repo
 
# Update Photon OS
tdnf --assumeyes update
 
# Install CSE dependencies
tdnf --assumeyes install build-essential python3-devel python3-pip git
 
# Create a new CSE user and application directories
mkdir -p /opt/vmware/cse
chmod 775 -R /opt
chmod 777 /
groupadd cse
useradd cse -g cse -m -p PASSWORD -d /opt/vmware/cse
chown cse:cse -R /opt
# Run as CSE user
su - cse
 
# Add your public SSH key to CSE server
mkdir -p ~/.ssh
cat >> ~/.ssh/authorized_keys << EOF
ssh-rsa AAAAB:q3NzaC1yc2EAAAADAQABAAABAQCnfTDN/JM3JPBeh6JkS/AeBRXWmSk3Q8NajE4gVWQMZTW9wQGLwjNP6jmuw62qm1KAyyPVlZFSm9iaefTwA9UGd+SlWDi76yH48oGLQAtXUvE0CseMHOUNideHLv/wDMNKCuGk8enngph2JiqOGKexgdec1EpGR7BBQ3Q0kX1ZasP1vzKA38vUtMkE+r/6XO99eAMU2oU8gdNHB6kkU8rgceQPdETyqQbAs3ZhxqT2wf+FEkv0QsrtL9l+GHETMsfx5GSv2FKhlewmuD6WdTXijBZffn2y9aImwFJ+7i2A+jG2KvYiQRwxykB6IAAdUOyHbZEMvgyUD+f+DKHn9j29 rsa-key-20211101
EOF
 
cat >> ~/.bash_profile << EOF
# For Container Service Extension
export CSE_CONFIG=/opt/vmware/cse/config/config.yaml
export CSE_CONFIG_PASSWORD=PASSWORD
source /opt/vmware/cse/python/bin/activate
EOF

Install the Container Service Extension

To install the CSE software in the Virtual Environment, please follow the commands below.

# Install CSE in the virtual environment
python3 -m venv /opt/vmware/cse/python
source /opt/vmware/cse/python/bin/activate
pip3 install container-service-extension==3.1.1

Associate the CSE extension to VCD-CLI

# Prepare vcd-cli
cat >  ~/.vcd-cli/profiles.yaml << EOF
extensions:
- container_service_extension.client.cse
EOF
 
# Verify if the extension is configured correctly
vcd cse version

# Use the following VCD-CLI command to log on to your VMware Cloud Director environment
vcd login tkg.vblog.nl system administrator -p PASSWORD
 
# Create a new CSE Service Role on your VMware Cloud Director environment
cse create-service-role tkg.vblog.nl
 
# Create a new CSE Service account on your VMware Cloud Director environment
vcd user create --enabled _svc_cse PASSWORD "CSE Service Role"

Prepare VMware Cloud Director

Before we can proceed with the installation of the Container Service Extension, we need to prepare VMware Cloud Director first.

The following must be in place:

  • Organization 
  • Organization VDC
  • VCD Catalog
  • Edge device with internet access (mine is an NSX-V Edge)
  • Firewall & NAT rules for VM traffic
  • VM Network with an IP pool or a DHCP server
  • DNS must work in this environment!

To satisfy these requirements, I have created a new Organization “TKGM”.

With an OrgVDC “OrgVDC-TKGM-01”.

Which include a new catalog “TKG-Catalog”.

And a new NSX Edge “Edge-TKGM-01”.

On the NSX Edge you can configure grouping objects for your subnets, and allow them internet access as per the following example.

Make sure you have a NAT rule configured for your subnet.

Have a VM network created.

With an IP pool or a DHCP server configured.

Make sure DNS works in your VM network environment!

In my POC I’ve configured an Active Directory server with DNS services configured.

CSE config.yaml

Now, back to the CSE server!

Config.yaml
In this file, you need to define your Cloud Director instance, vCenter Server, Storage Policy, VCD Organization, and more.

Run the following commands to setup your config.yaml file.

# Create config file
mkdir -p /opt/vmware/cse/config
 
cat > /opt/vmware/cse/config/config-not-encrypted.conf << EOF
mqtt:
  verify_ssl: false
  
vcd:
  host: tkg.vblog.nl
  log: true
  password: PASSWORD
  port: 443
  username: administrator
  verify: true
  
vcs:
- name: vcsa01.tkginfra.local
  password: PASSWORD
  username: administrator@vsphere.local
  verify: true
  
service:
  enforce_authorization: false
  legacy_mode: false
  log_wire: false
  no_vc_communication_mode: false
  processors: 15
  telemetry:
    enable: true
  
broker:
  catalog: TKG-Catalog
  ip_allocation_mode: pool
  network: Network-02
  org: TKGM
  remote_template_cookbook_url: https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml
  storage_profile: 'vSAN Default Storage Policy'
  vdc: OrgVDC-TKGM-01
EOF

Passwords are stored in plain-text in the config.yaml file, which is not very secure.
Encrypt the file with the following commands

cse encrypt /opt/vmware/cse/config/config-not-encrypted.conf --output /opt/vmware/cse/config/config.yaml
chmod 600 /opt/vmware/cse/config/config.yaml
cse check /opt/vmware/cse/config/config.yaml

When the config.yaml file is ‘valid’, you can request the available templates with the following command:

cse template list

Now, we’re ready to install CSE

# Install CSE
cse install -c /opt/vmware/cse/config/config.yaml
 
# This process can take up to 3 hours, depending on your internet connectivity and storage performance.
During the installation process, CSE templates are being downloaded and customzied.

As mentioned earlier, the update process can take up to 3 hours. If you desire to babysit the process, it will look like the following screenshot .
When you see “Waiting for guest tools status: vm=vim.VirtualMachine:vm-144′, status=GuestToolsNotRunning” a few times, don’t get worried. In time the status changes to “Waiting for guest tools status: vm=vim.VirtualMachine:vm-144′, status=GuestToolsRunning”

The installation of CSE server finished succesfully and all the templates have been downloaded and customized.

The templates can be found in the VCD Library as a vApp Template

Add CSE extension to VMware Cloud Director

Next step to do now is to register the extension in VMware Cloud Director by running the following commands:

# Log on to VMware Cloud Director using the VCD-CLI
vcd login tkg.vblog.nl system administrator -p PASSWORD
 
# Registreer de extension
vcd system extension create cse cse cse vcdext '/api/cse, /api/cse/.*, /api/cse/.*/.*'

In the Cloud Director UI, onder More > Customize Portal you can see the Container UI Plugin with status “Enabled”.

Configure CSE

# Setup cse.sh
cat > /opt/vmware/cse/cse.sh << EOF
#!/usr/bin/env bash
source /opt/vmware/cse/python/bin/activate
export CSE_CONFIG=/opt/vmware/cse/config/config.yaml
export CSE_CONFIG_PASSWORD=PASSWORD
cse run
EOF
  
# Make cse.sh executable
chmod +x /opt/vmware/cse/cse.sh
  
# Deactivate the python virtual environment and go back to root
deactivate
exit
  
# Setup cse.service, use MQTT and not RabbitMQ
cat > /etc/systemd/system/cse.service << EOF
[Unit]
Description=Container Service Extension for VMware Cloud Director
  
[Service]
ExecStart=/opt/vmware/cse/cse.sh
User=cse
WorkingDirectory=/opt/vmware/cse
Type=simple
Restart=always
  
[Install]
WantedBy=default.target
EOF
  
systemctl enable cse.service
systemctl start cse.service
  
systemctl status cse.service

In VMware Cloud Director navigate to Administration > Rights Bundles > Publish the following rights bundle to the tenants you setup for CSE:

  • cse:nativeCluster Entitlement
  • cse:tkgCluster Entitlement

In VMware Cloud Director navigate to Administration > Global Roles and edit the Global Role for Organization Administrator.

In my POC environment, I’ve scrolled down to “Other”, in the bottom and added all rights listed there:

Now, enable Native Kubernetes for a specific OrgVDC by running the following commands:

# Log on to VMware Cloud Director using the VCD-CLI
vcd login tkg.vblog.nl system administrator -p PASSWORD
# Enable Native Kubernetes for a specific OrgVDC
vcd cse ovdc enable --native --org TKGM OrgVDC-TKGM-01

Add more recente TKG templates to your environment by running the following commands:

  • Download .OVA template van my.vmware.com > ubuntu-2004-kube-v1.21.2+vmware.1-tkg.1-7832907791984498322.ova
  • Upload to the /tmp folder of your vcd-cse01 server with WinSCP.
  • Update the rights to 0644
cse template import -c /opt/vmware/cse/config/config.yaml -F /tmp/ubuntu-2004-kube-v1.21.2+vmware.1-tkg.1-7832907791984498322.ova

On the CLI, it will look like this:

Loading

Leave a Reply

Your email address will not be published. Required fields are marked *