Skip to content

vBlog.nl

All about technology

Menu
  • Home
  • VMware Cloud Director
  • vSphere
  • Automation
    • PowerCLI
    • PowerShell
    • Terraform
  • NSX
  • Horizon DaaS
  • About Us
Menu

Install CSE 3.1.1 with Cloud Director 10.3.1

Posted on January 7, 2022

In this article I will demonstrate how you can install Container Service Extension 3.1.1 (CSE 3.1.1) with VMware Cloud Director 10.3.1.

CSE is a VMware Cloud Director extension that helps tenants create and work with Kubernetes clusters.

CSE brings Kubernetes as a Service to VCD, by allowing providers to:

  • Create customized VM templates (Kubernetes templates)
  • Import standard VMware Tanzu Kubernetes Grid OVA (TKG templates)

And enabling tenant users to deploy fully functional Kubernetes clusters as self-contained vApps.

Table of Contents

  • Documentation
    • CSE 3.1.1 – What’s New?
  • Deploy CSE VM
  • Configure CSE VM
  • Update Photon OS
  • Install the Container Service Extension
  • Prepare VMware Cloud Director
  • CSE config.yaml
  • Add CSE extension to VMware Cloud Director

Documentation

Official VMware Documentation can be found here:
https://vmware.github.io/container-service-extension/cse3_1/CSE31.html

CSE 3.1.1 – What’s New?

  • Support for importing VMware Tanzu Kubernetes Grid OVAs and deploying Kubernetes clusters.
    • Learn more about using VMware Tanzu Kubernetes Grid OVAs with CSE
    • Learn more about deploying a Kubernetes cluster based on VMware Tanzu Kubernetes Grid here
    • Antrea as CNI
  • Kubernetes External Cloud Provider for VCD. Learn more about CPI for VCD
  • Kubernetes Container Storage Interface for VCD. Learn more about CSI for VCD
  • Kubernetes Container Clusters plugin is updated to version 3.1.0, which includes support for Tanzu Kubernetes Grid. The plugin ships with VCD 10.3.1.
  • Deploy externally accessible TKG clusters on NSX-T based Routed OrgVDC Networks from Kubernetes Container Clusters UI plugin v3.1.0.
  • Deprecation of Photon OS 2.0 based native templates, they will be removed in a future CSE release.


So much for official documentation, let’s get started!

Deploy CSE VM

Deploy a new Photon VM using this Photon .OVA.
Do not use Photon OS 4, as CSE requires Python 3.7.x

I gave my VM 2vCPU, 4 GB of RAM and deployed it in the same network as my vCenter Server, NSX Manager and Cloud Director cells.

Deployment will take a minute or two

Configure CSE VM

After deployment, connect to the VM Console and log on for the first time

  • root
  • changeme

Update the hostname to the name of your choice

# Update host name on Photon OS
vi /etc/hostname
# Check host name on Photon OS
cat /etc/hostname

Configure a static IP on the CSE Server with the following command:

cat > /etc/systemd/network/10-static-en.network << "EOF"
 
[Match]
Name=eth0
 
[Network]
Address=192.168.200.107/24
Gateway=192.168.200.254
DNS=192.168.200.101 192.168.200.102
Domains=yourdomain.local
EOF

Update the file’s rights and restart the services

chmod 644 /etc/systemd/network/10-static-en.network
systemctl restart systemd-networkd
systemdctl restart systemd-resolved
ping vblog.nl

Update Photon OS

cd /etc/yum.repos.d/
sed  -i 's/dl.bintray.com\/vmware/packages.vmware.com\/photon\/$releasever/g' photon.repo photon-updates.repo photon-extras.repo photon-debuginfo.repo
 
# Update Photon OS
tdnf --assumeyes update
 
# Install CSE dependencies
tdnf --assumeyes install build-essential python3-devel python3-pip git
 
# Create a new CSE user and application directories
mkdir -p /opt/vmware/cse
chmod 775 -R /opt
chmod 777 /
groupadd cse
useradd cse -g cse -m -p PASSWORD -d /opt/vmware/cse
chown cse:cse -R /opt
# Run as CSE user
su - cse
 
# Add your public SSH key to CSE server
mkdir -p ~/.ssh
cat >> ~/.ssh/authorized_keys << EOF
ssh-rsa AAAAB:q3NzaC1yc2EAAAADAQABAAABAQCnfTDN/JM3JPBeh6JkS/AeBRXWmSk3Q8NajE4gVWQMZTW9wQGLwjNP6jmuw62qm1KAyyPVlZFSm9iaefTwA9UGd+SlWDi76yH48oGLQAtXUvE0CseMHOUNideHLv/wDMNKCuGk8enngph2JiqOGKexgdec1EpGR7BBQ3Q0kX1ZasP1vzKA38vUtMkE+r/6XO99eAMU2oU8gdNHB6kkU8rgceQPdETyqQbAs3ZhxqT2wf+FEkv0QsrtL9l+GHETMsfx5GSv2FKhlewmuD6WdTXijBZffn2y9aImwFJ+7i2A+jG2KvYiQRwxykB6IAAdUOyHbZEMvgyUD+f+DKHn9j29 rsa-key-20211101
EOF
 
cat >> ~/.bash_profile << EOF
# For Container Service Extension
export CSE_CONFIG=/opt/vmware/cse/config/config.yaml
export CSE_CONFIG_PASSWORD=PASSWORD
source /opt/vmware/cse/python/bin/activate
EOF

Install the Container Service Extension

To install the CSE software in the Virtual Environment, please follow the commands below.

# Install CSE in the virtual environment
python3 -m venv /opt/vmware/cse/python
source /opt/vmware/cse/python/bin/activate
pip3 install container-service-extension==3.1.1

Associate the CSE extension to VCD-CLI

# Prepare vcd-cli
cat >  ~/.vcd-cli/profiles.yaml << EOF
extensions:
- container_service_extension.client.cse
EOF
 
# Verify if the extension is configured correctly
vcd cse version
# Use the following VCD-CLI command to log on to your VMware Cloud Director environment
vcd login tkg.vblog.nl system administrator -p PASSWORD
 
# Create a new CSE Service Role on your VMware Cloud Director environment
cse create-service-role tkg.vblog.nl
 
# Create a new CSE Service account on your VMware Cloud Director environment
vcd user create --enabled _svc_cse PASSWORD "CSE Service Role"

Prepare VMware Cloud Director

Before we can proceed with the installation of the Container Service Extension, we need to prepare VMware Cloud Director first.

The following must be in place:

  • Organization 
  • Organization VDC
  • VCD Catalog
  • Edge device with internet access (mine is an NSX-V Edge)
  • Firewall & NAT rules for VM traffic
  • VM Network with an IP pool or a DHCP server
  • DNS must work in this environment!

To satisfy these requirements, I have created a new Organization “TKGM”.

With an OrgVDC “OrgVDC-TKGM-01”.

Which include a new catalog “TKG-Catalog”.

And a new NSX Edge “Edge-TKGM-01”.

On the NSX Edge you can configure grouping objects for your subnets, and allow them internet access as per the following example.

Make sure you have a NAT rule configured for your subnet.

Have a VM network created.

With an IP pool or a DHCP server configured.

Make sure DNS works in your VM network environment!

In my POC I’ve configured an Active Directory server with DNS services configured.

CSE config.yaml

Now, back to the CSE server!

Config.yaml
In this file, you need to define your Cloud Director instance, vCenter Server, Storage Policy, VCD Organization, and more.

Run the following commands to setup your config.yaml file.

# Create config file
mkdir -p /opt/vmware/cse/config
 
cat > /opt/vmware/cse/config/config-not-encrypted.conf << EOF
mqtt:
  verify_ssl: false
  
vcd:
  host: tkg.vblog.nl
  log: true
  password: PASSWORD
  port: 443
  username: administrator
  verify: true
  
vcs:
- name: vcsa01.tkginfra.local
  password: PASSWORD
  username: administrator@vsphere.local
  verify: true
  
service:
  enforce_authorization: false
  legacy_mode: false
  log_wire: false
  no_vc_communication_mode: false
  processors: 15
  telemetry:
    enable: true
  
broker:
  catalog: TKG-Catalog
  ip_allocation_mode: pool
  network: Network-02
  org: TKGM
  remote_template_cookbook_url: https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml
  storage_profile: 'vSAN Default Storage Policy'
  vdc: OrgVDC-TKGM-01
EOF

Passwords are stored in plain-text in the config.yaml file, which is not very secure.
Encrypt the file with the following commands

cse encrypt /opt/vmware/cse/config/config-not-encrypted.conf --output /opt/vmware/cse/config/config.yaml
chmod 600 /opt/vmware/cse/config/config.yaml
cse check /opt/vmware/cse/config/config.yaml

When the config.yaml file is ‘valid’, you can request the available templates with the following command:

cse template list

Now, we’re ready to install CSE

# Install CSE
cse install -c /opt/vmware/cse/config/config.yaml
 
# This process can take up to 3 hours, depending on your internet connectivity and storage performance.
During the installation process, CSE templates are being downloaded and customzied.

As mentioned earlier, the update process can take up to 3 hours. If you desire to babysit the process, it will look like the following screenshot .
When you see “Waiting for guest tools status: vm=vim.VirtualMachine:vm-144′, status=GuestToolsNotRunning” a few times, don’t get worried. In time the status changes to “Waiting for guest tools status: vm=vim.VirtualMachine:vm-144′, status=GuestToolsRunning”

The installation of CSE server finished succesfully and all the templates have been downloaded and customized.

The templates can be found in the VCD Library as a vApp Template

Add CSE extension to VMware Cloud Director

Next step to do now is to register the extension in VMware Cloud Director by running the following commands:

# Log on to VMware Cloud Director using the VCD-CLI
vcd login tkg.vblog.nl system administrator -p PASSWORD
 
# Registreer de extension
vcd system extension create cse cse cse vcdext '/api/cse, /api/cse/.*, /api/cse/.*/.*'

In the Cloud Director UI, onder More > Customize Portal you can see the Container UI Plugin with status “Enabled”.

Configure CSE

# Setup cse.sh
cat > /opt/vmware/cse/cse.sh << EOF
#!/usr/bin/env bash
source /opt/vmware/cse/python/bin/activate
export CSE_CONFIG=/opt/vmware/cse/config/config.yaml
export CSE_CONFIG_PASSWORD=PASSWORD
cse run
EOF
  
# Make cse.sh executable
chmod +x /opt/vmware/cse/cse.sh
  
# Deactivate the python virtual environment and go back to root
deactivate
exit
  
# Setup cse.service, use MQTT and not RabbitMQ
cat > /etc/systemd/system/cse.service << EOF
[Unit]
Description=Container Service Extension for VMware Cloud Director
  
[Service]
ExecStart=/opt/vmware/cse/cse.sh
User=cse
WorkingDirectory=/opt/vmware/cse
Type=simple
Restart=always
  
[Install]
WantedBy=default.target
EOF
  
systemctl enable cse.service
systemctl start cse.service
  
systemctl status cse.service

In VMware Cloud Director navigate to Administration > Rights Bundles > Publish the following rights bundle to the tenants you setup for CSE:

  • cse:nativeCluster Entitlement
  • cse:tkgCluster Entitlement

In VMware Cloud Director navigate to Administration > Global Roles and edit the Global Role for Organization Administrator.

In my POC environment, I’ve scrolled down to “Other”, in the bottom and added all rights listed there:

Now, enable Native Kubernetes for a specific OrgVDC by running the following commands:

# Log on to VMware Cloud Director using the VCD-CLI
vcd login tkg.vblog.nl system administrator -p PASSWORD
# Enable Native Kubernetes for a specific OrgVDC
vcd cse ovdc enable --native --org TKGM OrgVDC-TKGM-01

Add more recente TKG templates to your environment by running the following commands:

  • Download .OVA template van my.vmware.com > ubuntu-2004-kube-v1.21.2+vmware.1-tkg.1-7832907791984498322.ova
  • Upload to the /tmp folder of your vcd-cse01 server with WinSCP.
  • Update the rights to 0644
cse template import -c /opt/vmware/cse/config/config.yaml -F /tmp/ubuntu-2004-kube-v1.21.2+vmware.1-tkg.1-7832907791984498322.ova

On the CLI, it will look like this:

 2,520 total views,  1 views today

Share on Social Media
twitter facebook linkedin reddit emailwhatsapptelegram

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent articles

  • VMware Cloud on AWS with FSx for NetApp ONTAP December 28, 2022
  • Workspace ONE Access – Change certificate December 22, 2022
  • UI themes for VMware Cloud Director 10.4.1 and later December 16, 2022
  • VMware Tanzu Kubernetes Grid December 15, 2022
  • Workspace One Access – Not logged in to server FQDN. Please invoke Save before Sync December 12, 2022
  • NSX Edge configuration has failed. 1G hugepage support required. December 7, 2022
  • Horizon DaaS – Unable to connect to Desktop November 24, 2022
  • How to: Request and Install a Lets Encrypt Wildcard SSL on VMware Cloud Director 10.4 October 17, 2022
  • New and improved vSAN 8 explained September 24, 2022
  • How to update a stand alone ESXi host September 13, 2022

Tags

Automation bootstrapping Container Service Extension Credential Manager Desktone.log ESXi EUC EXi GPU Horizon DaaS Instant-Clone Logging NSX nsx-t PowerCLI PowerShell Putty Raspberry PI SSL Tanzu Terraform update vcd-cli vCenter vCloud vCloud Availability vCloud Director VDI vGPU VMware VMworld vSAN vSphere vVols workspace one Zerto

VMware Cloud Provider Blog

  • NSX ALB Licensing with VMware Cloud Director
    by Jaikishan Tayal on March 27, 2023 at 10:33 am

    History of NSX ALB and VCD Licensing: From version 10.2, VMware Cloud Director began supporting NSX-T Advanced Load Balancer (AVI Load Balancer). This integration has become crucial for Cloud Services Providers, allowing them to provide their customers with a range of Load Balancing services (LBaaS) for Virtual Data Center workloads and Tanzu containers. Before the … Continued The post NSX ALB Licensing with VMware Cloud Director appeared first on VMware Cloud Provider Blog.

  • VMware vSAN 8 Update 1 for Cloud Services Providers
    by Christopher Wong on March 24, 2023 at 3:00 pm

    Recently, VMware announced the upcoming release of vSAN 8 Update 1. This latest update enhances vSAN’s capabilities and functionality with additional improvements for performance, data durability, and integration. Cloud Services Providers who are leveraging the capabilities of vSAN 8 can expect to see additional benefits in these areas as they deploy this upcoming release into … Continued The post VMware vSAN 8 Update 1 for Cloud Services Providers appeared first on VMware Cloud Provider Blog.

  • Terraform VMware Cloud Director Provider 3.9.0 – Beta early access build
    by Guy Bartram on March 24, 2023 at 11:06 am

    The release time for version 3.9.0 of Terraform VMware Cloud Director Provider is approaching,and for the first time, we’re releasing a beta, so users can try the new features and give helpful feedback that will improve the final release. What is a “beta” build? A beta build is a preliminary release of an intended new … Continued The post Terraform VMware Cloud Director Provider 3.9.0 – Beta early access build appeared first on VMware Cloud Provider Blog.

  • VMware Cloud Director Object Storage Extension 2.2.1
    by Astha Sharma on March 16, 2023 at 5:45 pm

    Object Storage Extension 2.1.1 The post VMware Cloud Director Object Storage Extension 2.2.1 appeared first on VMware Cloud Provider Blog.

  • Architecting VMware Cloud Director Availability Solution in a Multi-Cloud Environment
    by Nikolay Patrikov on March 15, 2023 at 1:09 pm

    Building a cloud based on VMware Cloud Director or Cloud Director service requires a considerable amount of deployment decisions for Cloud Providers concerning the infrastructure behind their services. They can operate fully on-premises within their data centers or combine them with any of the hyperscalers in a hybrid way. These design decisions affect the DRaaS … Continued The post Architecting VMware Cloud Director Availability Solution in a Multi-Cloud Environment appeared first on VMware Cloud Provider Blog.

©2023 vBlog.nl | Design: Newspaperly WordPress Theme