Over the past week I’ve posted a couple of articles that should help people new to vSphere with Tanzu. There are some common hurdles that can pop up. I’ve now deployed it in a couple of environments and each one has raised new quirks. There also isn’t a lot of straight forward content out there on how to quickly fix issues that come up after the deployment of a Tanzu Kubernetes Grid instance into a fresh namespace. There are kubernetes security and kubernetes storage volumes to deal with and for those not familiar with or new to Kubernetes… it can drive you mad!
The Problem:
In the new environment, I deviated a little from the Nested Homelab setup. I deployed Tanzu with different underlying storage conduct’s that where based on William Lam’s auto-deploy scripts. That deployment creates most of the vSphere pre-requisites for you including the TKG vSphere Content Library. The script creates and attaches the vsanDatastore to the new content library which can then be used during the Workload Management creation wizard.
About Subscribed Content Library for Tanzu Kubernetes Clusters
The vSphere administrator configures a Subscribed Content Library on the Supervisor Cluster. The virtual machine image that is used for the Tanzu Kubernetes cluster nodes is pulled from this library. A Subscribed Content Library originates from a Published Content Library. After the subscription is created, the system synchronizes it with the published library. To create the Tanzu Kubernetes cluster nodes, VMware publishes a Photon OS OVA library to which you subscribe. After the subscriber is synchronized with the publisher, you associate the content library with the Supervisor Cluster.
Though this is a standard way to work with the TKG content library there are situations were you want to introduce different classes of underlying storage on which to deploy Supervisor and TKH Guest components onto. In my case, I created a new SSD backed NFS share and created a new Storage Policy which I used for the Supervisor cluster. The policy didn’t include the vsanDatastore where the TKG Content Catalog was.
The Error:
When looking to create the initial TKG Guest Cluster in the new namespace, I came across this error:
|
1 2 |
# kubectl.exe apply -f .\cluster.yaml Error from server (unable to find Kubernetes distributions): error when creating ".\\cluster.yaml": admission webhook "version.mutating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: unable to find Kubernetes distributions |
Unable to find Kubernetes distributions was a pretty clear error knowing where it pulls them for the deployment. With that I checked to see what Virtual Machine Images where listed against the namespace.
As you can see above, I tried the -A flag to see if they where hiding, and I also switched contexts up a level without any luck.
The Fix:
I gathered right away that this must have something to do with the fact that the Storage Policy attached to the Namespace didn’t contain the TKG Content Library storage. With that, I tried to add the Storage Policy containing the vsanDatastore to the namespace.
Which was reflected in the namespace when calling the kubectl get storagesclass command
Even with this now in place, I still couldn’t get a list of the Virtual Machine Images, so I decided to force the issue and delete and re-create the namespace. This time I added the Storage Policy containing the TKG Content Library . This resulted in the images being available in the namespace.
I don’t know at this stage if I had waited for a while longer if the namespace inventory would have updated under its own steam. Given that this was an empty/new namespace there was no issue blowing it away and starting from scratch. If this doesn’t work you can look that edit the Content Library directly under the Tanzu Kubernetes box under the Namespace. Be careful with this one though as the warning does suggest this will impact existing namespaces.
References:
https://core.vmware.com/resource/vsphere-tanzu-quick-start-guide
pod s kubernetes deployment kubernetes persistent volumes security policy tanzu no virtual machine images
