How to use kubectl CLI from Master and Worker VMs in PKS

When cluster operations fails, you need access to the cluster through kubectl CLI. However, in some instances, pks get-credentials errors out while fetching kubeconfig for the cluster.

An alternate way to resume troubleshooting is to SSH into the Master or Worker VMs of the cluster, see the kubectl binary, and see the kubeconfigs present on the VMs. This article details the procedure to setup the kubectl CLI. 

Execute the following to get access to kubectl CLI from the Kubernetes Master VM. 

bosh ssh -d service-instance_50a78f35-df99-4cf3-8815-7b680577ef59 master/0

master/01850df7-daaa-4b9c-a61f-e921074546ad:~$ sudo su -
master/01850df7-daaa-4b9c-a61f-e921074546ad:~# alias kubectl=/var/vcap/packages/kubernetes/bin/kubectl

master/01850df7-daaa-4b9c-a61f-e921074546ad:~# kubectl get pods -n pks-system

NAME                               READY     STATUS    RESTARTS   AGE
fluent-bit-mwbxd                   1/1       Running   0          1d
fluent-bit-vp66b                   1/1       Running   0          1d
sink-controller-57df674b84-wjp4k   1/1       Running   0          1d

Execute the following to get access to kubectl CLI from the Kubernetes Worker VM. 

bosh ssh -d service-instance_50a78f35-df99-4cf3-8815-7b680577ef59 worker/0

worker/a3b4ccb6-1d14-4ee4-a1f1-6d5211dfe4b9:~$ sudo su -
worker/a3b4ccb6-1d14-4ee4-a1f1-6d5211dfe4b9:~# alias kubectl="/var/vcap/packages/kubernetes/bin/kubectl --kubeconfig=/var/vcap/jobs/kubelet/config/kubeconfig"

worker/a3b4ccb6-1d14-4ee4-a1f1-6d5211dfe4b9:~# kubectl get pods -n pks-system
NAME                               READY     STATUS    RESTARTS   AGE
fluent-bit-mwbxd                   1/1       Running   0          1d
fluent-bit-vp66b                   1/1       Running   0          1d
sink-controller-57df674b84-wjp4k   1/1       Running   0          1d

Done

Changing the Auth Mode in Harbor from Internal Database to OIDC - options greyed out

Customer is using TKGi 1.9.2 with embbeded Harbor 2.1. Harbor has database auth mode by default.

Now customer wants to integrate the same harbor with OIDC. I was not able to change the auth mode in harbor URL. The drop-down button is disabled. 

Solution:

SSH into Harbor VM and Access PostgreSQL Database

~$ bosh -e pks vms
 Using environment '10.193.121.11' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)
 Task 186. Done
 :
 Deployment 'harbor-container-registry-74b44adfd44a90f27cb3'
 Instance                                         Process State AZ IPs VM CID                           VM Type
 harbor-app/57d51b5a-cd39-4b71-bb66-a08184ae9842  running AZ1 10.193.121.13 vm-c5eb38a5-acd7-4769-a731-617b0de04dd5  large.disk
 :
  
 ~$ bosh -e pks -d harbor-container-registry-74b44adfd44a90f27cb3 ssh harbor-app/57d51b5a-cd39-4b71-bb66-a08184ae9842
 harbor-app/57d51b5a-cd39-4b71-bb66-a08184ae9842:~$ sudo -i
 harbor-app/57d51b5a-cd39-4b71-bb66-a08184ae9842:~# alias docker='/var/vcap/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock'
 

Clear out all users except admin/anonymous. If there is any foreign key association, it is also necessary to remove the association

harbor-app/57d51b5a-cd39-4b71-bb66-a08184ae9842:/# docker exec -it harbor-db bash
root [ / ]# psql -U postgres
\c registry
select * from harbor_user;
delete from harbor_user where user_id > 2;

Refresh the Harbor web console-> Configurations, then you can change the auth_mode to OIDC.

Done.

VMware Influencer 100 Club

I am so happy as well for this recognition and gladly accept the invitation to join the VMware Influencer 100 Club. Thanks to Noell Grier and VMware for this amazing award!