What happened:
The in-tree vpshere cloud provider does not support SAML auth tokens for the tag-based node labeling. The provider relies on the withTagsClient function to authenticate with the vapi rest endpoints. This function assumes that the username and password fields are set to a username and password. In the case of SAML auth, these fields must be set to a cert and key, respectively. The SAML auth setup is described here: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/saml-token-authentication.html

The vSphere cloud provider when not fetching tags uses a different auth handler which does properly handle SAML auth - https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vclib/connection.go#L91

So when using SAML auth tokens, the cloud provider is able to authenticate with the vSphere API to do things like identifying the vm from the uuid, but not able to authenticate with the vSphere API when fetching tags because it tries to use the cert and key as a basic auth username and password.

What you expected to happen:
The kubelet is able to acquire the tag-based labels for the node object using SAML token auth.

How to reproduce it (as minimally and precisely as possible):
Set up SAML token authentication as described in the official docs: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/saml-token-authentication.html

The kubelet logs indicate that zone info (from vm tags) cannot be retrieved due to an auth issue:

Mar 20 10:48:34 ul-lin-mast-8k kubelet[11752]: E0320 10:48:34.326226   11752 vsphere.go:1528] Get zone for node ul-lin-mast-8k: POST https://vcenterdo.arlington.predictivetechnologies.com:443/rest/com/vmware/cis/session: 401 Unauthorized

Anything else we need to know?:
Support for SSO in the govmomi vAPI was implemented here: vmware/govmomi@54a181a

Environment:

• Kubernetes version (use kubectl version):
  1.14.0-beta.1
• Cloud provider or hardware configuration:
  vSphere
• OS (e.g: cat /etc/os-release):
  ubuntu 18.04