Context

We're trying to use kubeadm in external CA mode with a two-tier CA configuration. This is done to improve security through shorter expiries while giving us an easy way to rotate the signing CA. We have this setup for each of the required CAs:

• Root CA
• Intermediate CA (issued by Root CA)

We use the kubeadm certs generate-csr command to generate CSRs and issue all of the certificates under the appropriate intermediate CA. With this setup, certificates such as apiserver.crt must be a bundle with the leaf certificate followed by the intermediate CA certificate. The root CA can also be included at the end of the bundle, but it's not required to construct a valid chain as the verifier must already know of the root CA.

What went wrong

Unfortunately, this approach doesn't seem to work with kubeadm as it performs some steps to verify that certificates were issued by the root CA. Our certificates are intentionally not issued by the root CA though, they're issued by the intermediate CA. What kubeadm should be doing is verifying that the certificate bundle contains a valid chain back to the root CA.

This verification step seems to cause all of the kubeadm init phases I've tested to fail with the following error:

W1203 04:04:10.101002    4228 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
crypto/rsa: verification error
certificate API server is not signed by corresponding CA
k8s.io/kubernetes/cmd/kubeadm/app/phases/certs.validateSignedCertWithCA
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/certs/certs.go:412
k8s.io/kubernetes/cmd/kubeadm/app/phases/certs.validateSignedCert
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/certs/certs.go:399

Simple fix

I've recompiled kubeadm with a quick change to comment out this line of code which seems to be sufficient to allow all of the init phases to run to completion. I also haven't observed any issues with my single node test cluster; the control plane components are all running and the kubeconfig files can be used to connect to the API. This suggests to me that the two-tier CA configuration is supported by the rest of the Kubernetes components, it's just this verification step in kubeadm that's failing.

Commenting out an error obviously isn't a real fix though. Fixing this properly would involve updating this verification step to check that there is a valid chain back to the root CA.