Is this a request for help?

Yes.

What keywords did you search in kubeadm issues before filing this one?

• Certificates
• Cert rotation
• PKI
• kubelet
• kubeadm

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST

• Bug report: unexpected functionality.

Versions

1.15.4-00

Environment:

• Kubernetes version: 1.15.4-00.
• Cloud provider or hardware configuration: kubeadm on Intel x86_x64 box.
• OS: Ubuntu Server 18.04 LTS x86_64.
• Kernel: 5.0.0-29-generic.

What happened?

I configured the ConfigurationManager static pod to use a run-time flag to force certs to expire every 15 minutes instead of every 365 days.

sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep -B20 -A5 duration

spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.244.0.0/16
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    - --experimental-cluster-signing-duration=0h15m0s
    image: k8s.gcr.io/kube-controller-manager:v1.15.4
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:

I then re-create the kubelet PEM cert, on the master and all nodes (or I can wait for the existing certs to expire; the PEM rotation/updates work either way):

# Optional: this will happen automatically on the master and nodes (eventually).
# Kubelet client cert (run on master and all nodes) so kubelet can talk to the API server.
# Working.
sudo rm -rf "/var/lib/kubelet/pki-backup"
sudo mv "/var/lib/kubelet/pki" "/var/lib/kubelet/pki-backup"
sudo systemctl restart kubelet
# Don't skip this delay. The new PEM file takes a while to appear.
sleep 30

I can confirm that the PEM cert has been updated. By polling via the following command, I see that it is updated periodically on the master and nodes via:
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep -A2 "Validity"

Validity
            Not Before: Oct  8 17:06:00 2019 GMT
            Not After : Oct  8 17:21:00 2019 GMT

However, other certs (e.g. API server cert, and basically all the certs in /etc/kubernetes/pki on the master) have two issues:

They appear to remain unchanged. This is confirmed via:
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A2 "Validity"

        Validity
            Not Before: Oct  7 21:08:23 2019 GMT
            Not After : Oct  7 14:56:02 2020 GMT

...and:
sudo kubeadm alpha certs check-expiration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Oct 07, 2020 14:56 UTC   364d            no
apiserver                  Oct 07, 2020 14:56 UTC   364d            no
apiserver-etcd-client      Oct 07, 2020 14:56 UTC   364d            no
apiserver-kubelet-client   Oct 07, 2020 14:56 UTC   364d            no
controller-manager.conf    Oct 07, 2020 14:56 UTC   364d            no
etcd-healthcheck-client    Oct 07, 2020 14:56 UTC   364d            no
etcd-peer                  Oct 07, 2020 14:56 UTC   364d            no
etcd-server                Oct 07, 2020 14:56 UTC   364d            no
front-proxy-client         Oct 07, 2020 14:56 UTC   364d            no
scheduler.conf             Oct 07, 2020 14:56 UTC   364d            no

Also, config files (e.g. /etc/kubernetes/admin.conf) also appear to be referring to out-of-date certs.
sudo cat /etc/kubernetes/admin.conf | grep "certificate-authority-data" | cut -d ':' -f2- | sed "s/^\s\+//g" | base64 -d | openssl x509 -noout -text -in - | grep -A2 "Validity"

However, the certs don't get (forcibly) updated. So, I execute the following:
sudo kubeadm alpha certs renew all

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healtcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Now, checking again:
sudo kubeadm alpha certs check-expiration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Oct 07, 2020 17:10 UTC   364d            no
apiserver                  Oct 07, 2020 17:10 UTC   364d            no
apiserver-etcd-client      Oct 07, 2020 17:10 UTC   364d            no
apiserver-kubelet-client   Oct 07, 2020 17:10 UTC   364d            no
controller-manager.conf    Oct 07, 2020 17:10 UTC   364d            no
etcd-healthcheck-client    Oct 07, 2020 17:10 UTC   364d            no
etcd-peer                  Oct 07, 2020 17:10 UTC   364d            no
etcd-server                Oct 07, 2020 17:10 UTC   364d            no
front-proxy-client         Oct 07, 2020 17:10 UTC   364d            no
scheduler.conf             Oct 07, 2020 17:10 UTC   364d            no

The certs are updated, but, the duration/lifetime is still a year, rather than 15 minutes. Additionally, it appears that /etc/kubernetes/admin.conf is not updated at all.
sudo cat /etc/kubernetes/admin.conf | grep "certificate-authority-data" | cut -d ':' -f2- | sed "s/^\s\+//g" | base64 -d | openssl x509 -noout -text -in - | grep -A2 "Validity"

        Validity
            Not Before: Oct  7 21:08:23 2019 GMT
            Not After : Oct  4 21:08:23 2029 GMT

What you expected to happen?

1. All the /etc/kubernetes/pki/ certs would be updated (good so far), and would have a 15 minute lifetime (didn't happen; is the --experimental-cluster-signing-duration expected to impact the config files and certs, or just the PEM file used by kubelet?).
2. The various config files (including /etc/kubernetes/admin.conf) would be (forcibly) updated to reflect new certificates being generated.

How to reproduce it (as minimally and precisely as possible)?

CLI examples provided above.

Anything else we need to know?

I'm attempting to implement a recurring job that will execute either once every 2 hours, or once every week, for a set of isolated kubeadm-based bare-metal and VM-based clusters on air-gapped private networks, being used by teams of students. I need to be able to forcibly update all the certificates periodcially (i.e. much more frequently than once per year) for some labs. Since these machines have no internet access, relying on the upgrade option to automatically handle cert upgrades is not practical/viable. However, I'm now concerned that both manually and automatically driven certificate updates might not update the config files and /etc/kubernetes/pki/ certs (and would like to be able to forcibly update them and reduce their duration/lifetime like the PEM files to confirm all certs are updated as expected).

Thank you for your time and assistance.

References:

1. "Improvement for k8s.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ #15292", <https://github.com/kubernetes/website/issues/15292>, last accessed 2019-10-08.
2. "kubeadm alpha phase certs renew all should also update certs in KubeConfig files kubeadm alpha phase certs renew all should also update certs in KubeConfig files #1361", <https://github.com/kubernetes/kubeadm/issues/1361>, last accessed 2019-10-08.
3. "Kubelet fails to authenticate to apiserver due to expired certificate #65991", <https://github.com/kubernetes/kubernetes/issues/65991>, last accessed 2019-10-08.