kubelet ignores updated etcd.yaml
and monitors only etcd.yaml.backup
#129364
Description
What happened?
During the kubeadm upgrade
process, kubeadm
creates a backup of the existing etcd manifest (etcd.yaml.backup
) and updates the etcd.yaml
manifest to a newer version (e.g., from etcd 3.4.13-0 to 3.5.16-0). However, post-upgrade, the kubelet
appears to ignore the updated etcd.yaml
and continues to monitor and apply changes only to etcd.yaml.backup
. This behavior prevents the etcd cluster from upgrading, causing the kubeadm upgrade
process to fail.
What did you expect to happen?
After initiating the kubeadm upgrade:
- kubeadm should update the etcd.yaml manifest to the new etcd version (3.5.16-0).
- kubelet should detect changes in etcd.yaml, apply the updated configuration, and successfully upgrade the etcd cluster to version 3.5.16-0.
- The etcd.yaml.backup should remain as a backup and kubelet should continue to monitor only the primary etcd.yaml manifest.
How can we reproduce it (as minimally and precisely as possible)?
There are two methods to reproduce the issue: Automated Upgrade via kubeadm and Manual Updating of etcd.
Automated Upgrade via kubeadm:
- Attempt to upgrade the cluster using kubeadm to upgrade etcd to version 3.5.16-0:
kubeadm upgrade apply v1.32.0
- kubeadm creates
etcd.yaml.backup
and updatesetcd.yaml
to version 3.5.16-0. kubelet
ignoresetcd.yaml
(verison 3.5.16-0 ) and only monitorsetcd.yaml.backup
( version 3.4.13-0), preventing the etcd upgrade.
Manual :
-
Create a backup of the existing
etcd.yaml
file:
cp /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.backup
-
Open etcd.yaml in a text editor and update the etcd image version from
3.4.13-0
to3.5.16-0
. -
Restart the kubelet service:
systemctl restart kubelet
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.31.4
Cloud provider
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
Metadata
Assignees
Labels
Type
Projects
Status
Triaged