Skip to content

kubelet ignores updated etcd.yaml and monitors only etcd.yaml.backup #129364

Open
@rad1k4l

Description

What happened?

During the kubeadm upgrade process, kubeadm creates a backup of the existing etcd manifest (etcd.yaml.backup) and updates the etcd.yaml manifest to a newer version (e.g., from etcd 3.4.13-0 to 3.5.16-0). However, post-upgrade, the kubelet appears to ignore the updated etcd.yaml and continues to monitor and apply changes only to etcd.yaml.backup. This behavior prevents the etcd cluster from upgrading, causing the kubeadm upgrade process to fail.

What did you expect to happen?

After initiating the kubeadm upgrade:

  • kubeadm should update the etcd.yaml manifest to the new etcd version (3.5.16-0).
  • kubelet should detect changes in etcd.yaml, apply the updated configuration, and successfully upgrade the etcd cluster to version 3.5.16-0.
  • The etcd.yaml.backup should remain as a backup and kubelet should continue to monitor only the primary etcd.yaml manifest.

How can we reproduce it (as minimally and precisely as possible)?

There are two methods to reproduce the issue: Automated Upgrade via kubeadm and Manual Updating of etcd.

Automated Upgrade via kubeadm:

  • Attempt to upgrade the cluster using kubeadm to upgrade etcd to version 3.5.16-0:

kubeadm upgrade apply v1.32.0

  • kubeadm creates etcd.yaml.backup and updates etcd.yaml to version 3.5.16-0.
  • kubelet ignores etcd.yaml(verison 3.5.16-0 ) and only monitors etcd.yaml.backup ( version 3.4.13-0), preventing the etcd upgrade.

Manual :

  • Create a backup of the existing etcd.yaml file:
    cp /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.backup

  • Open etcd.yaml in a text editor and update the etcd image version from 3.4.13-0 to 3.5.16-0.

  • Restart the kubelet service:
    systemctl restart kubelet

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.31.4

Cloud provider

On-premises bare-metal servers

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

containerd

Related plugins (CNI, CSI, ...) and versions (if applicable)

Metadata

Assignees

No one assigned

    Labels

    kind/documentationCategorizes issue or PR as related to documentation.kind/supportCategorizes issue or PR as a support question.sig/nodeCategorizes an issue or PR as relevant to SIG Node.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

    Type

    No type

    Projects

    • Status

      Triaged

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions