Cluster upgrade to v1.30.2 fails on "Upgrade first Control Plane" #11350
Closed
Description
What happened?
Attempted to upgrade a cluster from v1.29.3 to v1.30.2. The upgrade playbook fails on kubeadm upgrade apply, with error can not mix '--config' with arguments [allow-experimental-upgrades certificate-renewal etcd-upgrade force yes], in this task:
TASK [kubernetes/control-plane : Kubeadm | Upgrade first master] ************************************************
Wednesday 03 July 2024 17:18:34 +0000 (0:00:01.906) 0:31:56.562 ********
FAILED - RETRYING: [k8s-staging-01-master]: Kubeadm | Upgrade first master (3 retries left).
FAILED - RETRYING: [k8s-staging-01-master]: Kubeadm | Upgrade first master (2 retries left).
FAILED - RETRYING: [k8s-staging-01-master]: Kubeadm | Upgrade first master (1 retries left).
fatal: [k8s-staging-01-master]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["timeout", "-k", "600s", "600s", "/usr/local/bin/kubeadm", "upgrade", "apply", "-y", "v1.30.2", "--certificate-renewal=True", "--config=/etc/kubernetes/kubeadm-config.yaml", "--ignore-preflight-errors=all", "--allow-experimental-upgrades", "--etcd-upgrade=false", "--force"], "delta": "0:00:00.083731", "end": "2024-07-03 17:18:55.750605", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2024-07-03 17:18:55.666874", "stderr": "can not mix '--config' with arguments [allow-experimental-upgrades certificate-renewal etcd-upgrade force yes]\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["can not mix '--config' with arguments [allow-experimental-upgrades certificate-renewal etcd-upgrade force yes]", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}
What did you expect to happen?
Successful upgrade of the cluster
How can we reproduce it (as minimally and precisely as possible)?
Attempt to upgrade cluster from v.1.29 to v1.30
OS
Linux 5.15.0-113-generic x86_64
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Version of Ansible
ansible [core 2.16.8]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.12/dist-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] (/usr/bin/python3)
jinja version = 3.1.4
libyaml = True
Version of Python
python version = 3.12.3
Version of Kubespray (commit)
Network plugin used
calico
Full inventory with variables
[ Removed, since it was huge and was making the issue difficult to read. Will provide a gist on request, if needed ]
Command used to invoke ansible
ansible-playbook on custom playbook that imports kubespray/playbooks/upgrade_cluster.yml
Output of ansible run
TASK [kubernetes/control-plane : Kubeadm | Upgrade first master] ************************************************
Wednesday 03 July 2024 17:18:34 +0000 (0:00:01.906) 0:31:56.562 ********
FAILED - RETRYING: [k8s-staging-01-master]: Kubeadm | Upgrade first master (3 retries left).
FAILED - RETRYING: [k8s-staging-01-master]: Kubeadm | Upgrade first master (2 retries left).
FAILED - RETRYING: [k8s-staging-01-master]: Kubeadm | Upgrade first master (1 retries left).
fatal: [k8s-staging-01-master]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["timeout", "-k", "600s", "600s", "/usr/local/bin/kubeadm", "upgrade", "apply", "-y", "v1.30.2", "--certificate-renewal=True", "--config=/etc/kubernetes/kubeadm-config.yaml", "--ignore-preflight-errors=all", "--allow-experimental-upgrades", "--etcd-upgrade=false", "--force"], "delta": "0:00:00.083731", "end": "2024-07-03 17:18:55.750605", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2024-07-03 17:18:55.666874", "stderr": "can not mix '--config' with arguments [allow-experimental-upgrades certificate-renewal etcd-upgrade force yes]\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["can not mix '--config' with arguments [allow-experimental-upgrades certificate-renewal etcd-upgrade force yes]", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}
Anything else we need to know
No response