Skip to content

kubeadm and kubelet 1.15 fail to install on centos 7 after patches released today #92242

New issue
Jump to bottom
New issue
Jump to bottom
Closed
Closed
kubeadm and kubelet 1.15 fail to install on centos 7 after patches released today#92242
Assignees
justaugustussaschagrunert
Labels
area/release-engIssues or PRs related to the Release Engineering subprojectkind/bugCategorizes issue or PR as related to a bug.priority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.sig/releaseCategorizes an issue or PR as relevant to SIG Release.
@tstraley

Description

@tstraley
tstraley
opened on Jun 17, 2020

What happened:

Attempted to install kubernetes 1.15 binaries following documentation on Centos 7 (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) and received failures:

# yum install kubelet-1.15* kubeadm-1.15* kubectl-1.15*
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: repos.dfw.quadranet.com
 * extras: linux-mirrors.fnal.gov
 * updates: mirrors.raystedman.org
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
--> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.15.12-0.x86_64
---> Package kubectl.x86_64 0:1.15.12-0 will be installed
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
--> Processing Dependency: iptables >= 1.4.21 for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: socat for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: iproute for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: ethtool for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: ebtables for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.15.12-0.x86_64
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
--> Processing Dependency: libnetfilter_conntrack >= 1.0.6 for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libmnl.so.0(LIBMNL_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnfnetlink.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_conntrack.so.3()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libmnl.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package ebtables.x86_64 0:2.0.10-16.el7 will be installed
---> Package ethtool.x86_64 2:4.8-10.el7 will be installed
---> Package iproute.x86_64 0:4.11.0-25.el7_7.2 will be installed
---> Package iptables.x86_64 0:1.4.21-34.el7 will be installed
---> Package kubeadm.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Processing Dependency: libwrap.so.0()(64bit) for package: socat-1.7.3.2-2.el7.x86_64
--> Running transaction check
---> Package kubeadm.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package libmnl.x86_64 0:1.0.3-7.el7 will be installed
---> Package libnetfilter_conntrack.x86_64 0:1.0.6-1.el7_3 will be installed
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package libnfnetlink.x86_64 0:1.0.1-4.el7 will be installed
---> Package tcp_wrappers-libs.x86_64 0:7.6-77.el7 will be installed
--> Finished Dependency Resolution
Error: Package: kubelet-1.15.12-0.x86_64 (kubernetes)
           Requires: kubernetes-cni >= 0.7.5
           Available: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
               kubernetes-cni = 0.3.0.1-0.07a8a2
           Available: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-0
           Available: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-1
           Available: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
               kubernetes-cni = 0.6.0-0
           Available: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
               kubernetes-cni = 0.7.5-0
Error: Package: kubeadm-1.15.12-0.x86_64 (kubernetes)
           Requires: kubernetes-cni >= 0.7.5
           Available: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
               kubernetes-cni = 0.3.0.1-0.07a8a2
           Available: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-0
           Available: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-1
           Available: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
               kubernetes-cni = 0.6.0-0
           Available: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
               kubernetes-cni = 0.7.5-0
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

What you expected to happen:
Expected these to all install successfully as suggested by the documentation and based on this working prior to today.

How to reproduce it (as minimally and precisely as possible):

Follow these docs with a fresh centos 7 machine: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ use pinned versions of the binaries to 1.15

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

yum install -y kubelet-1.15* kubeadm-1.15* --disableexcludes=kubernetes

Anything else we need to know?:
This worked yesterday before the CVE patches today. This also works fine without the kuberenetes-cni dependency on ubuntu / debian (your apt packages) and with different versions of these binaries (eg. 1.16, 1.17. etc.)

Edit: Per comments below, this does look to affect certain patch levels of 1.17 and 1.18 as well, just not the latest patch of those versions. Might also be affecting certain apt installations, but not the 1.15 latest patch that we are using.

Here is what the repo claims are dependencies of these different versions:

# repoquery --requires --resolve kubelet-1.18
ethtool-2:4.8-10.el7.x86_64
iptables-0:1.4.21-34.el7.x86_64
util-linux-0:2.23.2-63.el7.x86_64
iptables-0:1.4.21-34.el7.i686
util-linux-0:2.23.2-63.el7.i686
conntrack-tools-0:1.4.4-7.el7.x86_64
iproute-0:4.11.0-25.el7_7.2.x86_64
ebtables-0:2.0.10-16.el7.x86_64
glibc-0:2.17-307.el7.1.i686
glibc-0:2.17-307.el7.1.x86_64
socat-0:1.7.3.2-2.el7.x86_64

# repoquery --requires --resolve kubelet-1.15*
ethtool-2:4.8-10.el7.x86_64
iptables-0:1.4.21-34.el7.x86_64
util-linux-0:2.23.2-63.el7.x86_64
iptables-0:1.4.21-34.el7.i686
util-linux-0:2.23.2-63.el7.i686
kubernetes-cni-0:0.7.5-0.x86_64
conntrack-tools-0:1.4.4-7.el7.x86_64
iproute-0:4.11.0-25.el7_7.2.x86_64
ebtables-0:2.0.10-16.el7.x86_64
glibc-0:2.17-307.el7.1.i686
glibc-0:2.17-307.el7.1.x86_64
socat-0:1.7.3.2-2.el7.x86_64

Notice kuberenets-cni showing with kubelet-1.15

Additionally, if you try to install that kuberenetes-cni version, it needs kubelet and cannot install with it ❗

# repoquery --requires --resolve kubernetes-cni-0:0.7.5-0.x86_64
kubelet-0:1.6.12-1.x86_64

# yum install kubelet-1.15* kubernetes-cni
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: repos.dfw.quadranet.com
 * extras: linux-mirrors.fnal.gov
 * updates: mirrors.raystedman.org
Package kubernetes-cni is obsoleted by kubelet, trying to install kubelet-1.18.4-0.x86_64 instead
Resolving Dependencies
--> Running transaction check
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
--> Processing Dependency: iptables >= 1.4.21 for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: socat for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: iproute for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: ethtool for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: ebtables for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.15.12-0.x86_64
---> Package kubelet.x86_64 0:1.18.4-0 will be installed
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
--> Processing Dependency: libnetfilter_conntrack >= 1.0.6 for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libmnl.so.0(LIBMNL_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnfnetlink.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_conntrack.so.3()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libmnl.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
---> Package ebtables.x86_64 0:2.0.10-16.el7 will be installed
---> Package ethtool.x86_64 2:4.8-10.el7 will be installed
---> Package iproute.x86_64 0:4.11.0-25.el7_7.2 will be installed
---> Package iptables.x86_64 0:1.4.21-34.el7 will be installed
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Processing Dependency: libwrap.so.0()(64bit) for package: socat-1.7.3.2-2.el7.x86_64
--> Running transaction check
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package libmnl.x86_64 0:1.0.3-7.el7 will be installed
---> Package libnetfilter_conntrack.x86_64 0:1.0.6-1.el7_3 will be installed
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package libnfnetlink.x86_64 0:1.0.1-4.el7 will be installed
---> Package tcp_wrappers-libs.x86_64 0:7.6-77.el7 will be installed
--> Finished Dependency Resolution
Error: Package: kubelet-1.15.12-0.x86_64 (kubernetes)
           Requires: kubernetes-cni >= 0.7.5
           Available: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
               kubernetes-cni = 0.3.0.1-0.07a8a2
           Available: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-0
           Available: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-1
           Available: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
               kubernetes-cni = 0.6.0-0
           Available: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
               kubernetes-cni = 0.7.5-0
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Installing kuberenets-cni on it's own tries to install kubelet / kubeadm 1.18 which we don't want... So there is currently NO way I can find to install kubelet-1.15 😢

Environment:

  • Kubernetes version (use kubectl version): 1.15*
  • Cloud provider or hardware configuration:
    any centos (AWS Ec2, bare metal, docker)
  • OS (e.g: cat /etc/os-release):
# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Kernel (e.g. uname -a):
# uname -a
Linux b80a0f2be28d 4.15.0-99-generic #100-Ubuntu SMP Wed Apr 22 20:32:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
    yum
  • Network plugin and version (if this is a network-related bug):
  • Others:

Metadata

Assignees

Labels

area/release-engIssues or PRs related to the Release Engineering subprojectkind/bugCategorizes issue or PR as related to a bug.priority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.sig/releaseCategorizes an issue or PR as relevant to SIG Release.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions