Pods not starting on Kubernetes 1.2-beta.1 with CNI #22932
Closed
Description
We have been running a Kube cluster on 1.2-alpha.7 for quite a while with no problems.
Now trying to upgrade to Kube 1.2-beta.1 the pods end up in a restart loop and never come online.
The upgrade is done via a complete reinstall of OS and Kubernetes and the issue has been recreated several times by upgrading and downgrading between alpha.7 and beta-1
OS: Centos 7.2
Docker: 1.9.1
Kube version: 1.2-beta.1
CNI Provider: calico cni 1.1.0 / calicoctl: 0.17.0
kubectl --namespace=kube-system get pod -o wide
NAME READY STATUS RESTARTS AGE NODE
kube-dns-v10-0nh1i 3/4 Running 68 20m srv07
kube-dns-v10-vfj6d 3/4 Running 60 20m srv05
kube-registry-v0-m0or6 1/1 Running 44 20m srv05
kube-ui-v5-gxs27 1/1 Running 197 20m srv04
kubectl get events for one pod
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
───────── ──────── ───── ──── ───────────── ──────── ────── ───────
11m 10m 5 {default-scheduler } Warning FailedScheduling no nodes available to schedule pods
10m 10m 1 {default-scheduler } Normal Scheduled Successfully assigned kube-dns-v10-0nh1i to srv07
10m 10m 1 {kubelet srv07} spec.containers{etcd} Normal Pulling pulling image "gcr.io/google_containers/etcd:2.0.9"
10m 10m 1 {kubelet srv07} spec.containers{etcd} Normal Pulled Successfully pulled image "gcr.io/google_containers/etcd:2.0.9"
10m 10m 1 {kubelet srv07} spec.containers{etcd} Normal Created Created container with docker id 3efd24477ee2
10m 10m 1 {kubelet srv07} spec.containers{etcd} Normal Started Started container with docker id 3efd24477ee2
10m 10m 1 {kubelet srv07} spec.containers{kube2sky} Normal Pulling pulling image "gcr.io/google_containers/kube2sky:1.12"
10m 10m 1 {kubelet srv07} spec.containers{kube2sky} Normal Pulled Successfully pulled image "gcr.io/google_containers/kube2sky:1.12"
10m 10m 1 {kubelet srv07} spec.containers{kube2sky} Normal Created Created container with docker id f6ce0663c34b
10m 10m 1 {kubelet srv07} spec.containers{kube2sky} Normal Started Started container with docker id f6ce0663c34b
10m 10m 1 {kubelet srv07} spec.containers{skydns} Normal Pulling pulling image "gcr.io/google_containers/skydns:2015-10-13-8c72f8c"
9m 9m 1 {kubelet srv07} spec.containers{skydns} Normal Pulled Successfully pulled image "gcr.io/google_containers/skydns:2015-10-13-8c72f8c"
9m 9m 1 {kubelet srv07} spec.containers{skydns} Normal Created Created container with docker id bf3ecc042ffd
9m 9m 1 {kubelet srv07} spec.containers{skydns} Normal Started Started container with docker id bf3ecc042ffd
9m 9m 1 {kubelet srv07} spec.containers{healthz} Normal Pulling pulling image "gcr.io/google_containers/exechealthz:1.0"
9m 9m 1 {kubelet srv07} spec.containers{healthz} Normal Pulled Successfully pulled image "gcr.io/google_containers/exechealthz:1.0"
9m 9m 1 {kubelet srv07} spec.containers{healthz} Normal Created Created container with docker id 28a0783cf48f
8m 8m 1 {kubelet srv07} spec.containers{healthz} Normal Started Started container with docker id 28a0783cf48f
8m 8m 1 {kubelet srv07} spec.containers{skydns} Normal Killing Killing container with docker id bf3ecc042ffd: Need to kill pod.
8m 8m 1 {kubelet srv07} spec.containers{kube2sky} Normal Killing Killing container with docker id f6ce0663c34b: Need to kill pod.
8m 8m 1 {kubelet srv07} spec.containers{healthz} Normal Killing Killing container with docker id 28a0783cf48f: Need to kill pod.
8m 8m 1 {kubelet srv07} spec.containers{etcd} Normal Killing Killing container with docker id 3efd24477ee2: Need to kill pod.
8m 8m 1 {kubelet srv07} spec.containers{etcd} Normal Created Created container with docker id f37dd5c64348
8m 8m 1 {kubelet srv07} spec.containers{etcd} Normal Started Started container with docker id f37dd5c64348
8m 8m 1 {kubelet srv07} spec.containers{kube2sky} Normal Created Created container with docker id bf2c1a408754
8m 8m 1 {kubelet srv07} spec.containers{kube2sky} Normal Started Started container with docker id bf2c1a408754
8m 8m 1 {kubelet srv07} spec.containers{skydns} Normal Created Created container with docker id 0668fa66b8f7
8m 8m 1 {kubelet srv07} spec.containers{skydns} Normal Started Started container with docker id 0668fa66b8f7
8m 8m 1 {kubelet srv07} spec.containers{healthz} Normal Created Created container with docker id 739e3f9c828d
7m 7m 1 {kubelet srv07} spec.containers{healthz} Normal Started Started container with docker id 739e3f9c828d
7m 7m 1 {kubelet srv07} spec.containers{etcd} Normal Killing Killing container with docker id f37dd5c64348: Need to kill pod.
7m 7m 1 {kubelet srv07} spec.containers{healthz} Normal Killing Killing container with docker id 739e3f9c828d: Need to kill pod.
7m 7m 1 {kubelet srv07} spec.containers{kube2sky} Normal Killing Killing container with docker id bf2c1a408754: Need to kill pod.
7m 7m 1 {kubelet srv07} spec.containers{skydns} Normal Killing Killing container with docker id 0668fa66b8f7: Need to kill pod.
7m 7m 1 {kubelet srv07} spec.containers{kube2sky} Normal Created Created container with docker id b208af539d35
7m 7m 1 {kubelet srv07} spec.containers{kube2sky} Normal Started Started container with docker id b208af539d35
6m 6m 1 {kubelet srv07} spec.containers{kube2sky} Normal Killing Killing container with docker id b208af539d35: Need to kill pod.
journalctl -xn 200 -u kubelet (kubelet running with -v=5)
I can't see anything weird in the logs for the API Server, Scheduler or Controller Manager