Skip to content

kube-scheduler failed with "listen tcp 127.0.0.1:10259: bind: address already in use" #101727

New issue
Jump to bottom
New issue
Jump to bottom
Closed
Closed
kube-scheduler failed with "listen tcp 127.0.0.1:10259: bind: address already in use"#101727
Labels
kind/bugCategorizes issue or PR as related to a bug.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.sig/appsCategorizes an issue or PR as relevant to SIG Apps.
@adabuleanu

Description

@adabuleanu
adabuleanu
opened on May 4, 2021

What happened:

kube-scheduler failed with the error with "listen tcp 127.0.0.1:10259: bind: address already in use". While debugging found out that two containers of kube-scheduler were running

$ kubectl get pods -A
...
NAMESPACE                           NAME                                                              READY   STATUS             RESTARTS   AGE
kube-system                         coredns-74ff55c5b-ggsw2                                           1/1     Running            0          8h
kube-system                         coredns-74ff55c5b-xxc5g                                           1/1     Running            0          8h
kube-system                         etcd-nodename-0                                                   1/1     Running            2          5h19m
kube-system                         kube-apiserver-nodename-0                                         1/1     Running            2          5h19m
kube-system                         kube-controller-manager-nodename-0                                1/1     Running            15         8h
kube-system                         kube-flannel-ds-amd64-swc6l                                       1/1     Running            0          8h
kube-system                         kube-proxy-9z84m                                                  1/1     Running            0          8h
kube-system                         kube-scheduler-nodename-0                                         0/1     CrashLoopBackOff   46         8h
kube-system                         metrics-server-5875f65858-9d96p                                   1/1     Running            7          8h
kube-system                         paas-hostpath-hostpath-554fc7f9cf-rccfr                           1/1     Running            16         8h
kube-system                         registry-6ccf6b9595-4ck7k                                         2/2     Running            0          5h15m
kube-system                         tiller-deploy-86bdbcbc4b-q98hw                                    1/1     Running            0          8h$ kubectl logs -n kube-system                         kube-scheduler-nodename-0
I0504 12:59:32.657942       1 serving.go:331] Generated self-signed cert in-memory
failed to create listener: failed to listen on 127.0.0.1:10259: listen tcp 127.0.0.1:10259: bind: address already in use
[cloud-user@ncicommon-nt1vmupgrhel8-42-1 ~]$ sudo netstat -tupan | grep 10259
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      1138859/kube-schedu 

$ sudo docker ps --all| grep scheduler
f4e6c720888d        5f8cb769bd73                                                                                "kube-scheduler --au…"    4 minutes ago       Exited (1) 4 minutes ago                       k8s_kube-scheduler_kube-scheduler-nodename-0_kube-system_137eac2191896f03479aee17f25b0a25_37
75d684738dcc        5f8cb769bd73                                                                                "kube-scheduler --au…"    2 hours ago         Up 2 hours                                     k8s_kube-scheduler_kube-scheduler-nodename-0_kube-system_137eac2191896f03479aee17f25b0a25_14
469ee580a24f        k8s.gcr.io/pause:3.2                                                                        "/pause"                  8 hours ago         Up 8 hours                                     k8s_POD_kube-scheduler-nodename-0_kube-system_137eac2191896f03479aee17f25b0a25_0
$ docker logs -f f4e6c720888d
I0504 13:46:20.492121       1 serving.go:331] Generated self-signed cert in-memory
failed to create listener: failed to listen on 127.0.0.1:10259: listen tcp 127.0.0.1:10259: bind: address already in use
[cloud-user@ncicommon-nt1vmupgrhel8-42-1 ~]$ docker logs 75d684738dcc
I0504 11:58:27.334250       1 serving.go:331] Generated self-signed cert in-memory
W0504 11:58:27.817903       1 authentication.go:336] Error looking up in-cluster authentication configuration: Get "https://10.10.11.35:10443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication?timeout=10s": dial tcp 10.10.11.35:10443: connect: connection refused
W0504 11:58:27.817957       1 authentication.go:337] Continuing without authentication configuration. This may treat all requests as anonymous.
W0504 11:58:27.817967       1 authentication.go:338] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0504 11:58:27.827031       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0504 11:58:27.827059       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0504 11:58:27.827370       1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0504 11:58:27.827439       1 tlsconfig.go:240] Starting DynamicServingCertificateController
E0504 11:58:27.827717       1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://10.10.11.35:10443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.828039       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.828909       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://10.10.11.35:10443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.829505       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://10.10.11.35:10443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.829629       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.10.11.35:10443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.829722       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://10.10.11.35:10443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.830202       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://10.10.11.35:10443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.830518       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://10.10.11.35:10443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.830544       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://10.10.11.35:10443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.830680       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://10.10.11.35:10443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.831380       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.10.11.35:10443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:27.831572       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:28.801571       1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://10.10.11.35:10443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:28.858035       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.10.11.35:10443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:28.889131       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:28.931099       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://10.10.11.35:10443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:28.980898       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://10.10.11.35:10443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:29.004055       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://10.10.11.35:10443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:29.007152       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.10.11.35:10443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:29.051071       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://10.10.11.35:10443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:29.186293       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:29.198079       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://10.10.11.35:10443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:29.316206       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://10.10.11.35:10443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:29.396071       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://10.10.11.35:10443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:30.664206       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://10.10.11.35:10443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:30.807940       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:30.887687       1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://10.10.11.35:10443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:30.925460       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://10.10.11.35:10443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:31.001559       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://10.10.11.35:10443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:31.021367       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://10.10.11.35:10443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:31.054778       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://10.10.11.35:10443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:31.165378       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://10.10.11.35:10443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:31.204248       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.10.11.35:10443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:31.674550       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://10.10.11.35:10443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:31.883935       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:32.074623       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.10.11.35:10443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:34.380524       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://10.10.11.35:10443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:34.473180       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.10.11.35:10443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:34.876608       1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://10.10.11.35:10443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:35.501791       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://10.10.11.35:10443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:35.514452       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:35.727693       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://10.10.11.35:10443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:35.943258       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:36.780921       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://10.10.11.35:10443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:37.019399       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://10.10.11.35:10443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:37.092071       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://10.10.11.35:10443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:37.667238       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://10.10.11.35:10443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:38.063872       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.10.11.35:10443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:41.674727       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://10.10.11.35:10443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:43.666873       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.10.11.35:10443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:44.261815       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://10.10.11.35:10443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:44.280258       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://10.10.11.35:10443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:44.633157       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:44.816442       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://10.10.11.35:10443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:45.218257       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.10.11.35:10443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:45.378785       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://10.10.11.35:10443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:45.536283       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://10.10.11.35:10443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:45.571586       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://10.10.11.35:10443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:45.696922       1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://10.10.11.35:10443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
E0504 11:58:49.072972       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://10.10.11.35:10443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 10.10.11.35:10443: connect: connection refused
I0504 11:59:05.084379       1 trace.go:205] Trace[1130951312]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (04-May-2021 11:58:55.078) (total time: 10005ms):
Trace[1130951312]: [10.005410204s] [10.005410204s] END
E0504 11:59:05.084435       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://10.10.11.35:10443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": net/http: TLS handshake timeout
I0504 11:59:09.081145       1 trace.go:205] Trace[355205737]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (04-May-2021 11:58:59.079) (total time: 10001ms):
Trace[355205737]: [10.001317971s] [10.001317971s] END
E0504 11:59:09.081187       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://10.10.11.35:10443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": net/http: TLS handshake timeout
I0504 11:59:09.719690       1 trace.go:205] Trace[1787676155]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (04-May-2021 11:58:59.718) (total time: 10001ms):
Trace[1787676155]: [10.001371885s] [10.001371885s] END
E0504 11:59:09.719755       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://10.10.11.35:10443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": net/http: TLS handshake timeout
I0504 11:59:10.257935       1 trace.go:205] Trace[821030791]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (04-May-2021 11:59:00.256) (total time: 10001ms):
Trace[821030791]: [10.00106918s] [10.00106918s] END
E0504 11:59:10.257960       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://10.10.11.35:10443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": net/http: TLS handshake timeout
E0504 11:59:10.443367       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0504 11:59:10.443428       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0504 11:59:10.443477       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0504 11:59:10.443537       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0504 11:59:10.443576       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0504 11:59:10.443624       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0504 11:59:10.443671       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
I0504 11:59:10.527309       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0504 11:59:58.027879       1 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-scheduler...
I0504 12:00:15.583305       1 leaderelection.go:253] successfully acquired lease kube-system/kube-scheduler

What you expected to happen:

I expect that the initial kube-scheduler container to be terminated and the new one to take its place.

How to reproduce it (as minimally and precisely as possible):

Reproduced this on an automated pipeline (very rare). Found another similar issue that is closed due to inactivity #84889. Also some other issues that report the same behavior, but for other components #85483 #61901

Anything else we need to know?:

Kubelet logs: kubelet.log
Docker logs: docker.log

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:03:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: Openstack
  • OS (e.g: cat /etc/os-release): Red Hat Enterprise Linux Server 7.9 (core)
  • Kernel (e.g. uname -a): 3.10.0-1160.21.1.el7.x86_64
  • Install tools: kubeadm
  • Network plugin and version (if this is a network-related bug): flannel v0.12.0-34-g8936e90-amd64
  • Others:

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.sig/appsCategorizes an issue or PR as relevant to SIG Apps.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions