Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Daemonset is trying to schedule on master #29108

Closed
noose opened this issue Jul 18, 2016 · 5 comments
Closed

Daemonset is trying to schedule on master #29108

noose opened this issue Jul 18, 2016 · 5 comments
Assignees
@davidopp
Labels
sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.

Comments

@noose
Copy link

noose commented Jul 18, 2016

For example:

% kubectl get pods --namespace kube-system
NAME                                                                   READY     STATUS             RESTARTS   AGE
[...]
kube-dns-v14-bx8q3                                                     2/3       CrashLoopBackOff   25         1h
[...]
% kubectl --namespace kube-system describe pod/kube-dns-v14-bx8q3
Name:       kube-dns-v14-bx8q3
Namespace:  kube-system
Node:       ip-172-20-91-250.us-west-1.compute.internal/172.20.91.250
Start Time: Mon, 18 Jul 2016 13:41:22 +0200
Labels:     k8s-app=kube-dns,kubernetes.io/cluster-service=true,version=v14
Status:     Running
IP:     100.96.5.3
Controllers:    ReplicationController/kube-dns-v14
Containers:
  kubedns:
    Container ID:   docker://d1fafa0eadec972f66887c3e575e9d049f4b348e14dac96a577a6d91d323175d
    Image:      gcr.io/google_containers/kubedns-amd64:1.3
    Image ID:       docker://sha256:e43041c3737569d3e218c9aae9013addd797c9b10f889aaf131c60eb4650cd5e
    Ports:      10053/UDP, 10053/TCP
    Args:
      --domain=cluster.local.
      --dns-port=10053
    QoS Tier:
      cpu:  Guaranteed
      memory:   Burstable
    Limits:
      cpu:  100m
      memory:   200Mi
    Requests:
      cpu:      100m
      memory:       50Mi
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Mon, 18 Jul 2016 14:58:23 +0200
      Finished:     Mon, 18 Jul 2016 14:59:33 +0200
    Ready:      False
    Restart Count:  25
    Liveness:       http-get http://:8080/healthz delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:      http-get http://:8081/readiness delay=30s timeout=5s period=10s #success=1 #failure=3
    Environment Variables:
  dnsmasq:
    Container ID:   docker://5ea40fd066740801b50a8c3be8fd4e406e2c1a349f13d421cf4fd0c71a456a06
    Image:      gcr.io/google_containers/dnsmasq:1.1
    Image ID:       docker://sha256:d965753679e59e52913edbc2ab7eced430e4eaaa5cc10a8dc7002eadea2b2471
    Ports:      53/UDP, 53/TCP
    Args:
      --cache-size=1000
      --no-resolv
      --server=127.0.0.1#10053
    QoS Tier:
      cpu:      BestEffort
      memory:       BestEffort
    State:      Running
      Started:      Mon, 18 Jul 2016 13:41:28 +0200
    Ready:      True
    Restart Count:  0
    Environment Variables:
  healthz:
    Container ID:   docker://0c503bb4439b832c0a199f3db420a30bd98ccce459c65413de480afc421e1722
    Image:      gcr.io/google_containers/exechealthz-amd64:1.0
    Image ID:       docker://sha256:82a141f5d06d2136e324e44e90225c672509d6c453b6a96911d3a5b880a9e013
    Port:       8080/TCP
    Args:
      -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
      -port=8080
    QoS Tier:
      cpu:  Guaranteed
      memory:   Guaranteed
    Limits:
      memory:   20Mi
      cpu:  10m
    Requests:
      cpu:      10m
      memory:       20Mi
    State:      Running
      Started:      Mon, 18 Jul 2016 13:41:29 +0200
    Ready:      True
    Restart Count:  0
    Environment Variables:
Conditions:
  Type      Status
  Initialized   True
  Ready     False
  PodScheduled  True
Volumes:
  default-token-hlg6g:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-hlg6g
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath           Type        Reason      Message
  --------- --------    -----   ----                            -------------           --------    ------      -------
  1h        4m      19  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    Normal      Started     (events with common reason combined)
  1h        4m      19  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    Normal      Created     (events with common reason combined)
  1h        4m      25  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    Normal      Pulled      Container image "gcr.io/google_containers/kubedns-amd64:1.3" already present on machine
  1h        3m      108 {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    Warning     Unhealthy   Readiness probe failed: Get http://100.96.5.3:8081/readiness: dial tcp 100.96.5.3:8081: getsockopt: connection refused
  1h        3m      30  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    Warning     Unhealthy   Liveness probe failed: HTTP probe failed with statuscode: 503
  1h        3m      17  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    Normal      Killing     (events with common reason combined)
  1h        2s      242 {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    Warning     BackOff     Back-off restarting failed docker container
  1h        2s      234 {kubelet ip-172-20-91-250.us-west-1.compute.internal}                   Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-v14-bx8q3_kube-system(8d4beba2-4cdc-11e6-9445-02083ebe7f49)"
@apelisse apelisse added team/cluster sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. labels Jul 18, 2016
@apelisse
Copy link
Member

@davidopp PTAL

@noose
Copy link
Author

noose commented Jul 18, 2016 •
edited
Loading

Ok, dns was bad example:

% kubectl get pods
default       reverse-proxy-1e5jp                                                    1/1       Running             0          5h
default       reverse-proxy-8ubrr                                                    0/1       ContainerCreating   0          5h
default       reverse-proxy-lgfrh                                                    1/1       Running             0          5h
default       squid-dmvsj                                                            1/1       Running             0          5h
default       squid-k4yin                                                            0/1       ContainerCreating   0          5h
default       squid-ldfxx                                                            1/1       Running             0          5h
% kubectl describe pod/reverse-proxy-8ubrr pod/squid-k4yin
Name:       reverse-proxy-8ubrr
Namespace:  default
Node:       ip-172-20-119-184.us-west-1.compute.internal/172.20.119.184
Start Time: Mon, 18 Jul 2016 14:04:24 +0200
Labels:     name=reverse-proxy
Status:     Pending
IP:
Controllers:    DaemonSet/reverse-proxy
Containers:
  reverse-proxy:
    Container ID:
    Image:      my-company/kubernetes-reverseproxy
    Image ID:
    Port:       80/TCP
    QoS Tier:
      cpu:      BestEffort
      memory:       BestEffort
    State:      Waiting
      Reason:       ContainerCreating
    Ready:      False
    Restart Count:  0
    Environment Variables:
      ETCD: 172.20.119.184:4001
Conditions:
  Type      Status
  Initialized   True
  Ready     False
  PodScheduled  True
Volumes:
  default-token-cxdw7:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-cxdw7
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath   Type        Reason      Message
  --------- --------    -----   ----                            -------------   --------    ------      -------
  5h        3s      1950    {kubelet ip-172-20-119-184.us-west-1.compute.internal}          Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: no available IPv4 addresses on this network's address pools: bridge (b355fd5c937ecebce956807ba588550f33589ff04ff2458f5593f2caa03af76a)"



Name:       squid-k4yin
Namespace:  default
Node:       ip-172-20-119-184.us-west-1.compute.internal/172.20.119.184
Start Time: Mon, 18 Jul 2016 14:00:48 +0200
Labels:     name=squid
Status:     Pending
IP:
Controllers:    DaemonSet/squid
Containers:
  squid:
    Container ID:
    Image:      sameersbn/squid:3.3.8-15
    Image ID:
    Port:       3128/TCP
    QoS Tier:
      cpu:      BestEffort
      memory:       BestEffort
    State:      Waiting
      Reason:       ContainerCreating
    Ready:      False
    Restart Count:  0
    Environment Variables:
Conditions:
  Type      Status
  Initialized   True
  Ready     False
  PodScheduled  True
Volumes:
  default-token-cxdw7:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-cxdw7
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath   Type        Reason      Message
  --------- --------    -----   ----                            -------------   --------    ------      -------
  5h        3s      1954    {kubelet ip-172-20-119-184.us-west-1.compute.internal}          Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: no available IPv4 addresses on this network's address pools: bridge (b355fd5c937ecebce956807ba588550f33589ff04ff2458f5593f2caa03af76a)"

3x squid and 3x reverse proxy, but only 2 minions and 1 master.

% kubectl get nodes                                                                                                                                                                                                        NAME                                           STATUS                     AGE
ip-172-20-110-246.us-west-1.compute.internal   Ready                      5h
ip-172-20-119-184.us-west-1.compute.internal   Ready,SchedulingDisabled   5h
ip-172-20-91-250.us-west-1.compute.internal    Ready                      5h

@davidopp
Copy link
Member

This is working as intended, see #28356

Regular pods will not schedule on master but DaemonSet pods do. We'll make this behavior selectable at the pod level soon (probably not for 1.4 though). For now, you need to not have the master node's kubelet register with the master if you don't want pods to schedule there. See the issue I mentioned for more details.

cc/ @mikedanese

@lukaszo lukaszo mentioned this issue Jul 19, 2016
Closed
@whitlockjc whitlockjc mentioned this issue Jul 20, 2016
Closed
@bgrant0607 bgrant0607 mentioned this issue Mar 21, 2017
Closed
@xiangpengzhao
Copy link
Contributor

DaemonSets now already respects taints and tolerations. You can register a taint for your master with a kubelet flag --register-with-taints=node.alpha.kubernetes.io/ismaster=:NoSchedule and don't add toleration to your daemonsets to make it not scheduled on master.

@xiangpengzhao
Copy link
Contributor

I think it's okay to close this issue now.

/cc @davidopp @thockin

@thockin thockin closed this as completed May 28, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidopp davidopp

Labels
sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@noose @apelisse @thockin @davidopp @xiangpengzhao