Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

taints & tolerations with daemonsets #29738

Closed
justinsb opened this issue Jul 28, 2016 · 7 comments
Closed

taints & tolerations with daemonsets #29738

justinsb opened this issue Jul 28, 2016 · 7 comments
Assignees
@lukaszo
Labels
area/workload-api/daemonset sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.

Comments

@justinsb
Copy link
Member

I'm trying to use taints and tolerations to run a daemonset only on my master nodes. The daemonset is scheduling pods on all nodes though. I don't know whether daemonsets bypass taints & tolerations, or whether I am doing something wrong.

I taint my master node:

kubectl taint  nodes ip-172-20-77-44.ec2.internal role=master:NoSchedule

My daemonset:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: aws-controller
  labels:
    tier: cloud
    app: aws-controller
    version: v1
spec:
  template:
    metadata:
      labels:
        name: aws-controller
      annotations:
        scheduler.alpha.kubernetes.io/tolerations: |
          [{"key": "role", "value": "master", "effect": "NoSchedule" }]
    spec:
      containers:
        - resources:
            requests:
              cpu: 0.20
          securityContext:
            privileged: true
          image: kope/aws-controller
          name: aws-controller
          command:
          - /usr/bin/aws-controller
          - --healthz-port=10245
          - --zone-name=yahoo.com
          - --v=4

The pod is scheduled on all my nodes though. One of their annotations (so we can see the final result):

        "annotations": {
            "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"DaemonSet\",\"namespace\":\"default\",\"name\":\"aws-controller\",\"uid\":\"47395313-54d4-11e6-9aa6-0681a2fd2e75\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"3520\"}}\n",
            "scheduler.alpha.kubernetes.io/tolerations": "[{\"key\": \"role\", \"value\": \"master\", \"effect\": \"NoSchedule\" }]\n"   
        }

And the annotations on the node:

    "annotations": {
      "scheduler.alpha.kubernetes.io/taints": "[{\"key\":\"role\",\"value\":\"master\",\"effect\":\"NoSchedule\"}]",
      "volumes.kubernetes.io/controller-managed-attach-detach": "true"
    }
@errordeveloper
Copy link
Member

I thought this was the desired functionality, i.e. you specify tolerations on some of your daemonsets that you want to run on masters. You probably want to add NodeLabels (or affinity) to express that this daemonset is supposed to run exclusively on masters.

@justinsb justinsb mentioned this issue Sep 29, 2016
Merged
@bgrant0607 bgrant0607 added sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. area/workload-api/daemonset and removed team/control-plane (deprecated - do not use) area/kubectl labels Mar 21, 2017
@bgrant0607 bgrant0607 mentioned this issue Mar 21, 2017
Closed
@0xmichalis
Copy link
Contributor

@kubernetes/sig-apps-bugs @kubernetes/sig-scheduling-bugs did we fix DSs to respect taints and tolerations or is this still an issue?

@justinsb
Copy link
Member Author

We can close this; this was implemented in 1.6

@lukaszo
Copy link
Contributor

lukaszo commented Apr 27, 2017

It was implemented in 1.5 then reverted and implemented again in 1.6 :)

@SleepyBrett
Copy link

This bit our 1.6 cluster hard. We run a self-hosted cluster and things like kube-proxy and flannel are distributed via daemonset. Suddenly every tainted node failed to come up properly because they don't have the overlay network or a kube proxy. So now instead of being able to add a new pool of tainted nodes (we do this not-infrequently) via a simple terraform apply (to spin up a new ASG of nodes) now I have to also patch every DS on my cluster.

I'd like to suggest there needs to be some kind of way to do a wildcard toleration for this scenario.

@davidopp
Copy link
Member

davidopp commented May 4, 2017 •
edited
Loading

What do you mean by wildcard toleration?
Check out the API at
https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go#L2223
and let us know if doesn't let you do what you want.

@gyliu513
Copy link
Contributor

gyliu513 commented May 5, 2017

Related to #45367

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lukaszo lukaszo

Labels
area/workload-api/daemonset sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@lukaszo @justinsb @errordeveloper @SleepyBrett @0xmichalis @gyliu513 @bgrant0607 @davidopp @k8s-github-robot