Implement taints and tolerations #24134

kevin-wangzefeng · 2016-04-12T13:40:49Z

taint & toleration api types
add taint to node.Annotations for alpha version
add toleration to pod.Annotations for alpha version
predicate algorithm for taint-toleration matching
priority algorithm for taint-toleration matching
add new kubectl command for applying taints to node
dedicated nodes implementation (will in a separate PR): Implement an admission controller that adds tolerations to pods that are supposed to be allowed to use dedicated nodes (for example, based on pod's namespace).

P.S. kubelet eviction behaviour will not be included in this PR

davidopp · 2016-04-17T21:13:54Z

pkg/api/types.go

+type Taint struct {
+	// Required. The taint key to be applied to a node .
+	Key string `json:"key" patchStrategy:"merge" patchMergeKey:"key"`
+	// Required. Value is the taint value the toleration matches to.


s/Value is the taint value the toleration matches to/The taint value corresponding to the taint key./

davidopp · 2016-04-17T22:04:55Z

Oh, I just saw your checklist explicitly mentioned "add taint to node.Annotations for alpha version,
add toleration to pod.Annotations for alpha version" so I didn't need to mention that in my review. :-)

Anyway, you're definitely on the right track; my comments are pretty minor. Let me know when you have a new commit for me to look at. Thanks again for working on this!

davidopp · 2016-04-17T22:31:02Z

BTW it would be great if you can also do the "dedicated nodes" implementation (that uses taints and tolerations) for 1.3. It should go in a different PR, of course. The design doc describes this as "Implement an admission controller that adds tolerations to pods that are supposed to be allowed to use dedicated nodes (for example, based on pod's namespace)."

kevin-wangzefeng · 2016-04-18T06:51:04Z

Thanks very much for your review, will address the comments in the following commits. :-)

BTW it would be great if you can also do the "dedicated nodes" implementation (that uses taints and tolerations) for 1.3. It should go in a different PR, of course. The design doc describes this as "Implement an admission controller that adds tolerations to pods that are supposed to be allowed to use dedicated nodes (for example, based on pod's namespace)."

Yes, my plan is to implement "dedicated nodes" after the first steps become solid. I also refreshed the PR description just to make a note.

k8s-github-robot · 2016-05-16T14:56:28Z

@kevin-wangzefeng
You must link to the test flake issue which caused you to request this manual re-test.
Re-test requests should be in the form of: k8s-bot test this issue: #<number>
Here is the list of open test flakes.

davidopp · 2016-05-16T22:59:15Z

@k8sbot e2e test this issue: #IGNORE

kevin-wangzefeng · 2016-05-17T01:25:31Z

@k8s-bot test this please, issue #22655

aveshagarwal · 2016-05-17T02:31:39Z

docs/man/man1/kubectl-taint.1

+.nf
+# Update node 'foo' with a taint with key 'dedicated' and value 'special\-user' and effect 'NoScheduleNoAdmitNoExecute'.
+# If a taint with that key already exists, its value and effect are replaced as specified.
+kubectl taint nodes foo dedicated=special\-user:NoScheduleNoAdmitNoExecute


NoScheduleNoAdmitNoExecute is not implemented yet so shouldn't be used here.

Thanks for reminding, should be "NoSchedule" here.

kevin-wangzefeng · 2016-05-17T05:54:39Z

@davidopp refreshed, PTAL

davidopp · 2016-05-17T15:57:40Z

test/e2e/scheduler_predicates.go

+
+		By("removing the taint " + taintName + " off the node " + nodeName)
+		framework.RunKubectlOrDie("taint", "nodes", nodeName, taintName+"-")
+		By("verifying the node doesn't have the taint " + taintName)


Add one more step, where you verify that the pod gets schedule after you remove the taint.

Sorry I was not clear what I meant. What I meant was: "add one more step, when you verify that the pod that was pending gets scheduled after you remove the taint. So do not create another pod. When you remove the taint from the node, the same pod that was pending, should now get scheduled.

Please do that in a follow-up PR; I am going to LGTM this PR now so we can make sure it gets merged in time.

davidopp · 2016-05-17T16:00:20Z

I just had one small comment on the e2e test. Once you implement that, please squash the commits and then I will add LGTM label. Thanks!!

k8s-bot · 2016-05-18T15:44:30Z

GCE e2e build/test passed for commit 52fb89f.

kevin-wangzefeng · 2016-05-18T15:53:57Z

@davidopp refreshed, PTAL

davidopp · 2016-05-18T22:36:18Z

LGTM

I had one more comment on the e2e test, but you can address it in a follow-up PR.

davidopp · 2016-05-18T22:37:00Z

BTW I do not see any label complaining about the docs, so I think this can merge via submit queue instead of manually.

davidopp · 2016-05-18T23:18:10Z

Hmm, it does have the kind/old-docs label (though not the do-not-merge) label. I'm a little worried submit queue might not take it, due to old-docs label, and you did update docs/user-guide. So I'm going to manually merge.

…s-e2e Automatic merge from submit-queue Remove redundant pod deletion in scheduler predicates tests and fix taints-tolerations e2e ~~In scheduler predicates test, some tests won't clean pods they created when exit with failure, which may lead to pod leak. This PR is to fix it.~~ Remove redundant pod deletion in scheduler predicates tests, since framework.AfterEach() already did the cleanup work after every test. Also fix the test "validates that taints-tolerations is respected if not matching", refer to the change on taint-toleration test in #29003, and #24134 (comment).

googlebot added the cla: yes label Apr 12, 2016

k8s-github-robot assigned thockin Apr 12, 2016

kevin-wangzefeng assigned davidopp and unassigned thockin Apr 12, 2016

kevin-wangzefeng force-pushed the taints-tolerations branch from 7307a66 to 09e0a46 Compare April 12, 2016 14:00

k8s-github-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Apr 12, 2016

kevin-wangzefeng force-pushed the taints-tolerations branch from 09e0a46 to bfa072c Compare April 13, 2016 02:02

davidopp reviewed Apr 17, 2016
View reviewed changes

bgrant0607 mentioned this pull request Apr 20, 2016

Add support for gating node creation on TPM-based remote attestation #23834

Closed

kevin-wangzefeng force-pushed the taints-tolerations branch from b88bd84 to c3e99d9 Compare May 16, 2016 13:55

davidopp mentioned this pull request May 16, 2016

Add support for admission controller based on namespace node selectors. #24980

Merged

aveshagarwal reviewed May 17, 2016
View reviewed changes

kevin-wangzefeng force-pushed the taints-tolerations branch from c3e99d9 to fcaa3b2 Compare May 17, 2016 03:58

kevin-wangzefeng force-pushed the taints-tolerations branch from b86c8b1 to f260673 Compare May 17, 2016 10:40

davidopp reviewed May 17, 2016
View reviewed changes

davidopp changed the title ~~[WIP] taints tolerations implementation~~ Implement taints and tolerations May 17, 2016

davidopp added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels May 17, 2016

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 18, 2016

implement taints and tolerations

52fb89f

kevin-wangzefeng force-pushed the taints-tolerations branch from f260673 to 52fb89f Compare May 18, 2016 15:08

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 18, 2016

davidopp added lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/backlog Higher priority than priority/awaiting-more-evidence. labels May 18, 2016

davidopp merged commit 35c9ca8 into kubernetes:master May 18, 2016

This was referenced Jul 19, 2016

use regular client instead of kubectl in scheduler predicate tests when checking/setting/cleanning taints/labels #29182

Merged

Remove redundant pod deletion in scheduler predicates tests and fix taints-tolerations e2e #29423

Merged

davidopp mentioned this pull request Oct 1, 2016

Taints/tolerations kubernetes/enhancements#108

Closed

23 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement taints and tolerations #24134

Implement taints and tolerations #24134

kevin-wangzefeng commented Apr 12, 2016 •

edited

Loading

davidopp Apr 17, 2016

davidopp commented Apr 17, 2016

davidopp commented Apr 17, 2016

kevin-wangzefeng commented Apr 18, 2016

k8s-github-robot commented May 16, 2016

davidopp commented May 16, 2016

kevin-wangzefeng commented May 17, 2016

aveshagarwal May 17, 2016

kevin-wangzefeng May 17, 2016

kevin-wangzefeng commented May 17, 2016

davidopp May 17, 2016

davidopp May 18, 2016

davidopp commented May 17, 2016

k8s-bot commented May 18, 2016

kevin-wangzefeng commented May 18, 2016

davidopp commented May 18, 2016

davidopp commented May 18, 2016

davidopp commented May 18, 2016

Implement taints and tolerations #24134

Implement taints and tolerations #24134

Conversation

kevin-wangzefeng commented Apr 12, 2016 • edited Loading

davidopp Apr 17, 2016

Choose a reason for hiding this comment

davidopp commented Apr 17, 2016

davidopp commented Apr 17, 2016

kevin-wangzefeng commented Apr 18, 2016

k8s-github-robot commented May 16, 2016

davidopp commented May 16, 2016

kevin-wangzefeng commented May 17, 2016

aveshagarwal May 17, 2016

Choose a reason for hiding this comment

kevin-wangzefeng May 17, 2016

Choose a reason for hiding this comment

kevin-wangzefeng commented May 17, 2016

davidopp May 17, 2016

Choose a reason for hiding this comment

davidopp May 18, 2016

Choose a reason for hiding this comment

davidopp commented May 17, 2016

k8s-bot commented May 18, 2016

kevin-wangzefeng commented May 18, 2016

davidopp commented May 18, 2016

davidopp commented May 18, 2016

davidopp commented May 18, 2016

kevin-wangzefeng commented Apr 12, 2016 •

edited

Loading