Scheduler extension #13580

ravigadde · 2015-09-04T07:09:28Z

Implementation of scheduler extension proposal as discussed in #11470

k8s-bot · 2015-09-04T07:11:05Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

ravigadde · 2015-09-04T07:18:12Z

cc @davidopp @bgrant0607

Please review when you get a chance.

k8s-github-robot · 2015-09-04T07:44:22Z

Labelling this PR as size/XL

ravigadde · 2015-09-17T18:27:42Z

@davidopp Any feedback? Can you please review after 1.1 freeze?

davidopp · 2015-09-17T19:15:21Z

I briefly took a peek at this last night and am planning to review this by end of tomorrow. Very sorry for the delay.

davidopp · 2015-09-23T09:38:12Z

Very sorry for the delay. I will review it this Friday.

karlkfi · 2015-09-23T18:51:50Z

I would like to see some kind of documentation included in this PR that describes the capability being added and how people might make use of it to externalize/extend scheduling logic.

davidopp · 2015-09-24T22:10:54Z

+1

davidopp · 2015-09-27T22:09:32Z

plugin/pkg/scheduler/api/types.go

@@ -26,6 +29,8 @@ type Policy struct {
 	Predicates []PredicatePolicy `json:"predicates"`
 	// Holds the information to configure the priority functions
 	Priorities []PriorityPolicy `json:"priorities"`
+	// Holds the information to communicate with the extender
+	ExtenderConfig *ExtenderConfig `json:"extender"`


Should we allow an array of these, so the user can configure more than one?

Makes sense, will do.

davidopp · 2015-09-27T22:16:59Z

Sorry it took me so long to take a look at this.

This looks OK to me, with all of the caveats @bgrant0607 mentioned here
#11470 (comment)

That is -- you are going to need to make significant changes to this when we start making planned scheduler changes that we just haven't had time for yet (e.g. unified extension mechanism, and changing the way we choose the "best" machine from summing up priority scores to using a decision tree). Since we also want to make improvements in scheduler performance and generally make the scheduler implementation a bit cleaner, probably we will just create a new scheduler for all of this, and allow people to continue using the old scheduler but won't do new work on it.

Of course, before submitting you need to write unit tests and either e2e or integration test.

ravigadde · 2015-09-29T01:20:26Z

@davidopp
Thanks for the review. I am working on the feedback. Wrt test case comment, is it adequate if the test cases launch a http server which supports the extensions and shut it down after they are done?

@karlkfi
I will add some documentation as well when I update the PR.

ravigadde · 2015-10-01T23:55:41Z

@davidopp Addressed review comments, PTAL. I have implemented unit test only at this point. Need to spend more time on figuring out e2e/integ test cases for this.

davidopp · 2015-10-18T02:29:10Z

docs/design/scheduler_extender.md

+
+# Scheduler extender
+
+Kubernetes scheduler schedules based on resources managed by Kubernetes. Scheduler extender is a way to extend the scheduling capability of k8s for resources that are not managed by k8s. A couple of example use cases are storage affinity and network availability.


I think this needs a bit more introduction, and actually shouldn't mention "resources that are not managed by k8s" because there are other ways to handle that (e.g. write your own scheduler). I'd just replace this paragraph with:

There are three ways to add new scheduling rules (predicates and priority functions) to Kubernetes: (1) by adding these rules to the scheduler and recompiling (described here: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/scheduler.md), (2) implementing your own scheduler process that runs instead of, or alongside of, the standard Kubernetes scheduler, (3) implementing a "scheduler extender" process that the standard Kubernetes scheduler calls out to as a final pass when making scheduling decisions. This document describes the third approach. (Note that the three approaches are not mutually exclusive.)

davidopp · 2015-11-25T09:04:34Z

LGTM

k8s-bot · 2015-11-25T09:30:02Z

GCE e2e test build/test passed for commit 31450b691048532a2001a4b278cf0eb87e311ffb.

davidopp · 2015-11-25T09:47:18Z

@ravigadde integration test is failing with

01:18:56 # k8s.io/kubernetes/test/integration
01:18:56 _output/local/go/src/k8s.io/kubernetes/test/integration/extender_test.go:228: unknown "k8s.io/kubernetes/pkg/client/unversioned".Config field 'Version' in struct literal
01:18:56 FAIL   k8s.io/kubernetes/test/integration [build failed]

Did you try running the integration tests locally? hack/test-integration.sh should do it.

ravigadde · 2015-11-25T15:45:15Z

@davidopp Let me update to latest and fix it. Looks like this commit renamed the field recently - 94ad6aa

k8s-github-robot · 2015-11-25T16:22:24Z

PR changed after LGTM, removing LGTM.

k8s-bot · 2015-11-25T16:49:52Z

GCE e2e test build/test passed for commit cadc24e.

ravigadde · 2015-11-25T17:05:48Z

@davidopp Done, all tests pass now

davidopp · 2015-11-25T19:37:35Z

LGTM

k8s-github-robot · 2015-11-25T19:54:07Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2015-11-25T20:25:55Z

GCE e2e test build/test passed for commit cadc24e.

k8s-github-robot · 2015-11-25T20:26:16Z

Automatic merge from submit-queue

Auto commit by PR queue bot

davidopp · 2015-11-25T20:31:55Z

Yay! Thanks for your patience with this, @ravigadde

ravigadde · 2015-11-25T20:36:01Z

@davidopp Thank you for your effort to make this happen

khoi0107 · 2016-11-04T23:42:13Z

@ravigadde Hi, is there sample implementation of an extender somewhere?

ravigadde · 2016-11-06T04:57:43Z

@khoi0107 I haven't seen an open implementation. @ping035627 submitted a patch sometime ago, not sure if their implementation is open. If you have any specific questions I can help with, ping me on slack - ravi.gadde

googlebot added the cla: yes label Sep 4, 2015

k8s-github-robot assigned karlkfi Sep 4, 2015

k8s-github-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Sep 4, 2015

mikedanese assigned davidopp and unassigned karlkfi Sep 4, 2015

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 10, 2015

davidopp reviewed Sep 27, 2015
View reviewed changes

davidopp mentioned this pull request Sep 28, 2015

Move scheduler fit predicates into a library #12744

Closed

ravigadde force-pushed the sched-ext branch from 6133460 to a3e9055 Compare October 1, 2015 23:52

davidopp reviewed Oct 18, 2015
View reviewed changes

davidopp added lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-merge labels Nov 25, 2015

Scheduler extension

cadc24e

ravigadde force-pushed the sched-ext branch from 31450b6 to cadc24e Compare November 25, 2015 16:19

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 25, 2015

davidopp added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 25, 2015

k8s-github-robot pushed a commit that referenced this pull request Nov 25, 2015

Merge pull request #13580 from ravigadde/sched-ext

3ffc680

Auto commit by PR queue bot

k8s-github-robot merged commit 3ffc680 into kubernetes:master Nov 25, 2015

davidopp mentioned this pull request Feb 19, 2016

Scheduler extension Proposal #11470

Closed

chakri-nelluri mentioned this pull request Mar 30, 2017

Add support for attacher/detacher interface in Flex volume #41804

Merged

bgrant0607 mentioned this pull request Jul 1, 2017

DaemonSet pods should be scheduled by default scheduler, not DaemonSet controller #42002

Closed

liwenwu-amazon mentioned this pull request Dec 23, 2017

Controller is not aware of the number of available IP addresses aws/amazon-vpc-cni-k8s#18

Closed

chadmcrowell mentioned this pull request Jul 25, 2019

Issue with k8s.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ kubernetes/website#15534

Closed

This was referenced Jun 1, 2020

Scheduler extender kubernetes/enhancements#1819

Closed

Convert scheduler extender design proposal in community repo to a KEP kubernetes/enhancements#1840

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scheduler extension #13580

Scheduler extension #13580

ravigadde commented Sep 4, 2015

k8s-bot commented Sep 4, 2015

ravigadde commented Sep 4, 2015

k8s-github-robot commented Sep 4, 2015

ravigadde commented Sep 17, 2015

davidopp commented Sep 17, 2015

davidopp commented Sep 23, 2015

karlkfi commented Sep 23, 2015

davidopp commented Sep 24, 2015

davidopp Sep 27, 2015

ravigadde Sep 29, 2015

davidopp commented Sep 27, 2015

ravigadde commented Sep 29, 2015

ravigadde commented Oct 1, 2015

davidopp Oct 18, 2015

davidopp commented Nov 25, 2015

k8s-bot commented Nov 25, 2015

davidopp commented Nov 25, 2015

ravigadde commented Nov 25, 2015

k8s-github-robot commented Nov 25, 2015

k8s-bot commented Nov 25, 2015

ravigadde commented Nov 25, 2015

davidopp commented Nov 25, 2015

k8s-github-robot commented Nov 25, 2015

k8s-bot commented Nov 25, 2015

k8s-github-robot commented Nov 25, 2015

davidopp commented Nov 25, 2015

ravigadde commented Nov 25, 2015

khoi0107 commented Nov 4, 2016

ravigadde commented Nov 6, 2016


		# Scheduler extender

		Kubernetes scheduler schedules based on resources managed by Kubernetes. Scheduler extender is a way to extend the scheduling capability of k8s for resources that are not managed by k8s. A couple of example use cases are storage affinity and network availability.

Scheduler extension #13580

Scheduler extension #13580

Conversation

ravigadde commented Sep 4, 2015

k8s-bot commented Sep 4, 2015

ravigadde commented Sep 4, 2015

k8s-github-robot commented Sep 4, 2015

ravigadde commented Sep 17, 2015

davidopp commented Sep 17, 2015

davidopp commented Sep 23, 2015

karlkfi commented Sep 23, 2015

davidopp commented Sep 24, 2015

davidopp Sep 27, 2015

Choose a reason for hiding this comment

ravigadde Sep 29, 2015

Choose a reason for hiding this comment

davidopp commented Sep 27, 2015

ravigadde commented Sep 29, 2015

ravigadde commented Oct 1, 2015

davidopp Oct 18, 2015

Choose a reason for hiding this comment

davidopp commented Nov 25, 2015

k8s-bot commented Nov 25, 2015

davidopp commented Nov 25, 2015

ravigadde commented Nov 25, 2015

k8s-github-robot commented Nov 25, 2015

k8s-bot commented Nov 25, 2015

ravigadde commented Nov 25, 2015

davidopp commented Nov 25, 2015

k8s-github-robot commented Nov 25, 2015

k8s-bot commented Nov 25, 2015

k8s-github-robot commented Nov 25, 2015

davidopp commented Nov 25, 2015

ravigadde commented Nov 25, 2015

khoi0107 commented Nov 4, 2016

ravigadde commented Nov 6, 2016