Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Priority to Kubernetes API #45610

Merged
merged 2 commits into from
Jun 28, 2017

Conversation

bsalamat
Copy link
Member

@bsalamat bsalamat commented May 10, 2017

What this PR does / why we need it: This is the first in a series of PRs to add priority to Kubernetes API. Subsequent PRs will add priority name resolution to admission controller.

Release note:

Add PriorityClassName and Priority fields to PodSpec.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 10, 2017
@k8s-github-robot k8s-github-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels May 10, 2017
@k8s-reviewable
Copy link

This change is Reviewable

@bsalamat
Copy link
Member Author

ref/ #22212

@bsalamat
Copy link
Member Author

@kubernetes/api-reviewers
@kubernetes/sig-scheduling-pr-reviews

@k8s-ci-robot k8s-ci-robot added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label May 10, 2017
@timothysc timothysc added this to the v1.7 milestone May 10, 2017
@timothysc timothysc added the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label May 10, 2017
@timothysc
Copy link
Member

Needs a release note + xref: the proposal.

@kubernetes/sig-api-machinery-feature-requests folks will need to sign off on this one.

@k8s-ci-robot k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label May 10, 2017
@bsalamat
Copy link
Member Author

ref/ kubernetes/community#604

@bsalamat bsalamat force-pushed the priority_api branch 3 times, most recently from da753ec to 8ecaab9 Compare May 11, 2017 01:39
@@ -2412,6 +2412,12 @@ type PodSpec struct {
// +patchMergeKey=IP
// +patchStrategy=merge
HostAliases []HostAlias `json:"hostMappings,omitempty" patchStrategy:"merge" patchMergeKey:"IP" protobuf:"bytes,23,rep,name=hostMappings"`
// If specified, indicates the pod's priority. "system" is a special keyword
// which indicates the highest priority. Any other name must be defined in
// Admission Controller config or pod will be rejected.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a TODO to update this with the name of the admission controller once you write it.
BTW do you think you can do the admission controller for 1.7? It shouldn't be too hard if you mostly copy one of the existing admission controllers. I don't think any of the existing admission controllers are configured by a ConfigMap, though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me call it "Priority" admission controller for now. If we decided that it is not a good name, I will change it.
And, yes, I would like to do admission controller for 1.7 as well if we reach a consensus on how priority should be defined and work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the admission controller name should be in the API - that has nothing to do with the end user's perception of the field. A custom admission controller can handle this as well as anything else.

@@ -2529,6 +2535,10 @@ type PodStatus struct {
// More info: https://github.com/kubernetes/kubernetes/blob/master/docs/design/resource-qos.md
// +optional
QOSClass PodQOSClass `json:"qosClass,omitempty" protobuf:"bytes,9,rep,name=qosClass"`
// The priority value which is resolved by the Admission Controller from
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing here, update with actual name of the admission controller once you write it.

@@ -2024,6 +2024,14 @@ func ValidateHostAliases(hostAliases []api.HostAlias, fldPath *field.Path) field
return allErrs
}

func ValidatePriorityName(priorityName string, fldPath *field.Path) field.ErrorList {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of comments about validation

(1) You should also do validation on Priority (not just PriorityName) even though it is set by an admission controller (user might write a buggy custom admission controller). BTW do we want to allow negative priorities?

(2) I think you need to modify ValidatePodUpdate() to make it possible to update PriorityName (and Priority)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Actually any valid integer is a valid priority, even negative numbers. I am not sure if we can do any particular validation for "Priority".

  2. I was not so sure whether we want to allow updating priority or not. I am going to modify ValidatePodUpdate() so that it allows change of priority with the assumption that all updates go through admission checks.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't allow priorityName to be updated unless you are absolutely sure you are going to use it in 1.7. If you aren't, leave it create only. We can never tighten validation.

pkg/api/types.go Outdated
// If specified, indicates the pod's priority. "system" is a special keyword
// which indicates the highest priority. Any other name must be defined in
// Admission Controller config or pod will be rejected.
// If not specified, the pod priority will be zero.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

zero is the lowest?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, priority could potentially be negative. Zero is default.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will admission controller set it to lowest priority? The concern is that: if not specified, there maybe an undefined priority in system, e.g. zero is not a defined priority.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the proposal a few minutes ago and added more details about the default name resolution. We probably go with resolving unspecified PriorityClassName to zero in the initial version, but in a follow up change, we will add the support to annotate one of the PriorityClasses with a "default" annotation. Once such a priority class exists, the system resolves unspecified PriorityClassName to the value of the default class. If there is no such a class, we will still resolve to zero. Zero is not the lowest possible priority in the system given that any valid integer is a valid priority.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use CamelCase for all constants, if this is a "opaque constant field". If this is a reference to a priority class API object, the field name should be PriorityClassName.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bsalamat , "default" annotation makes sense to me :).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smarterclayton This is a reference to a priority class object. The field is already named PriorityClassName.

@k82cn that's the plan.

@liggitt
Copy link
Member

liggitt commented May 11, 2017

who is allowed to set and update a pod's priority?

@timothysc
Copy link
Member

@liggitt I'd expect the user could set a value, but it may get scaled by an admission controller.

@liggitt
Copy link
Member

liggitt commented May 11, 2017

Is this proposing that the same priorityName would resolve to different numerical priorities for different users? e.g. I (as an end-user) submit a pod with "priorityName: system" and get priority of 0, and you (as a superuser) submit the same pod and get a priority of 1000?

Or would I be prevented from creating a pod with priorityName: system at all?

@timothysc
Copy link
Member

Is this proposing that the same priorityName would resolve to different numerical priorities for different users? e.g. I (as an end-user) submit a pod with "priorityName: system" and get priority of 0, and you (as a superuser) submit the same pod and get a priority of 1000?

Not yet, but I did have that intention in the long run.

Or would I be prevented from creating a pod with priorityName: system at all?

I hope we would lock down the highest priority to admin only.

@liggitt
Copy link
Member

liggitt commented May 11, 2017

Or would I be prevented from creating a pod with priorityName: system at all?

I hope we would lock down the highest priority to admin only.

Lock down by rejecting the pod, or by translating the system priority name to a lower numeric priority? I want to make sure we have an idea how we would do this (and how we would decide whether pods created by controllers should be allowed to have system priority). That seems like an important flow to nail down before making API changes.

@bsalamat
Copy link
Member Author

bsalamat commented May 11, 2017

@liggitt Priority is meant to be global, not per user or namespace. Users can specify PriorityName is their podspec. system is a special priority for system pods. I agree that we should limit it to admin so that only admin can set it. Pods by other users who put system in their pod spec will be rejected. I will add this to the design proposal.
Your suggestion does not apply to this PR, as it only adds the fields to PodSpec and PodStatus with almost no enforcement logic.

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 13, 2017
pkg/api/types.go Outdated
// If specified, indicates the pod's priority. "SYSTEM" is a special keyword
// which indicates the highest priority. Any other name must be defined by
// creating a PriorityClass object with that name.
// If not specified, the pod priority will will be default or zero if there is
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/will will/will

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks!

pkg/api/types.go Outdated
// If specified, indicates the pod's priority. "SYSTEM" is a special keyword
// which indicates the highest priority. Any other name must be defined by
// creating a PriorityClass object with that name.
// If not specified, the pod priority will will be default or zero if there is
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will be default or zero if there is no default.

Are we going to support default now? I found that in the design document, it is will, just want to confirm if the default will be available in the first implementation? If not, how about add some words such as will to identify it is future work?

Copy link
Member Author

@bsalamat bsalamat Jun 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is to have the default in the first version. We will definitely have the ability to define a "global" default priority. In the future versions, we may add per namespace default as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, then how about update the sentence default or zero if there is no default a bit?

Such as when using default one and when setting to zero? Using default or zero if there is no default may make people confused, as they may not clear when using default and when using zero?

Copy link
Member

@k82cn k82cn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, it's LGTM :); just some comments on cleanup.

@@ -2520,6 +2520,17 @@ type PodSpec struct {
// +patchMergeKey=IP
// +patchStrategy=merge
HostAliases []HostAlias `json:"hostMappings,omitempty" patchStrategy:"merge" patchMergeKey:"IP" protobuf:"bytes,23,rep,name=hostMappings"`
// If specified, indicates the pod's priority. "SYSTEM" is a special keyword
// which indicates the highest priority. Any other name must be defined by a
// PriorityClass object with that name.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

different comments with api/types.go, should be by creating a PriorityClass.

@@ -2272,6 +2276,20 @@ func ValidatePodSpec(spec *api.PodSpec, fldPath *field.Path) field.ErrorList {
allErrs = append(allErrs, ValidateHostAliases(spec.HostAliases, fldPath.Child("hostAliases"))...)
}

if len(spec.PriorityClassName) > 0 {
if !utilfeature.DefaultFeatureGate.Enabled(features.PodPriority) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

personally, I'd like the following if/else:

if utilfeature.DefaultFeatureGate.Enabled(features.PodPriority) {
  ...
} else {
  ...
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my version of the code, the feature gate is not checked when priority class is not specified. Also, the condition "if len(spec.PriorityClassName) > 0" should be repeated in both if and else blocks of your version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes; my version is for sometimes removing this feature gate (just remove the else block is OK); both are ok to me :).

@k82cn
Copy link
Member

k82cn commented Jun 17, 2017

btw, please run ./hack/update-staging-client-go.sh for client-go.

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 20, 2017
@gyliu513
Copy link
Contributor

@bsalamat how about run ./hack/update-all.sh -a? This can help generate code and update client-go, also have some CLI interaction to enable us to submit different commits.

@bsalamat
Copy link
Member Author

@gyliu513 I have already. Did you see files which are not updated?

@bsalamat bsalamat force-pushed the priority_api branch 2 times, most recently from 0ccf575 to 94f9147 Compare June 21, 2017 23:55
@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 22, 2017
@gyliu513
Copy link
Contributor

@bsalamat I found that dd priority to Kubernetes API also include some generated code, and also seems your patch do not updated the code for client-go. This seems strange to me.

But if you use ./hack/update-all.sh -a, you can first get the generated code for a commit and then another client-go update for another commit.

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 22, 2017
@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 26, 2017
@bsalamat
Copy link
Member Author

/test pull-kubernetes-unit

@bsalamat
Copy link
Member Author

/test pull-kubernetes-unit
/test pull-kubernetes-e2e-gce-etcd3

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jun 26, 2017

@bsalamat: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
Jenkins GCE etcd3 e2e f1b185964824afb831cd875295e36e8322dd208f link @k8s-bot gce etcd3 e2e test this
Jenkins kops AWS e2e f1b185964824afb831cd875295e36e8322dd208f link @k8s-bot kops aws e2e test this
Jenkins Kubemark GCE e2e f1b185964824afb831cd875295e36e8322dd208f link @k8s-bot kubemark e2e test this

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@davidopp
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 27, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bsalamat, davidopp
We suggest the following additional approver: smarterclayton

Assign the PR to them by writing /assign @smarterclayton in a comment when ready.

No associated issue. Update pull-request body to add a reference to an issue, or get approval with /approve no-issue

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@bsalamat
Copy link
Member Author

/test pull-kubernetes-unit

@bsalamat
Copy link
Member Author

@smarterclayton Could you please take another look?

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 45610, 47628)

@k8s-github-robot k8s-github-robot merged commit 82eff38 into kubernetes:master Jun 28, 2017
@bsalamat bsalamat deleted the priority_api branch September 15, 2017 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.