Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configurable HorizontalPodAutoscaler #74525

Merged
merged 5 commits into from
Dec 11, 2019
Merged

Configurable HorizontalPodAutoscaler #74525

merged 5 commits into from
Dec 11, 2019

Conversation

gliush
Copy link
Contributor

@gliush gliush commented Feb 25, 2019 •
edited
Loading

I introduce an algorithm-agnostic HPA object configuration that will configure each particular HPA scaling behavior.

KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-autoscaling/20190307-configurable-scale-velocity-for-hpa.md

This PR contains only API changes for now. The business logic will be introduced in a separate PR. For now I keep all the changes in my local repo until the current PR is approved:
gliush#2

What type of PR is this?
/kind feature

What this PR does / why we need it:
Different applications may have different business values, different logic and may require different scaling behaviors.
At the moment, there’s only one cluster-level configuration parameter that influence how fast the cluster is scaled down:
--horizontal-pod-autoscaler-downscale-stabilization-window (default to 5 min)

And a couple of hard-coded constants that specify how fast the cluster can scale up:

This PR introduces an algorithm-agnostic HPA object configuration that will configure each particular HPA scaling behavior.
For both directions (scale up and scale down) it will be possible to specify one or several behavior that will control how the HPA controller will scale the resources. Effectively each behavior is a specification of "how many percents/pods could be added/removed per some period of time".
Additionally, each direction might have a StabilizationWindowSeconds parameter set to gather recommendations for some time and pick the safest change.

For more information and motivation read the KEPs (links are above).

Which issue(s) this PR fixes:

Fixes #39090
Fixes #65097
Fixes #69428

It covers partly #56335

Special notes for your reviewer:
The API changes are backward compatible. All current defaults are kept the same.

Does this PR introduce a user-facing change?:

Added the HPA API, that allows scale behavior to be configured through the HPA
`behavior` field. Behaviors are specified separately for scaling up and down. In
each direction a stabilization window can be specified as well as a list of
policies and how to select amongst them. Policies can limit the absolute number
of pods added or removed, or the percentage of pods added or removed.

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. kind/feature Categorizes issue or PR as related to a new feature. labels Feb 25, 2019
@k8s-ci-robot
Copy link
Contributor

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Feb 25, 2019
@k8s-ci-robot
Copy link
Contributor

Hi @gliush. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 25, 2019
@gliush
Copy link
Contributor Author

gliush commented Feb 25, 2019

/sig autoscaling

@k8s-ci-robot k8s-ci-robot added the sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. label Feb 25, 2019
@k8s-ci-robot k8s-ci-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 25, 2019
@gliush
Copy link
Contributor Author

gliush commented Feb 25, 2019

@thockin: Could you have a look? @mwielgus says that you're the most familiar with the matter.
I can't do it by myself, sorry.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Feb 25, 2019
@gliush gliush mentioned this pull request Feb 25, 2019
Closed
@mwielgus mwielgus mentioned this pull request Feb 28, 2019
Closed
@mwielgus mwielgus mentioned this pull request Feb 28, 2019
Closed
@thockin
Copy link
Member

thockin commented Mar 1, 2019

Is there a KEP for this? Generally we want to see the design work independent of the code.

I am happy to take up the API review, but can't really do that until I know the domain experts (e.g. @mwielgus) are happy with the design.

@thockin
Copy link
Member

thockin commented Mar 1, 2019

I see there's a doc but that's not a KEP

@thockin
Copy link
Member

thockin commented Mar 1, 2019 via email

@mwielgus
Copy link
Contributor

mwielgus commented Mar 1, 2019

@thockin - I have worked with @gliush on this for a while. We agreed on the rough idea and the configuration parameters, so we are proceeding with the api.

This functionality has been requested multiple times already so I'm glad that we finally have someone who is working on it :).

@arjunrn arjunrn force-pushed the configurable-hpa-limits branch from 3ababc8 to 8b159e8 Compare December 9, 2019 07:06
@josephburnett
Copy link
Member

@arjunrn thanks for fixing the controller-manager crash! I've done some manual testing in a cluster and it works nicely. Added and removed behaviors which are respected.

Just two requests:

  1. Please squash this PR before I give an LGTM. There are 21 commits and this repo doesn't squash for you!
  2. Please provide a pointer to the E2E test for this. @arjunrn I think you have one already. But I want to be sure we have something that will guard against a bug like the controller-manager crashing.

@gliush gliush force-pushed the configurable-hpa-limits branch from 8b159e8 to bad0417 Compare December 10, 2019 16:42
@arjunrn
Adds tests
8ab2262 
Signed-off-by: Arjun Naik <arjun@arjunnaik.in>
@arjunrn arjunrn force-pushed the configurable-hpa-limits branch from bad0417 to 8ab2262 Compare December 10, 2019 17:09
@josephburnett
Copy link
Member

/lgtm

I checked and there is no difference between the latest squash and the unsquashed changes I tested. Nice work!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 11, 2019
@josephburnett
Copy link
Member

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 11, 2019
@arjunrn
Copy link
Contributor

arjunrn commented Dec 11, 2019

/test pull-kubernetes-kubemark-e2e-gce-big

@k8s-ci-robot k8s-ci-robot merged commit 928817a into kubernetes:master Dec 11, 2019
@k8s-ci-robot k8s-ci-robot added this to the v1.18 milestone Dec 11, 2019
@arjunrn arjunrn mentioned this pull request Dec 15, 2019
Merged
@saschagrunert
Copy link
Member

Hey @gliush 👋, Im currently looking at the latest generated releases notes and saw that this note needs to either state the user facing change or set it to NONE. Do you think you can change the PR description for that? The next re-generation of the notes will fix that issue on our side later on. :)

@saschagrunert saschagrunert mentioned this pull request Jan 6, 2020
Merged
@gliush
Copy link
Contributor Author

gliush commented Jan 7, 2020

@saschagrunert : I've updated the release notes, could you check, please? I'm a little bit concerned about the style and the amount of details.

@saschagrunert
Copy link
Member

Thank you @gliush 🙏 I assume the API is new and we should state that to the user as well, like:

Added the HPA API, which allows scale ...

So in general it is looking good, but we have include information like mentioned here: https://github.com/saschagrunert/community/blob/master/contributors/guide/release-notes.md#contents-of-a-release-note

@gliush
Copy link
Contributor Author

gliush commented Jan 7, 2020

@saschagrunert : thank you! Done!

@saschagrunert
Copy link
Member

Wonderful, thank you again! :)

@arjunrn arjunrn mentioned this pull request Feb 3, 2020
Merged
@DirectXMan12 DirectXMan12 mentioned this pull request Apr 20, 2020
Open
wking pushed a commit to wking/kubernetes that referenced this pull request Jul 21, 2020
@k8s-publishing-bot
Merge pull request kubernetes#74525 from gliush/configurable-hpa-limits
a20db94 
Configurable HorizontalPodAutoscaler

Kubernetes-commit: 928817a
@silenceper silenceper mentioned this pull request Sep 16, 2020
Closed
@gjtempleton gjtempleton mentioned this pull request Nov 20, 2020
Merged
@Ritikaa96 Ritikaa96 mentioned this pull request Jul 20, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mwielgus mwielgus mwielgus left review comments

@RainbowMango RainbowMango RainbowMango left review comments

@thockin thockin thockin left review comments

@josephburnett josephburnett josephburnett approved these changes

@robstrong robstrong robstrong left review comments

@arjunrn arjunrn arjunrn left review comments

@tedyu tedyu tedyu left review comments

@php-coder php-coder php-coder left review comments

@erictune erictune Awaiting requested review from erictune

@sttts sttts Awaiting requested review from sttts

@vllry vllry Awaiting requested review from vllry

Assignees

@josephburnett josephburnett

@thockin thockin

@mwielgus mwielgus

Labels
api-review Categorizes an issue or PR as actively needing an API review. approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubectl cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. sig/cli Categorizes an issue or PR as relevant to SIG CLI. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
API Reviews
Status: API review completed, 1.17
Milestone
v1.18