Do not count failed pods as unready in HPA controller #60648
Conversation
Currently, when performing a scale up, any failed pods (which can be present for example in case of evictions performed by kubelet) will be treated as unready. Unready pods are treated as if they had 0% utilization which will slow down or even block scale up. After this change, failed pods are ignored in all calculations. This way they do not influence neither scale up nor scale down replica calculations.
k8s-ci-robot
added
release-note-none
|
This should probably have a release note filled out, because it's a change in behavior. |
k8s-ci-robot
added
release-note
|
Fair point, added. |
|
@DirectXMan12 Since the bug seems to be quite an inconvenience (the only workaround I know of is to manually remove the evicted pods and since at least 1.7.5 the evicted pods seem to stay around for a fairly long amount of time - #55051 (comment)) do you think this could go into 1.10? |
|
yeah, I'll add it to the milestone. This seems like it could prevent the HPA from working at all, which makes it a decently bad bug. |
|
/kind bug |
k8s-ci-robot
added
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bskiba, DirectXMan12 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
/sig autoscaling |
k8s-ci-robot
added
sig/autoscaling
k8s-github-robot
added
milestone/needs-approval
and removed
milestone/incomplete-labels
labels
|
/status approved-for-milestone |
|
[MILESTONENOTIFIER] Milestone Pull Request: Up-to-date for process Pull Request Labels
|
|
Automatic merge from submit-queue (batch tested with PRs 60732, 60689, 60648, 60704). If you want to cherry-pick this change to another branch, please follow the instructions here. |
What this PR does / why we need it:
Currently, when performing a scale up, any failed pods (which can be present for example in case of evictions performed by kubelet) will be treated as unready. Unready pods are treated as if they had 0% utilization which will slow down or even block scale up.
After this change, failed pods are ignored in all calculations. This way they do not influence neither scale up nor scale down replica calculations.
@MaciekPytel @DirectXMan12
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #55630
Special notes for your reviewer:
Release note: