Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix a scheduler preemption issue where the victim isn't properly patched, leading to preemption not functioning as expected #126644

Merged
merged 1 commit into from
Aug 14, 2024
Merged

Fix a scheduler preemption issue where the victim isn't properly patched, leading to preemption not functioning as expected #126644

merged 1 commit into from
Aug 14, 2024

Conversation

Huang-Wei
Copy link
Member

@Huang-Wei Huang-Wei commented Aug 12, 2024 •
edited by liggitt
Loading

What type of PR is this?

/kind bug
/kind regression

What this PR does / why we need it:

Pod's status was incorrectly patched, which blocks the further deletion, and hence preemption doesn't work. It's a typo regression introduced in v1.29 in #121103.

Which issue(s) this PR fixes:

Reported by #126643

Special notes for your reviewer:

I didn't include test as 1) it's an obvious typo, and 2) in UT and integration test we don't have enforced API validation check, and hence victim would be deleted immediately after being patched, which makes it hard to verify the status.

Does this PR introduce a user-facing change?

Fix a 1.29 scheduler preemption regression where the victim pod was not deleted due to incorrect status patching. This issue occurred when the preemptor and victim pods had different QoS classes in their status, causing the preemption to fail entirely.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. kind/regression Categorizes issue or PR as related to a regression from a prior release. labels Aug 12, 2024
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.31 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.31.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Mon Aug 12 17:37:05 UTC 2024.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 12, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Aug 12, 2024
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 12, 2024
@Huang-Wei
Copy link
Member Author

cc @alculquicondor @mimowo

xiazhan
xiazhan approved these changes Aug 12, 2024
View reviewed changes
Copy link

@xiazhan xiazhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for quick fix.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei, xiazhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mimowo
Copy link
Contributor

mimowo commented Aug 13, 2024

/lgtm
Thank you for the fix!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 13, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 58e34b39a055e45292ac3fd14cb9a6ceff9a0fcd

@alculquicondor
Copy link
Member

Are bugfixes candidates for test freeze?

@alculquicondor
Copy link
Member

I suppose we have to wait for the next set of patch releases.

@Huang-Wei
Copy link
Member Author

Huang-Wei commented Aug 13, 2024 •
edited
Loading

I suppose we have to wait for the next set of patch releases.

Yup, it's not a regression introduced in 1.31. Let's wait for code freeze lift-up and cherrypick it back to 1.29 to 1.31.

@k8s-ci-robot k8s-ci-robot merged commit 03e8154 into kubernetes:master Aug 14, 2024
14 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.32 milestone Aug 14, 2024
@alculquicondor
Copy link
Member

The freeze is over :)

@alculquicondor
Copy link
Member

In the release notes:

leading to preemption not functioning as expected

Can you be more specific? What is not functioning as expected? Does the preemption not occur at all, or does the status get wiped out in an unpredictable way?

@Huang-Wei Huang-Wei deleted the fix-preemption branch August 14, 2024 18:26
@Huang-Wei
Copy link
Member Author

Can you be more specific? What is not functioning as expected? Does the preemption not occur at all, or does the status get wiped out in an unpredictable way?

It's preemption not occur at all as the faulty patch operation would abort the whole scheduling cycle to return Error. Reworded the release notes.

@alculquicondor
Copy link
Member

That sounds like a major problem, can you prepare cherry-picks?

@alculquicondor alculquicondor mentioned this pull request Aug 14, 2024
Closed
This was referenced Aug 14, 2024
Merged
Merged
Merged
@alculquicondor
Copy link
Member

We also need a fix for 1.28 https://github.com/kubernetes/kubernetes/blob/release-1.28/pkg/scheduler/framework/preemption/preemption.go#L365, which hasn't reached EoL, and was broken by #121379

@Huang-Wei
Copy link
Member Author

That sounds like a major problem, can you prepare cherry-picks?

Yup, creating now.

@Huang-Wei
Copy link
Member Author

We also need a fix for 1.28 release-1.28/pkg/scheduler/framework/preemption/preemption.go#L365, which hasn't reached EoL, and was broken by #121379

Oops, let me create one for 1.28.

@Huang-Wei Huang-Wei mentioned this pull request Aug 14, 2024
Merged
k8s-ci-robot added a commit that referenced this pull request Aug 15, 2024
@k8s-ci-robot
Merge pull request #126691 from Huang-Wei/automated-cherry-pick-of-#1…
60a402c 
…26644-upstream-release-1.31

Automated cherry pick of #126644: fix a scheduler preemption issue that victim is not patched
k8s-ci-robot added a commit that referenced this pull request Aug 15, 2024
@k8s-ci-robot
Merge pull request #126693 from Huang-Wei/automated-cherry-pick-of-#1…
cd0ea55 
…26644-upstream-release-1.30

Automated cherry pick of #126644: fix a scheduler preemption issue that victim is not patched
k8s-ci-robot added a commit that referenced this pull request Aug 15, 2024
@k8s-ci-robot
Merge pull request #126694 from Huang-Wei/automated-cherry-pick-of-#1…
32f2b29 
…26644-upstream-release-1.29

Automated cherry pick of #126644: fix a scheduler preemption issue that victim is not patched
k8s-ci-robot added a commit that referenced this pull request Aug 15, 2024
@k8s-ci-robot
Merge pull request #126695 from Huang-Wei/automated-cherry-pick-of-#1…
9f79836 
…26644-upstream-release-1.28

Automated cherry pick of #126644: fix a scheduler preemption issue that victim is not patched
This was referenced Aug 15, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiazhan xiazhan xiazhan approved these changes

@damemi damemi Awaiting requested review from damemi

@denkensk denkensk Awaiting requested review from denkensk

Assignees

@mimowo mimowo

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/regression Categorizes issue or PR as related to a regression from a prior release. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.32
Development

Successfully merging this pull request may close these issues.
5 participants
@Huang-Wei @k8s-ci-robot @mimowo @alculquicondor @xiazhan