scheduler volumebinding: leverage PreFilterResult for bound local PVs #109877

yibozhuang · 2022-05-07T00:09:02Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This change will leverage the new PreFilterResult
to reduce down the list of eligible nodes for pod
using Bound Local PVs during PreFilter stage so
that only the node(s) which local PV node affinity
matches will be considered in subsequent scheduling
stages.

Today, the NodeAffinity check is done during Filter
which means all nodes will be considered even though
there may be a large number of nodes that are not
eligible due to not matching the pod's bound local
PV(s)' node affinity requirement. Here we can
reduce down the node list in PreFilter to ensure that
during Filter we are only considering the reduced
list and thus can provide a more clear message to
users when node(s) are not available for scheduling
since the list only contains relevant nodes.

If error is encountered (e.g. PV cache read error) or
if node list reduction cannot be done (e.g. pod uses
no local PVs), then we will still proceed to consider
all nodes for the rest of scheduling stages.

Signed-off-by: Yibo Zhuang yibzhuang@gmail.com

Testing example:

100 nodes cluster
all 100 nodes tainted with different key, value
pod has a bound PVC to local PV with node affinity on node fake-0

without PreFilter change, scheduling output

  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  3s    default-scheduler  0/100 nodes are available: 1 node(s) had untolerated taint {key-fake-0: value-fake-0}, 1 node(s) had untolerated taint {key-fake-10: value-fake-10}, 1 node(s) had untolerated taint {key-fake-11: value-fake-11}, 1 node(s) had untolerated taint {key-fake-12: value-fake-12}, 1 node(s) had untolerated taint {key-fake-13: value-fake-13}, 1 node(s) had untolerated taint {key-fake-14: value-fake-14}, 1 node(s) had untolerated taint {key-fake-15: value-fake-15}, 1 node(s) had untolerated taint {key-fake-16: value-fake-16}, 1 node(s) had untolerated taint {key-fake-17: value-fake-17}, 1 node(s) had untolerated taint {key-fake-18: value-fake-18}, 1 node(s) had untolerated taint {key-fake-19: value-fake-19}, 1 node(s) had untolerated taint {key-fake-1: value-fake-1}, 1 node(s) had untolerated taint {key-fake-20: value-fake-20}, 1 node(s) had untolerated taint {key-fake-21: value-fake-21}, 1 node(s) had untolerated taint {key-fake-22: value-fake-22}, 1 node(s) had untolerated taint {key-fake-23: value-fake-23}, 1 nod ...

with PreFilter change, scheduling output

  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  1s    default-scheduler  0/100 nodes are available: 1 node(s) had untolerated taint {key-fake-0: value-fake-0}. preemption: 0/100 nodes are available: 1 Preemption is not helpful for scheduling, 99 No preemption victims found for incoming pod.

Related to #109476

Which issue(s) this PR fixes:

Fixes #109876

Special notes for your reviewer:

Does this PR introduce a user-facing change?

scheduler volumebinding: leverage PreFilterResult to reduce down to only eligible node(s) for pod with bound claim(s) to local PersistentVolume(s)

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2022-05-07T00:09:09Z

@yibozhuang: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2022-05-07T00:09:10Z

Hi @yibozhuang. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yibozhuang · 2022-05-07T00:10:04Z

/assign @Huang-Wei @cofyc @msau42

Huang-Wei

will review tests in the 2nd round review.

pkg/volume/util/util.go

pkg/scheduler/framework/plugins/volumebinding/binder.go

pkg/scheduler/framework/plugins/volumebinding/volume_binding.go

pkg/volume/util/util.go

Huang-Wei · 2022-05-10T00:41:38Z

/ok-to-test

yibozhuang · 2022-05-10T20:58:51Z

/retest

yibozhuang · 2022-11-18T16:44:29Z

/retest

Huang-Wei · 2022-11-18T19:14:10Z

/approve for scheduling

Leave review for the storage part to @msau42 .

yibozhuang · 2022-12-02T04:51:48Z

@msau42 will you have cycles to do another review on this? Thanks in advance

dims · 2022-12-12T15:34:02Z

If you still need this PR then please rebase, if not, please close the PR

msau42 · 2022-12-13T00:28:17Z

sorry for the delay, will try to take a look soon

msau42 · 2022-12-13T21:50:53Z

/lgtm
/approve

Thanks!

k8s-ci-robot · 2022-12-13T21:51:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei, msau42, yibozhuang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/scheduler/framework/plugins/volumebinding/OWNERS~~ [Huang-Wei,msau42]
~~pkg/volume/OWNERS~~ [msau42]
~~test/OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label May 7, 2022

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 7, 2022

k8s-ci-robot assigned cofyc, Huang-Wei and msau42 May 7, 2022

k8s-ci-robot requested review from jingxu97 and mattcary May 7, 2022 00:11

k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 7, 2022

yibozhuang force-pushed the local-pv-prefilter-result branch from 0040f87 to f2e2263 Compare May 7, 2022 00:12

yibozhuang changed the title ~~scheduler volumebinding: leverage PreFilterResult~~ scheduler volumebinding: leverage PreFilterResult for bound local PVs May 7, 2022

Huang-Wei reviewed May 9, 2022

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 10, 2022

yibozhuang force-pushed the local-pv-prefilter-result branch 2 times, most recently from 61d6020 to 33b237f Compare May 10, 2022 18:43

yibozhuang force-pushed the local-pv-prefilter-result branch from 33b237f to 2f64fc7 Compare May 11, 2022 00:23

k8s-ci-robot added the area/test label May 11, 2022

yibozhuang force-pushed the local-pv-prefilter-result branch from e7da130 to 380c7f2 Compare November 18, 2022 05:38

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 13, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 13, 2022

k8s-ci-robot merged commit c2b5457 into kubernetes:master Dec 13, 2022

k8s-ci-robot added this to the v1.27 milestone Dec 13, 2022

Huang-Wei mentioned this pull request Apr 25, 2024

PV nodeAffinity matchExpressions problem with array items and in operator since 1.27.0 #123465

Closed

This was referenced Jun 5, 2024

[scheduler] absent key in NodeToStatusMap implies UnschedulableAndUnresolvable #125197

Merged

volume-binding scheduler prefilter assumes that a node's metadata.name == metadata.labels["kubernetes.io/hostname"] #125336

Open

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Jun 8, 2024

manually revert kubernetes#109877

4ef58b2

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

2493663

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

7410e9b

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

29a433c

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

7f838a4

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

fdca80f

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

00fa30f

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

b1c9d7e

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

e74cba9

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

9ef0d36

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

9493b6e

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

d82b543

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Sep 24, 2024

manually revert kubernetes#109877

2361b1e

AxeZhan added a commit to AxeZhan/kubernetes that referenced this pull request Oct 12, 2024

manually revert kubernetes#109877

c4a8184

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scheduler volumebinding: leverage PreFilterResult for bound local PVs #109877

scheduler volumebinding: leverage PreFilterResult for bound local PVs #109877

yibozhuang commented May 7, 2022 •

edited

Loading

k8s-ci-robot commented May 7, 2022

k8s-ci-robot commented May 7, 2022

yibozhuang commented May 7, 2022

Huang-Wei left a comment

Huang-Wei commented May 10, 2022

yibozhuang commented May 10, 2022

yibozhuang commented Nov 18, 2022

Huang-Wei commented Nov 18, 2022

yibozhuang commented Dec 2, 2022

dims commented Dec 12, 2022

msau42 commented Dec 13, 2022

msau42 commented Dec 13, 2022

k8s-ci-robot commented Dec 13, 2022

scheduler volumebinding: leverage PreFilterResult for bound local PVs #109877

scheduler volumebinding: leverage PreFilterResult for bound local PVs #109877

Conversation

yibozhuang commented May 7, 2022 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Testing example:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented May 7, 2022

k8s-ci-robot commented May 7, 2022

yibozhuang commented May 7, 2022

Huang-Wei left a comment

Choose a reason for hiding this comment

Huang-Wei commented May 10, 2022

yibozhuang commented May 10, 2022

yibozhuang commented Nov 18, 2022

Huang-Wei commented Nov 18, 2022

yibozhuang commented Dec 2, 2022

dims commented Dec 12, 2022

msau42 commented Dec 13, 2022

msau42 commented Dec 13, 2022

k8s-ci-robot commented Dec 13, 2022

yibozhuang commented May 7, 2022 •

edited

Loading