Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DRA: competition between schedulers + allocators #128980

Open
pohly opened this issue Nov 26, 2024 · 2 comments
Open

DRA: competition between schedulers + allocators #128980

pohly opened this issue Nov 26, 2024 · 2 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. wg/device-management Categorizes an issue or PR as relevant to WG Device Management.

Comments

@pohly
Copy link
Contributor

pohly commented Nov 26, 2024

What would you like to be added?

At the moment, DRA uses the approach that one scheduler instance "owns" all resources on a node or available for a node (in the case of network-attached devices). This is the same approach that is used for other resources. It enables faster scheduling because allocation can happen without coordination with other entities.

This approach breaks down when there are multiple schedulers in the cluster such that each scheduler instance is responsible for its own subset of the nodes ("sharding") and there are network-attached devices that are available for more than one set of nodes.

Also, sometimes users run additional schedulers for the same nodes as the system scheduler. While that is already problematic regarding CPU and memory, with devices it might be even worse.

/sig scheduling
/wg device-management

Why is this needed?

For more advanced cluster setups.
@pohly pohly added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 26, 2024
@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. wg/device-management Categorizes an issue or PR as relevant to WG Device Management. labels Nov 26, 2024
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Nov 26, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@pohly
Copy link
Contributor Author

pohly commented Nov 26, 2024

Coordination could be done through a slice.status that records "reserved" devices. When coordination is needed (see below), a scheduler has to update that status first, then may allocate the device for a ResourceClaim. Somehow it has to be possible to detect stale reservations and remove then, because a scheduler might crash without coming back and never get to allocating the claim. Perhaps each reservation has to include an object reference? Then a controller can check reservations and remove them if the referenced object is gone.

This reservation protocol has to be optional to avoid its overhead when it is not needed. How to configure this is an open problem: putting it into the slice.spec is problematic because then the driver would need to know whether the resources are meant to be shared between schedulers. Putting it elsewhere leads to race conditions.

@pohly pohly mentioned this issue Nov 26, 2024
Open
@johnbelamaric johnbelamaric moved this from 🆕 New to 📋 Backlog in SIG Node: Dynamic Resource Allocation Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. wg/device-management Categorizes an issue or PR as relevant to WG Device Management.
Projects
SIG Node: Dynamic Resource Allocation
Status: 📋 Backlog
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pohly @k8s-ci-robot