-
Notifications
You must be signed in to change notification settings - Fork 39.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory manager support for Windows nodes #128560
Conversation
|
||
// bestEffortPolicy is implementation of the policy interfact for the BestEffort policy | ||
type bestEffortPolicy struct { | ||
static *staticPolicy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is not obvious to me what do we gain by wrapping the static policy. Could you please elaborate a bit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For Windows we want the new BestEffort
policy to have the same logic as the existing static
policy.
I figured that doing this would reduce a lot of code duplication.
I can update the comments here to reflect that or maybe I can have the kubelet arguments specify BestEffortPolicy
and then create a staticPolicy
instance here for Windows.
Do you all have a preference @ffromani @jsturtevant ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updating comments and wrapping seems to make sense to me. This will allow to adjust more easily if we learn of reasons to adjust.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll need to refresh my memory over what we discussed in the KEP before I can answer. Right now I can surely say comments is a great starting point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, first: I've nothing against reusing the code like this. We didn't do often in the codebase, but why not?
The issue I see however is that from my interpretation of the KEP I would have expected different hint generation on windows based on https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/4885-windows-cpu-and-memory-affinity#windows-memory-considerations
I think we can settle this with a good set of testcases, at very least unit tests. which seems lacking to me?
Granted, the policy is actually a thin wrapper over the static policy, which has its own tests; the tests I recommend to add are to ensure and document the expected behavior on windows.
The tests should cover preferably also the topology manager merge hint process. It will be similar in spirit to what you are adding to internal_container_lifecycle_windows_test.go
, but covering the generic topology manager flow.
There's likely not enough time to do this for this cycle. Plus I want to be practical: it's an alpha feature, disabled by default, and the early feedback from having some functionality in is important as well. So I don't think it is blocking, but I DO think this is beta-blocking (to discuss this topic in detail at least) and possibly we can need another alpha iteration to sort this out depending on the aforementioned feedback
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue I see however is that from my interpretation of the KEP I would have expected different hint generation on windows based on https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/4885-windows-cpu-and-memory-affinity#windows-memory-considerations
I didn't see a difference in what we would provide for a hint. The major difference is in the semantic meaning of the hint, which on Windows we can't guarantee that it the hint will be respected, only make that suggestion. What kind of difference are you thinking?
There's likely not enough time to do this for this cycle. Plus I want to be practical: it's an alpha feature, disabled by default, and the early feedback from having some functionality in is important as well. So I don't think it is blocking, but I DO think this is beta-blocking (to discuss this topic in detail at least) and possibly we can need another alpha iteration to sort this out depending on the aforementioned feedback
I appreciate this sentiment. We do have a user we are working with that is willing to provide feedback but is looking for it to be in an Alpha release. From my understanding of the challenges on Linux, these types of features need some real feedback to get them dialed in, and I anticipate we will get feedback that we will need to incorporate back into the design.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue I see however is that from my interpretation of the KEP I would have expected different hint generation on windows based on https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/4885-windows-cpu-and-memory-affinity#windows-memory-considerations
I didn't see a difference in what we would provide for a hint. The major difference is in the semantic meaning of the hint, which on Windows we can't guarantee that it the hint will be respected, only make that suggestion. What kind of difference are you thinking?
I'm basing my expectations on what I saw so far on linux for the cpumanager policy. I would like to see if for example returning a different set of NUMA bitmasks or different values for Preferred
field.
If we add a thin wrapper over the Static policy, we are saying that this policy should behave 1:1 like the static policy, and we are unnecessarily coupling the behavior. What I would rather see is a specification (and unit tests are a good vehicle for that) of the expected behavior, more precise than (yes, I'm oversimplifying here) "whatever Static policy does, but not enforcing".
Lacking specification is something that biten us already multiple already, so we should IMO go in the direction to have the spec spelled out more precisely, and the testcase well defined. If, after that, we can reuse the implementation, great.
If nothing else (which I can accept!) this should be discussed and elaborated in the KEP design details.
I'm happy to elaborate further in the next cycle. Again, more details in the KEP to elaborate more precisely the behavior can possibly be all we need.
There's likely not enough time to do this for this cycle. Plus I want to be practical: it's an alpha feature, disabled by default, and the early feedback from having some functionality in is important as well. So I don't think it is blocking, but I DO think this is beta-blocking (to discuss this topic in detail at least) and possibly we can need another alpha iteration to sort this out depending on the aforementioned feedback
I appreciate this sentiment. We do have a user we are working with that is willing to provide feedback but is looking for it to be in an Alpha release. From my understanding of the challenges on Linux, these types of features need some real feedback to get them dialed in, and I anticipate we will get feedback that we will need to incorporate back into the design.
Fine, so we will evaluate during the 1.33 cycle if we can move to beta (taking into account my oen feedback above) or if we need another alpha iteration.
The rest of the code overall LGTM, I'll have another pass ASAP but from my recollections I don't have anything major on my notes.
/triage accepted initial priority assessment subject to be changed |
/retitle Memory manager support for Windows nodes |
@marosset adding the policy name ultimates surfaces to the user and to the kubeletconfig API object so this change MAY require api-review, please take this into account. |
Changelog suggestion -Windows: Support memory manager on Windows
+Added Windows support for the node memory manager. |
|
||
// bestEffortPolicy is implementation of the policy interfact for the BestEffort policy | ||
type bestEffortPolicy struct { | ||
static *staticPolicy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, first: I've nothing against reusing the code like this. We didn't do often in the codebase, but why not?
The issue I see however is that from my interpretation of the KEP I would have expected different hint generation on windows based on https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/4885-windows-cpu-and-memory-affinity#windows-memory-considerations
I think we can settle this with a good set of testcases, at very least unit tests. which seems lacking to me?
Granted, the policy is actually a thin wrapper over the static policy, which has its own tests; the tests I recommend to add are to ensure and document the expected behavior on windows.
The tests should cover preferably also the topology manager merge hint process. It will be similar in spirit to what you are adding to internal_container_lifecycle_windows_test.go
, but covering the generic topology manager flow.
There's likely not enough time to do this for this cycle. Plus I want to be practical: it's an alpha feature, disabled by default, and the early feedback from having some functionality in is important as well. So I don't think it is blocking, but I DO think this is beta-blocking (to discuss this topic in detail at least) and possibly we can need another alpha iteration to sort this out depending on the aforementioned feedback
Ack - I'll rebase on that as soon as it merges. |
ab90eb1
to
284fa5a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
thanks for the rebase and sorry for the churn. You use the recently added containerMap.Clone()
, so looks fine again to me.
LGTM label has been added. Git tree hash: 28376828105d5f848a8c1e75791828342d5b5d5c
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: marosset, mrunalp The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
beta-blocking (or perhaps as later bugfix?) we need to update |
284fa5a
to
05a8977
Compare
/lgtm |
LGTM label has been added. Git tree hash: 9343fc1fc9ca096c90740b4240d16aae84833eed
|
05a8977
to
ad138d5
Compare
/lgtm |
Do we have documentation updates (including the feature gate) to the website? |
We'll use kubernetes/website#48469 to update docs for both cpu manager (#125296) and memory manager (this PR) support for windows. |
/kind feature
What type of PR is this?
What this PR does / why we need it:
This PR adds support for memory manager on Windows nodes and add a new
BestEffort
policy which allows giving preferences to the Windows OS for which CPUs to bind containers to.Windows cannot guarantee scheduling workloads to specific NUMA nodes so instead we query the OS for which CPUs are part of a given NUMA node and specify that when starting containers.
Which issue(s) this PR fixes:
Part of #125262
To enable memory manager support the kubelet configuration would look something like
Special notes for your reviewer:
Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
/sig windows node
/milestone v1.32
/area kubelet
/assign @jsturtevant