Currently with the presence of reservedSystemCPUs under defaultCPUset we have the ability to accommodate best effort and burstable pods on a node even if it is resource constrained (e.g. Guaranteed pods occupying majority of CPUs exclusively). But with this option enabled, we could end up in a lot of evictions.

The eviction manager on the node prioritizes pods based on their QoS Class. We should be okay in case of standalone deployments as the pods could go other nodes that have enough resources but what worries me is the fact that there are a lot of infrastructure components (e.g. Device plugins) deployed as daemon sets that might not necessarily belong to Guaranteed QoS class (I took a quick look at SR-IOV device plugin: https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/v3.6.2/deployments/sriovdp-daemonset.yaml#L49-L54, we can perhaps find other such examples). We could end up in a never ending cycle of eviction by Kubelet eviction manager and scheduler reschedulng the pods on the node by the daemon set controller.

What is our mitigation strategy for this ?

High performance workloads, the typical type that using SRIOV/DPDK, are also typically deployed as Guaranteed QoS, they are not affected when strict-cpu-reservation option is enabled.
You enable 'strict-cpu-reservation' when you have to protect infrastructure services, all the systemd daemons and interrupt processing, from busty workloads. Telco is a typical use case where we do our business. If you don't have this issue, then don't enable 'strict-cpu-reservation' option.

Will add feature interaction test cases with the existing policy options.

I concur with @swatisehgal . We explored a very similar concept in the not-so distant past and exhausting the burstable pool is something which I believe should not be done lightly. In this case at least we have an option (vs a builtin default behavior) which is a step in the right direction, but I still believe we need some form of protection to prevent exhaustion of the shared cpu pool. We will perhaps need a new option or to overload the semantics of existing options.

We explored a very similar concept in the not-so distant past and exhausting the burstable pool is something which I believe should not be done lightly. In this case at least we have an option (vs a builtin default behavior) which is a step in the right direction, but I still believe we need some form of protection to prevent exhaustion of the shared cpu pool. We will perhaps need a new option or to overload the semantics of existing options.

Exactly my point!

I understand that the intention here is to protect infrastructure services (systemd daemons and interrupt processing) but my comment above was about the potential impact on infrastructure pods deployed using daemonset which might not always belong to Gu QoS Class especially when they are running on a node that has resource contention. Kubernetes doesn't have a way to distinguish between application pods and infrastructure pods.

If we don't want to restrict reservedSystemCPUs for OS processes / systemd services, you can simply not opt-in to this option but daemonsets are often used for deploying infrastructure pods on all the nodes irrespective of the configured Kubelet policy or configured option.

Typically in telco use cases, the goal is to maximize CPU utilization by providing CPUs for exclusive allocation to high performance workloads. I would like us to consider and capture in the KEP what happens to BE and BU pods when high performance workloads have been allocated all the CPUs available (total -reserverd) to them. With the current proposal it sounds like we could we end up with an empty defaultCPUSet which IMO should be avoided.

Does CPU exhaustion on a node result in eviction of such pods (even if they are important infra pods)? Could we end up in a never ending cycle of eviction by Kubelet eviction manager and scheduler reschedulng the pods respawned on the node by the daemon set controller?

ok, this is my take of the discussion:

(1) From the scheduler point of view, there is a single set of CPUs to schedule pods, which is the node allocatable = total - reserved. It is at the node level, burstable and best-effort pods are allowed to run on the reserved cores. Yes, it is very common people/(our customers) are surprised that burstable and best-effort pods are allowed to run on the reserved cores since it is against "common" sense. The 'strict-cpu-reservation" feature is effectively removing this discrepancy.

(2) However, there is this concern of no longer be able to "hide" burstable and best-effort pods on the reserved cores, so we will add minSharedCPUs. Introducing minSharedCPUs will impact scheduler because it is a discrepancy that has to be taken care of. This scheduler impact changes this 'strict-cpu-reservation' feature from trivial to at least not trivial.

Please review and correct my understanding.

Your understanding is accurate, thanks for summarizing this.

And, is it acceptable to document the scheduler impact in the alpha version while only do the scheduler code change in beta version? for the sake of simpler feature interaction and thanks to feature flag is by default false in the alpha version

My take would be that correct scheduling behavior is a very critical aspect of this feature and needs to be handled at alpha stage itself. We need the right end to end behavior in case the feature is enabled at alpha stage.

SIG scheduling's opinion should be weighed in as well on this subject.

Hi @ffromani thank you for replying and confirming it is a valid option to keep the feature off forever (Alpha/Beta/GA).

HI @swatisehgal, thank you for replying and confirming we are aligned on the feature has no eviction impact.
What other feature impact do you think could involve scheduler? At this moment, I don't see obvious scheduler impact, so I do not have any in the KEP itself. How do I invite scheduler maintainers for comments? I did "/add-sig scheduling". Please advise.

I'm thinking about exhaustion of the shared cpu pool and its implications. With this change, can we ever reach a point on which there are no cpus left in the shared pool?

Adding minSharedCPUs (final name TBD) will prevent that, effectively partitioning the CPUs in the node in 3 pools. But the existence of that third pool need to be exposed to enable the scheduler (and the system in general) to make the correct decisions. If nothing else, the CPUs available for exclusive allocation won't be capacity-reserved as it is today, but will be capacity-reserved-minSharedCPUs.

IOW, I don't see (yet?) a way to implement robust system behavior without somehow exposing to the rest of the system the minSharedCPUs pool size existence, which justifies the scheduler changes.

thank you for replying and confirming it is a valid option to keep the feature off forever (Alpha/Beta/GA).

Since this is not a norm (typically Beta onwards a feature is enabled by default), I would suggest explicitly discussing this during PRR (Performance Readiness Review) stage.

How do I invite scheduler maintainers for comments? I did "/add-sig scheduling". Please advise.

I can see that SIG Scheduling is already cc'd on the KEP, they should triage it soon. If we don't hear from them in a week or so, please give a nudge on slack (https://kubernetes.slack.com/archives/C09TP78DV).

I concur with @swatisehgal . We explored a very similar concept in the not-so distant past and exhausting the burstable pool is something which I believe should not be done lightly. In this case at least we have an option (vs a builtin default behavior) which is a step in the right direction, but I still believe we need some form of protection to prevent exhaustion of the shared cpu pool. We will perhaps need a new option or to overload the semantics of existing options.