The same arguments for the Burstable tier would apply to the Guaranteed tier. I could see users that do not want to specify explicit requests/limits for each container, but just want to bound the pod with fixed resources.

The current behavior for the Guaranteed tier is already exactly this. A pod is in the guaranteed QoS if and only if it always has a request equal to the limit for all the containers in the pod. If this is the case, the current code already sets the pod level cgroup to the sum of the limits for each of the resources.

But because in the Guaranteed case all of the containers also have a limit set to them, setting the pod level cgroup to any limit is meaningless. The pod level cgroup limit is by definition greater than any individual container, which means that the individual container limit would apply before the pod level limit would apply.

A pod that wants to set pod-level fixed resources and not container level resources is by definition not in the Guaranteed QoS level.

for clarity, cpu is not unlimited... it is setting shares, just quota is unbounded.

Correct, I will fix the KEP.

the biggest benefit for this is when its difficult to size two containers that may individually write to a memory backed volume.

pod resource overhead feature is in flight, it would be good to combine this example with a pod that has a runtimeclass that has an overhead. i also wonder as i type this if its better to associate an attribute on the runtime class to say if the pod should be bounded or not rather than change the pod spec itself. have you explored this idea?

Can you link to the pod resource overhead feature? I'm not familiar with this.

Intuitively the pod level seems to me to be a better fit since this sort of resource allocation is not dependent on any specific runtime, and has no meaning outside of the specific Burstable QoS level. But I'd love to read about the pod resource overhead feature, and give it some thought...

While we are waiting for merge of the updated Beta docs, I'd suggest checking out the PR: https://github.com/kubernetes/website/pull/19059/files

Thanks, I'll go over this.

I wonder if we should have the phrase "the Kubelet translates these limits into the appropriate cgroup definitions". For runtimes like kata-containers, there is an additional level of indirection, because each pod is a VM and the cgroups (when they exist) are actually inside the VM.

This section is awkwardly worded. I think we can find a better way to express it. Something like:

"""

For each core resource type (cpu, memory) a pod may have a pod-scoped resource request and/or limit. Pods which have a defined request or limit for a given resource type are said to "have explicit resources" or "be explicitly resourced". Pods which DO NOT have a defined request or limit for a given resource type are said to "have implicit resources" or "be implicitly resourced".

A pod is considered Guaranteed if, for each core resource type, either of the following is true:

• It is explicitly resourced for that resource type AND the the pod's request is equal to its limit.
• It is implicitly resourced for that resource type AND all containers' requests are equal to their limits.

A pod is considered Burstable if, for each core resource type, any of the following is true:

• It meets the criteria for "Guaranteed" for that resource
• It is explicitly resourced for that resource type AND the pod's request is specified and less than or equal to its limit.
• It is implicitly resourced for that resource type AND all containers' requests are specified and less than or equal to their limits.

A pod is considered BestEffort if, for each core resource type:

• It is implicitly resourced for that resource type AND any container's request is not specified

We could maybe simplify more. Could we say that:

"""
For each core resource type, a Pod's resources are either (evaluated in order):

• the values specified in the pod's resources stanza
• the sum of the values specified in all of the pod's containers' resources stanzas, where a single unspecified value means the result is unspecified

A pod is considered guaranteed if, for each core resource type, the pod's request equals the limit.
"""

set this

Eventually I think the right behavior of these admission controllers would be to add the resources of the injected containers to the pod-level resources.

What do container-level resource requests mean if pod-level requests are set? Are they completely ignored in that case? If so, should that be an error case?

EDIT: Actually, I don't think it should be an error case to account for injected sidecars. Maybe a warning though.

I disagree. I think we should go with the second option for consistency (apply pod-level cgroup to init containers). Consider the case where pod-level limits are defined, and the user didn't bother to set container-level limits. I would expect the pod limits to cap the init containers.

@tallclair I agree with you from the UX point of view.

One problem I see is that if the init container requires much more resources than the regular + sidecar containers, users will be forced to set a very high pod resource limits to accommodate initial surge. This might not be a concern given that we already account for the init container resource requirements during scheduling, but just want to bring it up as it's legit use case.

Should we address how will pod level resource limits affect ephemeral containers?

Might be okay (not worth a warning) if we account for ephemeral containers

Another use case is to allow better resource utilization since we don't need to specify the limits (i.e., peak usage) for each individual containers. Instead, we do this at the pod level, which allows more flexibility through sharing.

@tallclair I agree with you from the UX point of view.

One problem I see is that if the init container requires much more resources than the regular + sidecar containers, users will be forced to set a very high pod resource limits to accommodate initial surge. This might not be a concern given that we already account for the init container resource requirements during scheduling, but just want to bring it up as it's legit use case.

• What's the behavior if resources are not specified for the init containers?
• Also, wouldn't this complicate the implementation significantly?

Would you also add a keps/prod-readiness/sig-auth/1592.yaml file with:

kep-number:1592
alpha:
  approver: "@jpbetz"

? Thanks!

Is this still targeting 1.31 for alpha? If so, please copy in the Production Readiness Review Questionnaire section from the template and fill out the alpha sections.

@jpbetz #4678 is the revised version of this KEP that is being worked on actively. May I please add you as the PRR reviewer since John might be busy.

Yes, please sign me up