This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

/sig autoscaling
/sig apps
/wg batch

/cc

If we had 10000 Pods for a workload, how would using this subresource work? Any concerns there?

I'm a bit worried about the scale too. It reminds me a little of services blowing up with too many pod IPs.
And what would the semantic be for patching? If one only wants to update a subset of pods, does the patch support that (e.g. SSA).

Also, it seems like a new style of interaction where the handler of the scale subresource needs to patch annotations in all pods. How would we do that? What are the risks? Could this cause requests to stall if we fan out a bunch of patches?
This ties an annotation very explicitly to a non-annotation field. Maybe the annotation should first move to a proper API field (or something different like Pod Disruption Budgets).

I'm not very familiar with the annotation, but from a quick read would more expect that users run a cron or something that updates them at certain intervals. Updating them at scale down certainly works, but these annotations feel like metrics so I wonder if there is a better system to handle this in general (maybe using metrics API).

If no pod-deletion-cost is set as a parameter of the scale operation (only pods name/ip), it could be implicitly -MAX_INT by default

I'm a bit worried about the scale too. It reminds me a little of services blowing up with too many pod IPs. And what would the semantic be for patching? If one only wants to update a subset of pods, does the patch support that (e.g. SSA).

The idea of only updating specific Pods is inherent to the design of spec.podDeletionCosts. Think of spec.podDeletionCosts as a "transient" list of "requests" to update the pod-deletion-cost annotations of specific Pods, the goal of the Deployment/ReplicaSet controller is to make it empty by applying it to the Pods.

For example, if a 3-replica Deployment (my-deployment) has a ReplicaSet which has 3 Pods (pod-1, pod-2, and pod-3), consider the following scale patch on my-deployment:

spec:
  replicas: 2
  podDeletionCosts:
    pod-1: -100

The deployment controller can answer this request by doing the following (IN ORDER):

1. Updating the pod-deletion-cost annotation of pod-1 to be -100
2. Updating the spec.replicas of the deployment to be 2

We can be quite lax about processing spec.podDeletionCosts. For example, if the controller encounters a non-existent Pod name (or one not owned by the Deployment/ReplicaSet being patched), then it can just ignore that key and continue silently (possibly creating an event to bubble this up to the user).

Also, it seems like a new style of interaction where the handler of the scale subresource needs to patch annotations in all pods. How would we do that? What are the risks? Could this cause requests to stall if we fan out a bunch of patches? This ties an annotation very explicitly to a non-annotation field. Maybe the annotation should first move to a proper API field (or something different like Pod Disruption Budgets).

Unless the user requests a patch of all pods (by listing every pod in spec.podDeletionCosts), the controller should only need to make a small number of patches. But as a protection mechanism, we could have a timeout or a maximum number of spec.podDeletionCosts keys.

Regarding whether we should introduce a new Pod field, and remove the controller.kubernetes.io/pod-deletion-cost annotation, I think this was not done previously in KEP-2255 because such a field would not be applicable to all Pods (only ones in ReplicaSets), and we don't really like extending the Pod API.

I'm not very familiar with the annotation, but from a quick read would more expect that users run a cron or something that updates them at certain intervals. Updating them at scale down certainly works, but these annotations feel like metrics so I wonder if there is a better system to handle this in general (maybe using metrics API).

The KEP-2255 actually explicitly recommends against updating the pod-deletion-cost using automated methods, and only recommends people update it just before scale down. (Which is also noted in the docs).

It's really important NOT think of pod-deletion-cost as a metric, because the actual value is not important. Rather, the relative values across all Pods in a ReplicaSet (at the moment of scale-down) is what matters. During a scale-down, the controller will remove Pods with the lowest pod-deletion-cost first (with 0 being the implied default if no pod-deletion-cost annotation is set on a Pod).

If we had 10000 Pods for a workload, how would using this subresource work? Any concerns there?

@sftim Here is my thinking about Deployments with MANY pods.

In the "update" direction:

• The controller only needs to take an action for Pods explicitly listed in spec.podDeletionCosts, the overall number of Pods that happen to be in the Deployment is not relevant. (Unless the user does something like try and update all 10,000 Pods at the same time, but we could have a "max updates per request" for scale patches to mitigate this).

In the "read" direction:

• The status.podDeletionCosts (note: status) should contain only the Pods which have a pod-deletion-cost annotation set, which should be small or empty in most cases.
  • If the pod-deletion-cost annotation is used correctly (only setting it just before scale-down, not with a controller), the number of Pods with the annotation should be very low (only non-0 during scale-down), and the frequency of changes to the annotations should be very small.
• Rather than checking the annotations of all Pods for each request (by looping over all the owned Pods), we can store this information in the status field of the ReplicaSet (similar to our current status.availableReplicas and status.fullyLabeledReplicas, but rather than a single integer, as a mapping of pod_name -> pod_deletion_cost).
  • The ReplicaSet controller is already checking things about all the Pods it "owns", and watching for changes in "ready" status, so we can use a similar watcher pattern to reconcile the new status.podDeletionCosts field of the ReplicaSet in an efficient way.

I'm worried that this won't scale and will be a burden to maintain. You don't need to convince me, but SIG Apps and SIG Node might want to see your working.

FYI #124306

Although it's quite more limited than what is proposed here

In wg-serving we've identified a few places where an improved pod deletion cost would be relevant, although not quite aligned with this proposal.

One LLM serving use case is to be able to bias scale down towards specific replicas based on their load which varies over a few to tens of seconds for many use cases. In these cases an upstream component may be biasing traffic flow (since LLM latency is proportional to traffic, by controlling the simultaneous requests to a server you can achieve different latency goals) to specific instances, and would like to have a way to ensure scale down happens on the instances that are not at their target traffic (i.e. autoscale should scale the one that the upstream component is already steering traffic away from).

I would generally not think of the solution being specific to a particular workload controller - implementing it as a field on replica sets unfortunately prevents statefulsets or jobs or argo rollouts from participating. In the use case above, we'd want to be thinking about pods (from the traffic shaping level), not the controller that created those pods.

A comment on CUJs, not on this specific proposal: One more use case to consider is the alignment of behavior between parent controllers and kube-scheduler. If kube-scheduler is configured to target most utilized nodes first, it would make sense for the parent controller to delete pods from least utilized nodes first. Similarly, if kube-scheduler favors spreading by targeting least utilized nodes with new pods, the controllers should then favor deleting pods from most utilized nodes first. Otherwise the scheduler settings are not actually meaningful in clusters with a lot of autoscaling.

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale

This is a longstanding important feature, to avoid needing to patch the pod-deletion-cost as annotations.

Being able to set them temporarily for a specific downscale would be a game changer for Kubernetes Deployment autoscaling, as it would make it much easier to provide candidates for deletion.

This feature and its problems have been discussed in #124306. It could potentially be solved by the proposed Evacuation API: #124306 (comment)

I see the same scalability concern on this, like other folks, and I don't see a valid solution for that.

In the "update" direction:
The controller only needs to take an action for Pods explicitly listed in spec.podDeletionCosts, the overall number of Pods that happen to be in the Deployment is not relevant. (Unless the user does something like try and update all 10,000 Pods at the same time, but we could have a "max updates per request" for scale patches to mitigate this).

So, what if there're 10k Pods, and it's about to get scaled down to 1? How do we update the annotation of 10k-1 Pods in a scalable way? and who? (<- apparently SIG-App maintainers don't want to include any scheduling related logic in the RS controller (see comments on my issue from folks). I believe they also would not want to do with the deployment controller as well for the same reason)
I do prefer to use Evacuation API than this proposal.

Transfer your comment here:

I say this because currently, the evacuation API KEP (kubernetes/enhancements#4565) does not seem to address the core purpose of pod-deletion-cost, which is influencing which replicas are removed during down-scale (when reducing the replicas of a deployment).
#124306 (comment)

The Evacuation API does... right? like @atiratree mentioned at the KEP's motivation and the descheduling/downscaling section.

So, what if there're 10k Pods, and it's about to get scaled down to 1? How do we update the annotation of 10k-1 Pods in a scalable way?

Yeah, this is very difficult during the scale subresource request.

We do not connect to the deployment/replicaset controller, we only manipulate the API objects. The advantage of the scale is that it is generic and can be used for many controllers. So to implement this we either:

• Pass the pod cost to the underlying object (e.g. deployment), but that does not scale well and would require support and API change from all the controllers. And it is impossible to enforce such support for the CRDs.
• Update the pods before scaling. But how do we manage such a large number of pods and possible update failures? And without breaking the API contract?

I think an easier alternative is to update all the pods before calling the scale endpoint.

The Evacuation API does... right? like @atiratree mentioned at the KEP's motivation and the descheduling/downscaling section.

Yes. It does not implement the workload scaling and scheduling itself, but it could be used as a scalable delivery mechanism for the downscaling that other components (e.g. HPA, descheduler) can use and build upon.