It's a design similar to what we discussed in the issue. But, I switched this to fallbackCriteria for the reason described here. Open for discussion about which design we should choose.

@alculquicondor What do you think?

@wojtek-t I assigned it to you as other sig-scheduling KEPs do.
But, please reassign it to other people if needed. 🙏

Just quickly skimmed through it - I will reviewer deeper when you will have buy-in from the SIG.

Just quickly skimmed through it - I will reviewer deeper when you will have buy-in from the SIG.

I don't understand it - can you clarify?

For example, let's say PodA has a required Pod Topology Spread with fallbackCriteria: ScaleUpFailed. When PodA's first scheduling, all Nodes went through Pod Topology Spread filter plugin, but they were rejected by other plugins (e.g., the resource fit filter plugin because of insufficient resources.), and PodA got unschedulable state as a result. Then, the cluster autoscaler tried to create a new Node for PodA but it couldn't make it because of stockout.

In this case, Pod Topology Spread doesn't need to fallback actually because it said OK to all Nodes in the previous scheduling cycle -- meaning Pod Topology Spread isn't the cause of unschedulable in the past scheduling cycle and the fallback won't make PodA schedulable.
But, the problem is that Pod Topology Spread cannot see why this Pod was rejected in the previous scheduling cycle. It only understands that PodA was rejected by someone in the previous scheduling, and the cluster autoscaler couldn't create a new Node.

nit:

Could it not simply default to FailedScheduling. As far as I know, ScaleUpFailed is a cluster-autoscaler specific event so for those who use something else (e.g. karpenter), then this will not work.

One challenge with FailedScheduling is that's what cluster-autoscaler reacts on so there could be a race condition in cluster-autoscaler scaling up vs. switching to fallback mode.

This part only means that ScaleUpFailed is a new value that will be added to fallbackCriteria. Yes, I took this name from the cluster autoscaler's event name, but it should work with any cluster autoscaler if they update TriggeredScaleUp Pod condition, which is also proposed in this KEP.

Curious if there's an analogous use case for:

• Preferential node affinity
• Preferential pod affinity/antiaffinity

Is there any world where a user may want to have more control over how preferences are handled with autoscalers?

if there's an analogous use case for:

Yes, might be. We would wait for the actual usecases from users before implementing the similar stuff in them though, we should make this design to be easy to follow in other scheduling constants

This makes me think that we might want to model fallback criteria at the pod level.

preferentialFallback:
  - PodAffinity
  - AntiAffinity
  - TopologySpreadConstraints

It may be difficult to get the granularity right, though. At another extreme (as you propose), you'd add a preference policy for each preferred term across the pod spec.

That's an interesting idea.
Because when people want to use fallback, they may want to do the same in all required scheduling constraints the Pod has. Also, we may want to give preferentialFallback the way to "fallback in all".

So, maybe it'd be like this:

preferentialFallback:
  schedulingConstraints: 
  - "*" # do fallback in all scheduling constraints.
  # You can also specify individual scheduling constraints name, which supports fallback.
  # (But, note that this KEP's scope is only TopologySpread.)
  fallBackCriteria:
  - ScaleUpFailed

Let me include this idea in the alternative section for now. Like the idea of introducing enums, we can gather feedback from more people about the design choice among them.

There are some benefits to reusing this an existing field with new enum values. I'm a touch concerned with how complicated topologyspreadconstraints are getting, with two new honor fields (with different defaults).

I interact with a bunch of customers who are learning kubernetes scheduling for the first time when onboarding to Karpenter, and communicating the complexity and gotchas of topology is always a huge hurdle to get over.

Obviously, I agree that we need to make it easy to understand as much as possible. But, also introducing a new field doesn't always result in confusion.
For this particular KEP, I think FallBackCriteria is not much more complicated than introducing new enum values in WhenUnsatisfiable. Then, currently I'm leaning towards FallBackCriteria for the ease to introduce it into other scheduling constraints, as described.
But, maybe I'm biased since I know the scheduler/CA, I know everything that this KEP wants to do, etc. 😅
So, I'd like to keep this discussion open to gather more feedback from reviewers.

I agree that topologyspreadconstraints are difficult to explain to clients (been there), but I think this added field is optional complexity. It is also the other way around, right? It is a complex solution to a complex setup, but the complex setup came first. Clients that need to hear this are already dealing with specific cases that call for all of this, and in theory are easier to talk to. Also agree that I don't see it as easier to explain when considering existing field with new enum values, so I guess that if it is hard to explain, it would be hard either way.