We've encountered this race previously in other contexts. See also #7328.

So kubectl will compare the rv it got via its resize update with the rv in the rc's status? that would work but i think we've stayed away from comparing rvs for various reasons in many other clients. Fyi I think we should fix this for 1.0, either hacky or clean, or people will be confused. We started rate limiting the controller-manager to 20qps, so it takes 10s to create an rc with 200 pods. Stopping during those 10s has undefined consequences.

The proposal in #7328 would add a new sequence number, not use resourceVersion.

If we don't want to bite the bullet and properly implement graceful termination (#1535) in the server, then another alternative would be to just delete the replication controller, and then the pods matching its selector. Rolling update already has such a cleanup loop:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubectl/rolling_updater.go#L244

Just cleaning up orphaned pods won't cut it because deleting the rc != the rc manager noticing the delete. So if we cleanup orphaned pods and the rc manager is processing the rc, it will still create pods. Looping around till there are none left might, but feels dirty. I'm going to take a stab at a few experimental implementations and see what works best given our 1.0 constraints.

I think I'm going to stick with watching pods instead of status.Replicas in kubectl, that'll fix a majority of the confusion and confine the changes to kubectl, though some races will still be possible. In theory graceful termination is a feature and we have feature freeze on friday, so that's probably not getting.

This is too error prone to do without some changes to the rc manager. The best change discussed so far (and one that will be helpful long term) is to add a sequence number to the spec and status of the rc. Everytime the rc manager takes any action it mirrors the sequence number from the spec into the rc, and kubectl does:

• resize to 0
• poll on status.sequenceNumber from previous step so it knows the rc has seen the reisze
• poll on status.Replicas
• delte rc

Fixed via #9739