cc @mfojtik

@Kargakis Sorry, quick comments for now.

We may want to think about a subresource, similar to /scale, that could do this uniformly across workload APIs. Maybe even we could extend /scale to be a generic workload-controller interface, but I haven't thought about that much yet.

cc @erictune

@bgrant0607 agreed that this should be similar to /scale, not sure about making it generic.

Use-case: Easier Blue-Green deployments: openshift/origin#11954

See also comments here:
#36859 (comment)

Note that one pattern that's safe and doesn't require complex orchestration when one wants to introduce a new attribute is to modify selectors of existing workload controllers to select pods without the new label key, and then just add the key to new pod templates. This isn't possible with RCs, however, since they don't support the more powerful selectors.

Would causing a relabel to happen spin down old pods and spin up new pods, or would existing pods be reused?

I don't know if this is by design, but at present it isn't really possible to store pod state in labels as changing the RC causes new pods to be created. Having pod state is useful for services as it allows them to be able to select pods based on their state.

Changing the RC is not causing new pods to be created, you are probably talking about Deployments / DeploymentConfigs. Changing a RC will create new pods if the RC is scaled up or old pods die. Relabeling would merely relabel all old pods to use the new label(s).

If the RC selector is changed the system would suddenly see 0 pods and try to start scaling up. Since existing pods cannot have their labels changed in an atomic operation when the RC selector is changed there is possible contention.

If the RC selector is changed the system would suddenly see 0 pods and try to start scaling up. Since existing pods cannot have their labels changed in an atomic operation when the RC selector is changed there is possible contention.

The RC selector should change as the last part of the "transaction" ie. once all old pods and the rc pod template labels have changed. If any of the latter actions fails, we won't change the selector. I guess once we start using transactions in etcd3, this won't be a problem.

@bprashanth because we had discussed pod labels changing to track state (like leader). I think the atomicity challenge remains even with etcd3, so we should at least consider whether this is something that we might indicate another way.

Was that officially decided somewhere? I think there are some reasonable use cases for multi-object transactions, for example the way we do pod binding today is not great because it only updates the pod, not the node, so you can't actually detect conflicts. Even just being able to bump the resource version on the node and the pod atomically, while only actually mutating the NodeName in the pod, would be useful. But I suspect there are other useful scenarios (whether the one here is one of those, I don't know).

We were afraid of the implication for sharding etcd, but we never actually needed to shard etcd (yet)...

It's off topic for this issue, so please don't discuss it further in this issue, but I agree with smarterclayton and lavalamp: that we should not support atomic transactions across multiple resources, and have documented that:
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md

OTOH, I could imagine some operation similar to rollback, which performs the relabeling orchestration server-side.

See also #14894

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

/remove-lifecycle rotten
/lifecycle frozen

Deployment controller stops relabeling adopted ReplicaSets and Pods after #61615 got merged. Selectors became immutable in apps/v1 endpoints. Can we close this issue now that it's no longer needed by workloads controllers?

I agree we are unlikely to get to this, and permitting it would increase complexity of object management.