-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable deployment strategies to tolerate concurrent external scaling events #14062
Comments
Related: #11874 |
Also relevant: In the case of Deployment, the desired replicas needn't be stored in any of the underlying ReplicationControllers. If one were to attach the desired replicas to a single RC, then one needs to deal with the case of creating a new RC with a new desired replicas while an existing rollout was in flight. A new field would mean that a newer client wouldn't work with older apiservers, and the new rolling update wouldn't be compatible with older clients performing scaling. Keeping track of changes made by rolling update instead of the desired replicas count would enable desired replicas to be calculated from the underlying RCs in the presence of simultaneous scaling. Here's an example to see how complicated this would be.
Done. Seems to work. |
Looks like you're right; delta tracking would do the trick. Thanks for the thorough analysis. |
How are we keeping track of the deltas? Annotations on the RCs? |
cc @jszczepkowski as he is interested in getting deployments to work with autoscaler. |
Horizontal Pod Autoscaler when working with Deployments, should resize the Deployment, not a RC being a part of it. So, the situation @ironcladlou mentioned shouldn't happen. |
Can this issue be closed? Sounds like it's not going to be a problem. |
I don't think this can be closed unless/until we delete the replica-count annotations from the rolling-update implementation. |
We need to decouple scaling from rolling update, and get rid of original-replicas. The solution to #14669 made rolling updates even less tolerant to simultaneous scaling. Rolling update shouldn't change the number of replicas. |
Deployment also needs to work with simultaneous scaling and rolling update. It shouldn't generate a new RC just to scale. |
We need these annotations in order to facilitate re-adoption after orphaning. |
Doesn't the new deployment controller rolling updater design invalidate my original issue? The autoscaler (and scaling endpoint) should be mutating only the deployment spec replicas (not the RCs directly). The reconciler will consider the latest deployment spec replicas each interval. |
I think this is correct. |
The rolling updater should be tolerant of external scaling events concurrent with the update, but lacks the required context to reevaluate its internal state. As a result, updates can conflict with manual scaling and the autoscaler in unpredictable ways.
The rolling updater computes the desired replica count for the target replication controller only once during initialization based on a snapshot of the the target replication controller state. If during the rolling update the target replication controller
spec.replicas
is modified externally (via scale operations), the rolling updater is unable to deduce the intent of the replica count change in order to recompute the target desired replica count for the replication controller.A new replication controller field such as
spec.trendingToReplicas
could be added and considered a formalization of thekubectl.kubernetes.io/desired-replicas
annotation. This field would provide a means for deployment strategies and scaling systems to set a desired replica count for the replication controller itself in the context of replication controller transitions (such as a rolling update).The rolling updater (and any other strategies) would drive the replication controller
spec.replicas
towardsspec.trendingToReplicas
. Any changes tospec.trendingToReplicas
could safely result in recalculating the scaling internals.Scaling systems and other API calls which ultimately want to mutate
spec.replicas
could then work safely with deployment strategies: Ifspec.trendingToReplicas
is set, updatespec.trendingToReplicas
and cause rolling updates (or other deployment strategies) to adapt. Ifspec.trendingToReplicas
is unset, directly update the RCspec.replicas
.cc @nikhiljindal @bgrant0607 @smarterclayton @Kargakis
The text was updated successfully, but these errors were encountered: