@kubernetes/sig-apps @kubernetes/deployment pausing a Deployment foo does not stop the deployment controller from managing replicas for foo. Not sure if we can have pausing for a ReplicationController foo-1 that would make the RC manager not manage replicas.

Manually creating pods and adding them to a replication controller by having the pod use labels that will get picked up by the replication controller

I'm not sure is a use case that we intended to support, although I understand the desire to do it in some cases. Can you describe some of the detail about the underlying use case that matters for you?

This relates to my thread here: openshift/origin#11954.

The setup is as follows using a blue/green deployment strategy with an additional abstraction of stage/active.

Before (dc = deployment config / deployment):

stg rte-> stg svc-> null
act rte-> act svc -> dc X

Stage Y

stg rte-> stg svc-> dc Y (give the rc template a label of status: stage)
act rte-> act svc -> dc X

Promote Y (delete dc X)

stg rte-> stg svc-> null
act rte-> act svc -> dc Y (give the rc template a label of status: active)

At the "promote Y" stage the pods created by dc Y will still have the label of status:stage. We were investigating how we could update those labels on both the RC and the pre-existing pods without creating new pods or fighting with the RC autoscaler. This doesn't seem possible at the moment.

One of the ideas we had around this was we could perform promotion on a pod level instead of carrying a dc through the stage -> active promotion.
Before:

stg rte-> stg svc-> stg dc
act rte-> act svc -> act dc

Stage Y

stg rte-> stg svc-> stg dc (update dc and rc template)
act rte-> act svc -> act dc

Promote stg -> act in the following steps:

• disable stg dc (in particular the replication controller) and update it
• disable act dc (in particular the replication controller) and update it
• get a list L1 of pods from stg dc, L2 of pods from act dc
• For each pod in L1, re-parent it to be owned by "act dc"
• For each pod in L2, re-parent it to be owned by "stg dc"
• (optional) scale down stg dc to 0
• enable stg dc
• enable act dc

A "PodSwitch" strategy could be written as a custom deployment strategy (openshift concept but I think it has trickled back into kubernetes with the deployment object). It would have an input of a source dc / rc:

1. disable the source and target dc / rc
2. swap pods by changing their labels
3. (optional) scale down the source dc to 0 thereby deleting the old active pods
4. re-enable the source and target dc / rc

Another use case: Having pods in "standby mode". This can be helpful for quick scaling when pod startup is slow.

Example:

1. RC controls 3 pods A, B, and C. Pods D and E are in standby mode - they exist and are ready to perform work but just are not being selected by a service.
2. Load increases; Add Pods D and E to the mix and have them be controlled by the RC by changing their labels to match what the RC and service is expecting

Problems happen with step 2. The RC replica count needs to change at the exact same time as the RC gains more pods. If the replica count changes too soon it may try to spin up spurious pods. If the replica count changes too late it may try to spin down pods that were just added to the RC.

Independent of the merits of pause on the RC, I did want to mention the
comment from the other thread that in general, doing a blue green swap is
about minimizing the number of changes in either direction. Ideally a
single atomic operation can do the flip. Re-labelling of pods is not
atomic like that, so I would generally suggest for this specific case that
we try to find a way to make service / ingress / route + deployments behave
the way we want where only a single change is necessary to swap.

On Fri, Nov 18, 2016 at 11:09 AM, Mark notifications@github.com wrote:

This relates to my thread here: openshift/origin#11954
openshift/origin#11954.

The setup is as follows using a blue/green deployment strategy with an
additional abstraction of stage/active.

Before (dc = deployment config / deployment):

stg rte-> stg svc-> null
act rte-> act svc -> dc X

Stage Y

stg rte-> stg svc-> dc Y (give the rc template a label of status: stage)
act rte-> act svc -> dc X

Promote Y (delete dc X)

stg rte-> stg svc-> null
act rte-> act svc -> dc Y (give the rc template a label of status: active)

At the "promote Y" stage the pods created by dc Y will still have the
label of status:stage. We were investigating how we could update those
labels on both the RC and the pre-existing pods without creating new pods
or fighting with the RC autoscaler. This doesn't seem possible at the
moment.

One of the ideas we had around this was we could perform promotion on a
pod level instead of carrying a dc through the stage -> active promotion.
Before:

stg rte-> stg svc-> stg dc
act rte-> act svc -> act dc

Stage Y

stg rte-> stg svc-> stg dc (update dc and rc template)
act rte-> act svc -> act dc

Promote stg -> act in the following steps:

• disable stg dc (in particular the replication controller) and update
  it
• disable act dc (in particular the replication controller) and update
  it
• get a list L1 of pods from stg dc, L2 of pods from act dc
• For each pod in L1, re-parent it to be owned by "act dc"
• For each pod in L2, re-parent it to be owned by "stg dc"
• (optional) scale down stg dc to 0
• enable stg dc
• enable act dc

A "PodSwitch" strategy could be written as a custom deployment strategy
(openshift concept but I think it has trickled back into kubernetes with
the deployment object). It would have an input of a source dc / rc:

1. disable the source and target dc / rc
2. swap pods by changing their labels
3. (optional) scale down the source dc to 0 thereby deleting the old
   active pods
4. re-enable the source and target dc / rc

Another use case: Having pods in "standby mode". This can be helpful for
quick scaling when pod startup is slow.

Example:

1. RC controls 3 pods A, B, and C. Pods D and E are in standby mode - they
   exist and are ready to perform work but just are not being selected by a
   service.
2. Load increases; Add Pods D and E to the mix and have them be controlled
   by the RC by changing their labels to match what the RC is expecting

Problems happen with step 2. The RC replica count needs to change at the
exact same time as the RC gains more pods. If the replica count changes too
soon it may try to spin up spurious pods. If the replica count changes too
late it may try to spin down pods that were just added to the RC.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#37086 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p023kMNmSGPTMoOtbpuDvfjQtQhAks5q_ff4gaJpZM4K2ehq
.

One could always delete the RCs and then re-create them.

I think the most compelling case is scaling down and choosing a victim, which some other systems (e.g., Marathon) support. For that, a simple grace period before choosing a victim would suffice. We've discussed that previously in a number of issues.

Deleting the RCs could leave you in a bad state should you come across a failure. I had considered it but it also means that if something died along the way I could be left without an RC which could make a mess.

Any case in which a pod might be shared by two replication controllers

I am interested in what cases you have for this kind of setup. We have a lot of issues with overlapping controllers and I am not sure we want to allow ways of getting this any more complex than it already is. Managing labels in general is hard for average users. Wouldn't #36897 be helpful for you?

@Kargakis yes #36897 would be helpful.

The case I was thinking of was a rolling strategy that reused old pods when the content of the pods did not change. Hence, instead of new pods and old pods being spun up / down one-by-one, they would be transferred over one-by-one and then any additional scaling that is needed would be performed. At present my desire for this is to try to work around not being able to atomically change RC selectors and pod labels at the same time (openshift/origin#11954, #36897)

@MarkRx #9043 is related

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@MarkRx: Reopened this issue.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@fejta-bot: Closing this issue.