I've proposed a solution to this upstream: kubernetes/kubernetes#36897

I think having two dc's and two services is the natural way to solve this. I think it's also the right mechanism for many isolation cases.

Multiple dc's are used in our case - when staging. We are doing switching at a service level instead of a route level due to some reasons that were explained to me by networking and the infrastructure team (something to do with load balancing, wide ip's, and HA proxy).

Before:

stg rte-> stg svc-> null
act rte-> act svc -> dc X

Stage Y

stg rte-> stg svc-> dc Y
act rte-> act svc -> dc X

Promote Y (delete dc X)

stg rte-> stg svc-> null
act rte-> act svc -> dc Y

I understand another approach would be using blue/green services and dc's that are agnostic of if they are production items or not. It would look like this:

Before

stg rte -> blue svc -> null
act rte -> green svc -> dc green

Stage

stg rte -> blue svc -> dc blue
act rte -> green svc -> dc green

Promote

stg rte -> null
act rte -> blue svc -> dc blue

We ended up not taking this approach due to potential confusion from an operator standpoint as to knowing what the state of the system is. In particular, an openshift project could contain N number of components with some being blue and some green. The additional stage/active abstraction makes it clear if a pod is serving production traffic or not. What is more labels can now be used by selectors to easily pick which pods to route to based on if they have a "stage" or "active" label. At least, they would if the label could be dynamically updated.

Generally when people want to do blue green, they're doing it so the fewest possible moving pieces change between stage and production. Part of my concern with relabelling (to have one deployment get another's pods) is that it's a lot of individual moving part actions (if one of those moves fail, you might break production and staging). I think there is a lot of advantages to focusing on using only things that can be a single atomic operation in order to do swaps. Today that would only be the service label selector or the route backend selector. Even if we added relabelling, the potential for failure at any step goes up, and the possibility of rollback is much harder. But changing a service label selector or route backend selector back is pretty easy.

If i had to make a tradeoff personally, I think I'd be willing to deal with some potential confusion about which deployment is "live" if it meant that my rollout / rollback is a single atomic operation.

A canary approach has been proposed which looks like the active / stg concept you described:

1. two deployments, active and next
2. service pointing to active
3. new deployments always happen on next first
4. once stage is validated, switch service to next
5. run for a while - once happy, redeploy active (which is hidden) to the same version as next
6. switch service back to active
7. scale next to zero.

That does mean occasionally "next" is taking the traffic, but has the advantage of always allowing you to know what the next state is (if next > 0 and service points to it, your next step is to update active, etc)

Having less moving parts does make sense.

In that canary approach the pods that get deployed to active are never validated as they are new. The pods from next are not reused. While I am of the opinion that it doesn't matter (it's the pod template that should be validated and not the actual pods themselves), some colleagues of mine insist that the pods that are validated be the same ones that are promoted and used in active. That's not hard to do (keep track of dc's being promoted), but it becomes an issue when they also want to know the state of a pod from metadata (to allow services to always point to the same labels and for internal business logic).

I get the impression that those two asks put together don't fit the platform model. It seems right now the platform is designed for RC's to be immutable (minus replication counts) and instead be recreated when changes are needed. Hence, changing label information on RC's / pods doesn't fit. Neither does reusing pods across RC's. Instead, pods should all be the same so in theory spinning them down/up in rolling deployments results in no net change. My colleges insist that it could (such as database connections going astray) but I argue it's no different than pods becoming unhealthy and being restarted.

In any case the question for the topic remains - are labels on pods meant to be immutable or can they be modified? If they can be modified does it make sense to update the RC as well to keep them in sync?

Because we don't provide atomicity across RC and Pods it's likely to remain
fairly fragile to change labels while the controller is actively managing
them. Pods are mostly immutable today, will likely stay that way for a
while, and there's lots of good reasons to think about them that way. I
don't think it's wrong to look at patterns that take advantage of the
limited mutability we allow, but we're unlikely to be capable of providing
a lot of support for that in any near term unless it's truly critical to
enable specific use cases.

Certainly the two deployment "swap" model is likely to work much more
cleanly in the near term under the constraints of pod reuse for the next
6mo to a year.

On Nov 23, 2016, at 7:24 AM, Mark notifications@github.com wrote:

Having less moving parts does make sense.

In that canary approach the pods that get deployed to active are never
validated as they are new. The pods from next are not reused. While I am of
the opinion that it doesn't matter (it's the pod template that should be
validated and not the actual pods themselves), some colleagues of mine
insist that the pods that are validated be the same ones that are promoted
and used in active. That's not hard to do (keep track of dc's being
promoted), but it becomes an issue when they also want to know the state of
a pod from metadata (to allow services to always point to the same labels
and for internal business logic).

I get the impression that those two asks put together don't fit the
platform model. It seems right now the platform is designed for RC's to be
immutable (minus replication counts) and instead be recreated when changes
are needed. Hence, changing label information on RC's / pods doesn't fit.
Neither does reusing pods across RC's. Instead, pods should all be the same
so in theory spinning them down/up in rolling deployments results in no net
change. My colleges insist that it could (such as database connections
going astray) but I argue it's no different than pods becoming unhealthy
and being restarted.

In any case the question for the topic remains - are labels on pods meant
to be immutable or can they be modified? If they can be modified does it
make sense to update the RC as well to keep them in sync?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#11954 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pzPgiAJyufHFyDcfYSYZ0GudxM5Sks5rBD6lgaJpZM4K1ex4
.

Closing this in favor of kubernetes/kubernetes#36897