On Oct 10, 2014, at 7:17 PM, bgrant0607 notifications@github.com wrote:

OpenShift supports parameterized templates and deployment resources:
http://rawgit.com/openshift/origin/master/api/openshift3.html#_templateConfigs
http://rawgit.com/openshift/origin/master/api/openshift3.html#_deploymentConfigs
https://github.com/openshift/origin/blob/master/examples/guestbook/template.json
https://github.com/openshift/openshift-pep/blob/master/openshift-pep-013-openshift-3.md

@smarterclayton Are those the best documentation sources for these concepts/APIs?

Those are - we haven't done a good job of updating the pep based on the exploratory design though.

We should consider whether/how that fits in with other config/deployment approaches.

For templates and Config specifically, we tried to start from #1007 and the discussion and keep them as simple as possible. Config is a list of arbitrary resources sent to the server, template is the same but with the addition of the ability to templatize ENV vars. The core use case for template was to describe how a secret could be generated and stored as an ENV var across two pod templates (web and db). We are trying to keep those simple.

Deployment config is special in the sense that it tries to sit at a useful intersection of a number of concepts which might be conflicting - we're trying to experiment with it as a higher level replication controller.

It's essentially a resource representing the desired deployed state (the config) and the desired normal deployment process (the process), with an optional set of triggers. It's trying to hit an 80/20 split for single pod use cases (mutate a DB pod over time, mutate a web pod when new images are available).

A deployment config controller tries to ensure a deployment exists (create if not found) that matches the current state of the deployment config (essentially that they have the same template).

If a deployment exists, the deployment controller tries to ensure that the deployment process has been started (as a job / run once pod). The deployment controller will try to stop deployment processes belonging to deployments with a version less than its version.

A deployment process is a run once pod that connects to the apiserver to mutate the replication controllers that have the same labels as the deployment. It tries to iteratively mutate them towards a steady state. If it detects a higher deployment number rc, or other changes, it exits. We've discussed it creating a new repl controller as well as trying to mutate the existing ones.

Deployment triggers dictate external changes that should alter the pod template (right now through image id only, but eventually through something like podex). The trigger controller watches for trigger events (image tagged in repo) and then does a GET to /deploymentConfigGenerators/ which returns the "correct" current state (if all triggers were up to date). It then compares that to the current config state, and if they are not matched, updates the deployment config (triggering the flow above).

Concerns I have so far:

1. That generation is too coupled to the state
2. That more than one pod template may be necessary
3. That other inputs to the deployment process may be necessary (although in theory any image can be used and any input could be used, so different strategies could correspond to different parameters

What I like is that it roughly replicates a particular flow that we know exists today - periodically updated images that may drift over time, and ensures that the highest level concept (the deployment config) represents intent (I want this pod template to use this image, but also to be updated when new images become available at a given location).

See also #503, though that mostly discusses builds, images, and jobs rather than deployments and templates.

—
Reply to this email directly or view it on GitHub.

A blog article: https://blog.openshift.com/builds-deployments-services-v3/

Example use of annotations to store deployment state: https://gist.github.com/smarterclayton/0862906d4aea8c8466e6

Latest iteration, which turns deployment config into a replication controller replicator (creates controllers for each distinct template change): openshift/origin#584 (comment)

On Dec 19, 2014, at 5:43 PM, bgrant0607 notifications@github.com wrote:

Example use of annotations to store deployment state: https://gist.github.com/smarterclayton/0862906d4aea8c8466e6

—
Reply to this email directly or view it on GitHub.

Here's an update to the replication controller data: http://fpaste.org/167788/42082392/ (ignore the \u0001, which is just a bug). Essentially we try to record the core state. When pod templates get split out the object gets much smaller, as does the RC.

Re: deployment @pmorie and I had a good discussion today. We think of DeploymentConfig in OpenShift as:

SimpleAutomaticDeploymentController - a controller that takes a pod template, can mutate it based on extensible changes that occur elsewhere (and can trigger on those), then writes the new pod template into a new replication controller, then launches a job that mutates the two.

Admittedly, all written out, that sounds complex, but it's kind of a sweet spot type of controller for code style apps. I think all automatic deployment controllers require some level of mutation, with the ability to stop if a mutation is invalid for manual intervention. An ADC is a fire-and-forget, self perpetuating thing, that when it can't proceed flags an operator. Other types of controllers can easily overlap with an ADC - for instance, an autoscaler should be a peer of an ADC, not a parent or child.

Each of the mutators may be an optional "trigger", where the entire mutation pipeline is run. The ideal triggers are edge driven - well defined value X at key Y (docker image repository "google/golang" changes the tag "foo" from pointing at image ID 1 to image ID 2), which means that the mutators don't need to take input from the trigger. However, when the mutation is run, each mutator should be invoked to gather its latest state (which handles multiple edge triggered changes happening in a close window and damps out noise).

Some mutators may have transient state - for instance, a web hook is an imperative call (POST to an ADC), so it's not possible to represent the data in the web hook as retrievable during mutation. However, you could argue that an ADC is about reacting to the world around it, so an ideal web hook would not communicate any info that is not discoverable by the mutator itself. That would remove the need for the mutator to take input from the trigger (other than recording the trigger). A mutator could in theory record annotation data into the ADC itself to track high water mark style behavior that can't be represented in the field, or simply define itself.

@smarterclayton Personally I fall into the 'ADC is about reacting to the world around it' camp.

A trigger shouldn't inject any input into the mutator and all inputs should be discoverable by the mutator. This is what we've implemented so far - the configuration of triggers is an input to the mutation process, but no input is injected by triggers. So, the mutation process (which we call 'generation') is driven purely by configuration, and not by the specifics of the triggering event.

Are users expected to run tests before pushing code that results in a deployment?

Are there any hooks for post-deployment end-to-end tests prior to exposing newly deployed instances to user traffic?

Yeah - that's something being worked on now. Builds can also execute tests before promoting an image. There's a fair amount of options for someone to hook into flows, although I would say we are still trying to build the experience that makes it easy to specify those.

There's some discussion about whether the "deployer" is really a "stage" component. Because the deployment process can be custom code, nothing stops it from being end user custom code. I'm hesitant to say it's had the rough edges cleaned off, but the deployment "process" is really just a pointer to a pod template /script in the image that has access to call the API.

I think it would be ideal to be able to integrate unit like tests earlier in the chain, with the deployment process being focused on canary and individual suite tests. I'd like to spend some time sketching out that complexity.

On Mar 17, 2015, at 10:44 PM, Brian Grant notifications@github.com wrote:

Are users expected to run tests before pushing code that results in a deployment?

Are there any hooks for post-deployment end-to-end tests prior to exposing newly deployed instances to user traffic?

—
Reply to this email directly or view it on GitHub.

https://github.com/ironcladlou/origin/blob/6bf387b6ac3eb738eac220137ce0ac6c89e16468/docs/proposals/post-deployment-hooks.md has some examples of lifecycle hooks before and after a deployment process completes, with a new lifecycle type "execNewPod" which creates a one off copy of your existing pod definition.

https://github.com/ironcladlou/origin/blob/6bf387b6ac3eb738eac220137ce0ac6c89e16468/docs/proposals/post-deployment-hooks.md has some examples of lifecycle hooks before and after a deployment process completes, with a new lifecycle type "execNewPod" which creates a one off copy of your existing pod definition.

Updated draft containing more details at https://github.com/openshift/origin/blob/master/docs/proposals/post-deployment-hooks.md.

@davidopp

There were a few comments on #1353 relating to a deployment controller:
#1353 (comment)
#1353 (comment)
and also here:
#4210 (comment)

Something like the deployment controller could do a few things:

• Generate a new replication controller, or at least a pod template, such as based on information (e.g., new image/tag) posted to a subresource
• Perform the rolling update, and perhaps support rollbacks and rollovers, also, as a service rather than in the client -- it should still be possible to execute this functionality from a client, though; also, it would need to declaratively represent rollout parameters, such as the update interval, but we want that even for the client-based implementation
• Provide an umbrella object with a durable name for tracking, managing, deleting a set of replication controllers (Consistently support graceful and immediate termination for all objects #1535)
• Assign identities a la nominal services PetSet (was nominal services) #260
• Provide a /size (resize - Proposal: scaling interface #1629) interface for the whole collection of RCs -- I don't think it would perform the actual auto-scaling itself

Openshift also supports triggers (e.g., new images/tags created), but it's not clear there are any useful triggers we could provide in Kubernetes at the moment.

Openshift also provides parameter-based substitution, but that's broader than just RCs/DCs (e.g., it can apply to services, builds, routes, images).

Ideally, the deployment controller could facilitate either fully declarative (#1702) or imperative updates/rollouts.

One of the main questions is the representation. RC representation was discussed in #3058 and #170. RC is still basically a replica count, selector, and pod template (or reference to one). DC could use a label selector to match RCs, or could maintain an array of RCSpecs or RC references. Given that the DC itself would be modifying replica counts and even which RCs exist (creation, deletion), the arrays seem unworkable.

Like RC itself, the DC would need a template from which to instantiate new RCs. That template could be explicitly parameterized, which maybe would overlap/conflict with a more general parameterization mechanism, or could be patched when generating the new RC for a rollout, or, like run-container, could support an explicitly specified restrictive subset of pod functionality. I'd be tempted to support a simplified, schema-aware patching API for common use cases (e.g., updates of images, commands, or env vars), but also support rollouts of full new RCs, similar to the existing kubectl rolling-update functionality.

cc @kelseyhightower

cc: quinton-hoole

----- Original Message -----

There were a few comments on #1353 relating to a deployment controller:
#1353 (comment)
#1353 (comment)
and also here:
#4210 (comment)

Something like the deployment controller could do a few things:

• Generate a new replication controller, or at least a pod template, such as
  based on information (e.g., new image/tag) posted to a subresource
• Perform the rolling update, and perhaps support rollbacks and rollovers,
  also, as a service rather than in the client -- it should still be possible
  to execute this functionality from a client, though; also, it would need to
  declaratively represent rollout parameters, such as the update interval, but
  we want that even for the client-based implementation

I think a nice compromise is to expect to run rolling-updates as a job in pods with access to the master API, in which case the rolling-update job is no different than the client.

• Provide an umbrella object with a durable name for tracking, managing,
  deleting a set of replication controllers (Consistently support graceful and immediate termination for all objects #1535)
• Assign identities a la nominal services PetSet (was nominal services) #260
• Provide a /size (resize - Proposal: scaling interface #1629) interface for the whole collection of RCs
  -- I don't think it would perform the actual auto-scaling itself

I was thinking about how to implement this - it's not obvious what resize on the whole collection should do unless there's a primacy relationship among the RCs. For our DC, we have the concept of "latest" which is sufficient, but if we want to manage multiple RC we'd need a way to manage dominators.

Openshift also supports triggers (e.g., new images/tags created), but it's
not clear there are any useful triggers we could provide in Kubernetes at
the moment.

Our triggers are based on level triggered image change detection. You can do that against existing Docker registries, or mirror a docker registry into a watchable resource (which we have done). I'd like to make that decoupled enough from this work that someone could have that feature without requiring it.

Openshift also provides parameter-based substitution, but that's broader than
just RCs/DCs (e.g., it can apply to services, builds, routes, images).

And we don't do parameter-based substitution on DCs today truly, but I think down the road both of those are relevant.

Ideally, the deployment controller could facilitate either fully declarative
(#1702) or imperative updates/rollouts.

One of the main questions is the representation. RC representation was
discussed in #3058 and #170. RC is still basically a replica count,
selector, and pod template (or reference to one). DC could use a label
selector to match RCs, or could maintain an array of RCSpecs or RC
references. Given that the DC itself would be modifying replica counts and
even which RCs exist (creation, deletion), the arrays seem unworkable.

Like RC itself, the DC would need a template from which to instantiate new
RCs. That template could be explicitly parameterized, which maybe would
overlap/conflict with a more general parameterization mechanism, or could be
patched when generating the new RC for a rollout, or, like run-container,
could support an explicitly specified restrictive subset of pod
functionality. I'd be tempted to support a simplified, schema-aware patching
API for common use cases (e.g., updates of images, commands, or env vars),
but also support rollouts of full new RCs, similar to the existing kubectl
rolling-update functionality.

There are probably two distinct classes of problem:

1. I want to get updates when images change or minor config changes happen
2. Everything else

Most of the demand is to implement 1, but I think it's important to have a solution that makes sense going from 1-2 (so that you could do config reconciliation across multiple sets of RCs and have things converge).

cc @kelseyhightower

---

Reply to this email directly or view it on GitHub:
#1743 (comment)

For reference here are the design docs (recent, but not total) https://github.com/openshift/origin/blob/master/docs/deployments.md and the slightly more user docs http://docs.openshift.org/latest/using_openshift/deployments.html

We probably are going to need to rename "Deployment" to avoid confusion. I haven't thought much about alternatives yet, though.

Kubectl run: Yes, a new version that generates deployments and jobs would be good.

One alternative is that we could rename Deployment to ReplicaSet to match DaemonSet, and choose another name for the new ReplicationController API.

Filed #14954 for kubectl run changes.

If we use ReplicaSet for Deployment, we could choose one of the previously rejected names (#3024) for RC for the RC with new selector and other changes. If we expect it to just be an implementation detail of Deployment/ReplicaSet, the name matters somewhat less. Shorter would be better. I could live with Replicator.

Just for posterity: Marathon terminology is Application.

We considered and rejected application because of a) baggage and b)
inaccuracy in the face of larger composite applications. It does need to
indicate a unit of function for sure - it's the most likely building block
for applications (and reduced to the minimum, can be an application). We
considered and rejected service, micro service, application unit, and
component for various clarity reasons.

On Oct 2, 2015, at 7:31 AM, Brian Grant notifications@github.com wrote:

Just for posterity: Marathon
https://mesosphere.github.io/marathon/docs/application-basics.html
terminology is Application.

—
Reply to this email directly or view it on GitHub
#1743 (comment)
.

Regarding "Replicator": I thought we decided all of the API objects that describe collections of workers should sound like the workers rather than the thing that controls the workers -- e.g. DaemonSet rather than DaemonController, etc. I think Replicator violates that principle.

Was "Job" already rejected? If not, that has my vote. It's already a well known term.

Closing in favor of more specific follow-up issues.

@nikhiljindal quick clarification - will kubectl rolling-update remain intact and DeploymentControllers with a "recreate" DeploymentStrategy (redundant in functionality to that of kubectl rolling-update) be separately implemented?