Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial StatefulSet updates proposal #503

Merged
merged 12 commits into from
May 27, 2017
Prev Previous commit
Next Next commit
typos
  • Loading branch information
Kenneth Owens committed Apr 6, 2017
commit d5d445a5e79814a9011f6cdd21e027a5f8b7393f
5 changes: 2 additions & 3 deletions contributors/design-proposals/statefulset-update.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,6 @@ The following modifications will be made to the StatefulSetSpec API object.
type StatefulSetSpec struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need something like update strategy?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't that be similar to what deployments currently provide?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are going to add a struct as per @smarterclayton's suggestion

Copy link
Contributor

@0xmichalis 0xmichalis Apr 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also add a rollback spec and a rollback subresource, otherwise every client needs to implement its own logic for rollback.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kow3ns what is the conclusion here ? I am guessing we all are in agreement , that we need a struct for strategy and rollbackConfig

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For rollback, if we are worried about spec mutation (the controller cleans up the rollback spec once it successfully rollbacks a controllee), could we still have a server-side implementation w/o using a struct in the spec (the rollback is performed by the api server and not the controller)?

// Replicas, Selector, Template, VolumeClaimsTemplate, and ServiceName
// omitted for brevity.
v1.PodTemplateSpec `json:"template"`

// TemplateGeneration is a monotonically increasing, 64 bit, integer used to
// indicate the version of the of the PodTemplateSpec. If nil, the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which component is going to increase this field?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API Server

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why isn't this a generation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will update for consistency with DaemonSet.

Expand Down Expand Up @@ -255,7 +254,7 @@ the largest ordinal will be deleted.

### StatefulSet Revision History
The StatefulSet controller will use labeled, versioned PodTemplates to keep a
Copy link
Contributor

@lukaszo lukaszo Apr 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a final decision? In DaemonSets we didn't decided yet how to store a history. cc @Kargakis @janetkuo @smarterclayton

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we haven't yet. I think we need to get a quick list of options written up and discussed:

  1. PodTemplate (collision between use for different controller types)
  2. Specific type for each controller: DaemonSetHistory, StatefulSetHistory, etc (no reuse between controllers)
  3. Storing a limited subset of specs on the object (size limits)
  4. ???

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm obviously partial to 1. My primary reason is that controllers should not be special and require magic. If we get TPRs right then PodTemplates can be leveraged by arbitrary operators/controllers, and they, therefore, provide a primitive upon which others can build.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for pod template here's what we need to handle:

  1. Collisions across controllers by naming
  2. Idempotency of product versions (the same pod template in a controller object should map to exactly one PodTemplate across multiple upgrades)
  3. Human readability - a human should be able to find a PodTemplate
  4. Collisions across controllers by pod template spec - if two controllers deploy the exact same pod template (daemon set vs stateful set) the PodTemplate has two owners, not just one, so garbage collection won't work
  5. Future versions of controllers may involve renames of the controller resource name, so we would still have to be able to locate old PodTemplate revisions even if the resource name changes
  6. Immutability of PodTemplate - a template used in this fashion needs to prevent updates, or at least strongly discourage them (if they are mutable, we can't compare the pods against the pod template version).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the real issue is 1 - the rest of the problems apply to ReplicaSets owned by Deployments too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For clarity, I am talking about the UID of the parent.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smarterclayton @janetkuo @Kargakis @lukaszo

Before we go forward with PodTemplate creation we should also discuss adding a History sub resource to the controllers (I believe that is something like option 3 above).

Having a per Kind History sub resource mitigates History overlap across Kinds with the same .Name in the same namespace. My argument about PodTemplate being a primitive that is usable without controller specific magic is valid (imo), but perhaps its also a straw man.

Right now the category of controllers could loosely be defined like this.
A generative controller takes, as an argument, an Object that represents the declared target state. The Object must have the following properties

  1. A .Spec.Template that is a template for Objects of the generated Kind.
  2. A .Spec.Replicas that expresses the desired cardinality of the generated Kind.
  3. A Generation that indicates the version of the declared target state.
  4. A .Status.Replicas that indicates the number of created replicas.
  5. If the Objects of created Kind are not immediately available upon creation, a .Status.ReadyReplicas to indicate the number of available created resources.
    1. A .Satus.ObservedGeneration that indicates the version of the target state specification that the controller has observed.

I confirmed with @foxish that the plan to support third party controller and operators is to make these properties well supported. That is, we want them to create custom Kinds that contain these sub resources. Given that, adding a History sub resource might be the best fit.

We could say that controllers that implement revision history must have

  1. A History.Revision where the type of this object is an []PodTemplate or []PodTemplateSpec.
  2. A History.RevisionLimit that indicates the number of stored revisions.
  3. A Spec.TemplateGeneration that is incremented by the API server upon mutation of the .Spec.Template.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There has been some discussion before and I think there is general agreement that we want a history subresource. I am not sure whether we need to have this discussion in this proposal or in a separate one since it 1. affects all of the workload controllers and 2. is not required for having StatefulSet upgrades.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a problem with putting uids in names of generated resources provided they are ONLY used for uniquification and NEVER used for discovery/searching.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prevention of mutation and deletion is also desired in other contexts (e.g., addons): kubernetes/kubernetes#10179

history of updates preformed on a StatefulSet. The number of stored PodTemplates
history of updates performed on a StatefulSet. The number of stored PodTemplates
is considered to be the size of the StatefulSet's revision history. The
maximum size of a StatefulSet's revision history is two (these are the current
and target PodTemplates) plus the history limit (represented by its
Expand Down Expand Up @@ -621,7 +620,7 @@ spec:
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
generationParition: 2
generationPartition: 2
volumeClaimTemplates:
- metadata:
name: www
Expand Down