Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: scaling interface #1629

Closed
bgrant0607 opened this issue Oct 7, 2014 · 18 comments
Closed

Proposal: scaling interface #1629

bgrant0607 opened this issue Oct 7, 2014 · 18 comments
Assignees
Labels
area/api Indicates an issue on api area. area/app-lifecycle priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling.
Milestone

Comments

@bgrant0607
Copy link
Member

Auto-scaling has been discussed in many issues/PRs: #1007, #1178, #170, ...

This is about how an auto-scaler should be layered on the API.

In the case of auto-scaling, the currently desired number of replicas won't be statically configured by the user, and should not be among the fields diff'ed by the configuration system. #1201 addresses that problem.

It should be possible to auto-scale applications with multiple release tracks, and it should be possible to auto-scale and perform a rolling update (#1353) simultaneously. If the auto-scaler were tied to a specific replication controller, then it would at least need a label selector (or possibly reference to a service object, but I prefer direct use of label selectors) to identify the whole serving set in order to assess current capacity.

Additionally, if the approach of creating a new replication controller were used to deploy new images, the auto-scaler would need to be updated to resize the new replication controller.

And, if we introduced higher-level API objects to manage multiple underlying objects or other new controller types (e.g., JobController - #1624), the auto-scaler would need to support all those API types.

Instead of declaratively manipulating all these object types directly, the auto-scaler should target a narrower API: an imperative resize "verb" (i.e., POST to a URL). Any API object capable of being auto-scaled (e.g., replicationController) could support that verb. The verb would update the desired replica count for that object. With this approach, resizing could also be supported generically by the CLI (lots of discussion in #1325), would simplify client code more generally, and could simplify authorization logic.

Yes, verbs of all flavors are discouraged by REST purists, but this approach avoids per-object-type verb proliferation and facilitates metaprogramming, addressing their main concrete concerns. The alternative would be a generic declarative subobject containing (essentially) just the size. That seems like a lot of metadata and object-lifecycle overhead for little gain.

Details to be worked out:

  • Whether everyone is ok with URL-encoded API verbs
  • URL format for API verbs generally. Simple strawman proposal: collection or object URL followed by /.verb -- period prefix to avoid collisions with sub-collections/sub-objects (only unescaped punctuation across all RFCs are -, _, and .).
  • Whether we support absolute and/or relative resizing -- either could work in most cases, though relative can be useful for pure reactive behavior
  • Preconditions for resize (e.g., current size), which would be optional (though necessary for idempotence in the relative case)
  • JSON vs. URL parameters for operation parameters. I like URL parameters, since they facilitate simple web hooks.
@smarterclayton
Copy link
Contributor

We already have 3 API verbs URL-encoded - watch, proxy, and redirect. I think I'm ok with a few more.

Regarding relative vs absolute - In practice we started with relative and realized we wanted absolute for idempotence. I would argue relative should be optional and absolute should be the default (like specifying a name is required and generating it is optional).

Is resize limited to a single dimension for all verbs? Is it parameterizable in the future? Or should all objects have a dimensionality of one for resize?

@bgrant0607 bgrant0607 added this to the v1.0 milestone Nov 4, 2014
@bgrant0607
Copy link
Member Author

Verbs: These verbs appear early in URL paths, preceding the resource collections (kinds), namespaces, and names?

Relative vs. absolute: I agree.

Dimensionality: I was only thinking about a single dimension: horizontal scaling. What I imagine is that an autoscaler would watch a set of pods and invoke a resize operation. It wouldn't need to understand what was controlling the pods or what was implementing resize. It might be a single replication controller, a replication controller controller, replication and nominal service controller, a multi-release deployment controller, a job controller, etc. However, it would need to have an expectation that tweaking resize would change the number of monitored pods accordingly.

I consider vertical resource resizing (cpu, memory, etc.) of pods to be a separate issue (#2072), one which is more complex not only due to multiple dimensions, but also due to pod resources vs. container resources, request vs. limit, in place update vs. rescheduling, and other issues.

@bgrant0607 bgrant0607 added the priority/backlog Higher priority than priority/awaiting-more-evidence. label Dec 3, 2014
@derekwaynecarr
Copy link
Member

I am ok with this approach if the resize verb is something implemented by a runtime.Object, and not that a RESTStorage resize operation is authored. I want to make sure we have a single choke point for intercepting all Update operations as part of my ability to weave in admission control. I have some other concerns on a resize verb and policy sychronization.

@derekwaynecarr
Copy link
Member

On policy, it sounds like I need to separately manage /resize/ns/{namespace}/object/name from /ns/{namespace}/object/name. That kind of complicates the system in my opinion so we should be careful before taking on new verbs without thinking it through.

@bgrant0607
Copy link
Member Author

I was thinking of something like POST .../ns/{namespace}/{kind}/{name}/op/resize

That would make it easier to delegate the handling of special verbs to each registry object, and for the set of verbs to be discoverable. This is sort of the approach used by GCE, though I don't think it puts verbs into a sub-collection.

@smarterclayton
Copy link
Contributor

Structurally, it would be better to watch via GET /ns/kind/obj/name/op/watch because then watch can be properly sharded

On Dec 17, 2014, at 7:42 PM, bgrant0607 notifications@github.com wrote:

I was thinking of something like POST .../ns/{namespace}/{kind}/{name}/op/resize

That would make it easier to delegate the handling of special verbs to each registry object, and for the set of verbs to be discoverable. This is sort of the approach used by GCE, though I don't think it puts verbs into a sub-collection.


Reply to this email directly or view it on GitHub.

@brendandburns
Copy link
Contributor

+1 to this proposal in general.

Caveats:
1)
We need to support a resource version on this request to prevent read -
update - write races.

This is likely to make client codegen harder (not that we're doing it
today...)
We should look at what (if anything) swagger supports in this space.

I think we should intercept these even higher in the stack, likely the
apiserver itself, and handle them in with a VerbHandler interface or some
such that parallels RESTStorage.

I don't think we should support incremental resize, as it seems inherently
racy.

Brendan
On Dec 17, 2014 5:36 PM, "Clayton Coleman" notifications@github.com wrote:

Structurally, it would be better to watch via GET
/ns/kind/obj/name/op/watch because then watch can be properly sharded

On Dec 17, 2014, at 7:42 PM, bgrant0607 notifications@github.com
wrote:

I was thinking of something like POST
.../ns/{namespace}/{kind}/{name}/op/resize

That would make it easier to delegate the handling of special verbs to
each registry object, and for the set of verbs to be discoverable. This is
sort of the approach used by GCE, though I don't think it puts verbs into a
sub-collection.


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub
#1629 (comment)
.

@bgrant0607
Copy link
Member Author

This proposal is really about an abstract interface for resizable entities.

So, to put forth a more RESTful alternative -- feel free to shoot it down:

We could create a new virtual REST object:

type Scaler struct {
    TypeMeta   `json:",inline"`
    ObjectMeta `json:"metadata,omitempty"`
    Spec ScalerSpec `json:"spec,omitempty"`
    Status ScalerStatus `json:"status,omitempty"`
}
type ScalerSpec struct {
    Count int `json:"count"`
}
type ScalerStatus struct {
    ScaledObjectRef *ObjectReference `json:"scaledObjectRef,omitempty"`
    Count int `json:"count"`
}

Any object that wanted to be scalable, such as ReplicationController, would automatically create a Scaler object. Modifications to the Scaler object would in turn modify the corresponding ReplicationController.

This would achieve the goal of making the auto-scaler not need to understand the full details of all types of controllers that wanted to be scalable.

Note that etcd is adding multi-object transactions, which should make such multi-object "macros" easier to manage.

@bgrant0607 bgrant0607 changed the title Proposal: resize verb for auto-scaling Proposal: resize interface for auto-scaling Dec 18, 2014
@smarterclayton
Copy link
Contributor

Or similarly, expose

/replicationcontrollerscalers/<name of rc>

type Scaler struct {
  Metadata struct {
    ResourceVersion
  } 
  Spec struct {
    Count int64
  }
  Status struct {
    Count int64
  }
}

which supports GET / PUT.

I don't think /replicationcontrollers/<name>/op/scaler is more or less special than /replicationcontrollerscalers/<name> - all that really matters is can a client easily figure out whether X has a scaler, and then do something consistent to it. That argues a bit more for <resource>/.../op/<op>, because as a client I need to know these things:

  • What I want to scale (replicationcontroller)
  • What is its current scale
  • What do I want its current scale to be

So the mechanical transformation you want is

  • Given resource name, check whether it has scaler
  • Given scaler, "get" its current scale
  • Given scaler, "update" the current scale

Unlike spec/status sub objects, there's no real need to mutate the metadata at the same time as the scaler. So having a totally decoupled resource representation in the API makes sense.

----- Original Message -----

This proposal is really about an abstract interface for resizable entities.

So, to put forth a more RESTful alternative -- feel free to shoot it down:

We could create a new virtual REST object:

type Scaler struct {
  TypeMeta   `json:",inline"`
  ObjectMeta `json:"metadata,omitempty"`
  Spec ScalerSpec `json:"spec,omitempty"`
  Status ScalerStatus `json:"status,omitempty"`
}
type ScalerSpec struct {
  Count int `json:"count"`
}
type ScalerStatus struct {
        ScaledObjectRef *ObjectReference `json:"scaledObjectRef,omitempty"`
  Count int `json:"count"`
}

Any object that wanted to be scalable, such as ReplicationController, would
automatically create a Scaler object. Modifications to the Scaler object
would in turn modify the corresponding ReplicationController.

This would achieve the goal of making the auto-scaler not need to understand
the full details of all types of controllers that wanted to be scalable.

Note that etcd is adding multi-object transactions, which should make such
multi-object "macros" easier to manage.


Reply to this email directly or view it on GitHub:
#1629 (comment)

@pweil-
Copy link
Contributor

pweil- commented Dec 18, 2014

I agree with @smarterclayton that <resource>/.../op/<op> seems like the best option. Exposing a resize operation is exposing the Resizable object, not the scaler itself. The scaler is simply a boundary definition that triggers resize actions on a Resizable target.

@goltermann goltermann removed this from the v1.0 milestone Feb 6, 2015
@bgrant0607 bgrant0607 removed this from the v1.0 milestone Feb 6, 2015
@bgrant0607 bgrant0607 added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. status/help-wanted and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Feb 28, 2015
@davidopp
Copy link
Member

davidopp commented Mar 3, 2015

Could someone explain the resize verb concept a little more? What object is the target? From some of the examples above it seemed the target was a pod (for example), but I don't understand what it means to horizontally resize a pod (wouldn't you just create a copy of it, or delete it)? Or maybe the target is some kind of meata-object?

@davidopp
Copy link
Member

davidopp commented Mar 3, 2015

Sorry for being dense, I just figured out that the replication controller is the object with a resize verb. That makes sense.

@smarterclayton
Copy link
Contributor

#5054 contains some fixes required to allow sub resources like /resize to work, as well as an example with /binding. I made "Binding" generic to type, (it's a name, target, and an object reference to be bound to).

@bgrant0607
Copy link
Member Author

Note that the current proposal is /size. See #2726 (comment)

@bgrant0607
Copy link
Member Author

After further thought, I'm thinking we should call this /scale, and rename kubectl resize to kubectl scale, in order to match standard terminology (auto-scaling), to distinguish from cpu/memory resizing/auto-sizing, and to be more similar to other systems (e.g., CF).

@bgrant0607
Copy link
Member Author

In progress: #12217

@bgrant0607 bgrant0607 added this to the v1.1 milestone Aug 7, 2015
@bgrant0607 bgrant0607 added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. team/api and removed help-wanted priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Aug 7, 2015
@stormltf
Copy link

when atuoscaling can be done? do you need our team help to do this?

@fgrzadkowski
Copy link
Contributor

Several issues here:

I think we should close this particular issue as it seems it's redundant.

bertinatto pushed a commit to bertinatto/kubernetes that referenced this issue Jul 14, 2023
…-fix

OCPBUGS-15726: UPSTREAM: 118881: fix openapi/v3 non local apiservices aggregation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api Indicates an issue on api area. area/app-lifecycle priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling.
Projects
None yet
Development

No branches or pull requests