FTR, storage class supports multiple provisioners

Another issue in this space: TPR is namespaced, while not all extension controllers are (example: storage class is global)

I think this is critically important. Allowing custom extension of core API concepts makes those API concepts more important because clients can revolve around them even if they don't know what they do.

I.e. an HPA object that points to a custom metric that the client doesn't understand can still provide value for a UI or CLI, because it can show that metric to a human who can understand it. The UI or CLI can delete that HPA (stops scaling) or "pause" the autoscaler, or set limits, or allow generic metrics setting.

Having the standard API object that people can coordinate around is to me the best outcome for Kubernetes - giving people room to provide alternate reactions to standard API objects.

I think Annotation is useful for users defining their own need flexibly, like strategy, attribute and so on. It can reduce the work load to maintain them as API Resource type in types.go. However, the current map[string]string Annotation key-value pair also has its limitations, it's unable to describe more complicated scenarios.

@smarterclayton It sounds like what you're talking about might be closer to the "opaque integer resource" idea (#28312) whereas I think @bgrant0607 was talking more about some kind of delegation or partitioning mechanism to direct requests to the proper instance of something that has multiple instances (most likely one "out-of-the-box" instance and one or more custom instances). Or do you see these are more similar than I do?

In the long run, I think all APIs that need this capability should use fields. Annotations aren't discoverable.

Perhaps we should even ask the question what resources won't ever need this?

I mentioned HPA because while discussing it it was clear the pattern for custom metrics resembles - core API object, many controllers, preferred controller. An object that can be used by many clients to coordinate may not understand all the clients, and while we might have a first party field (CPU) a totally generic field (GenericMetricReference) which is opaque to clients can actually be the signal to the controller to act on it.

Ie the presence of generic metric "biz.metric.1023" is a signal to an autoscaler extension controller in the same way that scheduler annotation may put you into a queue on an extension scheduler. That controller would be outside the system - HPA controller would observe the totally generic metric (as distinct from heapster opaque generic metric) and do nothing.

cc @erictune

Other recent examples where this would be useful:

• Service broker, being discussed as part of the service catalog SIG
• Service account token generation, for generating other types of tokens
• DaemonSet updates ([WiP] DaemonSet updates #31693), similar to the Deployment use case

cc @mfojtik

Making a high-level proposal in the core/layers doc extensions section:
https://docs.google.com/document/d/1XkjVm4bOeiVkj-Xt1LgoGiqWsBfNozJ51dyI-ljzt1o/edit#

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

/remove-lifecycle stale
/lifecycle frozen

/priority backlog

/remove-priority important-soon

We have a pretty well established pattern based on CRDs now. Closing old issues is fun :)