Questions-- what does this get us over just creating multiple APIServers in pkg/master?

Thoughts on seeing a new Operations object per group-- I think operations should get their ID by creating an object in etcd. I feel like an operation is global to the cluster, no matter which api group caused it. Just like, if we served pods in multiple API groups, they'd still be the same pods. That said, we do eventually need a story around who can see what operations...

So the context here is we want to offer api operations that are related to Kubernetes, but not part of the core api. We want to do that on the same port (so we don't have to secure multiple ports to clients). In my head, that seems to me like breaking APIServer up so that the "support" operations (which you only need one per port) from the "version of the api" code (APIGroup, which you will end up needing multiple of). The next step then is to make it possible for someone to run the master and have all of that code work cleanly, and then to be able to use the API operations that are exposed by that server cleanly. So moving httpserver out of master reduces the coupling between master and a specific http server (what if I wanted TLS, not HTTP? Different time-outs? etc), and also means that a consumer can start servers with multiple api groups.

Multiple APIServers still is combining one "support" group for each version of the API. At some point, an old resource object is going to be deprecated and then removed in a new api v�ersion - I think APIGroup would allow that more cleanly than the current code. The master's API_version call would include a different storage map and codec for that server, but consumers would just construct the handler for it along with the others and tie them into a single mux, passing a clear prefix for each. Eventually, downstream consumers may not want to support all the older API versions, so being able to choose outside of master what API versions are exposed is valuable.

On operations per group: operations depend on the codec for serialization, so implicitly "all of the operations managed by this group have to be serializable in the same way". Across API versions of the same object group (core api, extension api, plugin api), those operations are consistent. But across different sets of api objects that want to be consistent with Kubernetes API patterns, it may not be. I think clients will have to be aware of accessing resource foo on api prefix /bar/v1/foo that they'll have to get operations from /bar/v1/operations, instead of /api/v1beta1/operations.

Agree operations feel global to a particular set of api objects, although creating an object in etcd is going to double write load on etcd for a feature that is not terribly valuable (given that the vast majority of operations should be very short, and in some cases we've got agreement we're waiting too long). I can see the argument for operations on pods being the same across different api servers, but at the same time, the client only knows about the apiserver it's talking to (under a LB or not). I don't think the client has to know about all possible sets of apiservers as long as the one he's talking to can return consistent operation values. We can push a bit more work back to the clients there by having them retry 404 responses for ops (for stateless load balancing), or we could ensure each apiserver backend has a DNS entry that it knows about (api1.bar.baz.com) and have it return a Location header with 202 accepted requests. That's a bit more work to set up, but then clients can just follow location, and if the server dies, the operations are technically dead anyway. I don't know if there's a use case for really long running apiserver operations that are retried by the server on restart - I'd prefer to push that to the client with proper idempotency and retry logic than requiring statefulness on the server.

This LGTM (and will be useful for Kubelet too).

I'd second the thought that writing operations to etcd seems like overkill, and the work involve to resuscitate an ongoing operation in a different API server is going to be painful, and possibly unnecessary.

I'd prefer to see a scheme where the multiple API servers know about each other, and we use sharding and 302's to forward the "get operation" request to the appropriate API server, rather than trying to track state in persistent storage.

(that said, I think we should log API requests for posterity/tracking, but that's a different PR...)

This needs a rebase, tho'

Hm... Effectively, this position means that it's the client's responsibility to check to see that the server actually finishes a task, even if the server responded with a 202 accepted. As long as we're OK with that, I'm OK not writing operations to etcd.

However, even for the scheme where api servers know about each other, we still need some sort of system for uniquely identifying operations for any sort of forwarding contraption to work. I guess you can namespace or do uuids.

@lavalamp yeah, I think the security aspects of operations are interesting as well - i.e. how much security context do we need to store to make it work. A cheap way to do that in the short term is to remove operation listing and use a UUID for an operation (effectively unguessable unless you can sniff traffic, in which case you already have the security credentials).

Rebased