Why do we do this?
I've seen it in a lot of places, but I don't understand why we need it here...

I've seen it in a lot of places, but I don't understand why we need it here...

It allows you to decode the options for requests send to your group. The cross registration makes it a valid conversion target.

nit: SortedByGroup.

It's not in place.

not only host verification, also the whole cert check.

which will be used

Isn't this inherited from TypeMeta?

all?

Which means you don't want to allow root cert bundle verification of non-self-signed certs?

Which means you don't want to allow root cert bundle verification of non-self-signed certs?

fixed

Shouldn't this be a list?

Shouldn't this be a list?

I don't think we want to make this a list. Thinking about how I want to roll out a second version, I don't want to risk my v1 server to run a v2 server. That means I'm looking at a side-by-side during the transition period, which means GroupVersion granularity instead group granularity.

does "side-by-side" mean more than one server could serve the same group/version?

do we want to have to repeat the whole object with all the server-specific bits if a single server is serving multiple groups and/or versions?

does "side-by-side" mean more than one server could serve the same group/version?

No. One service for v1, another service for v2.

do we want to have to repeat the whole object with all the server-specific bits if a single server is serving multiple groups and/or versions?

Yes. There's no reason to believe that things like priority order would be the same for every GroupVersion being served.

does "side-by-side" mean more than one server could serve the same group/version?

No. One service for v1, another service for v2.

how do you plan to enforce that in case of conflicts? oldest wins?

Sure, you could do it that way, but I don't think we should force people to do it that way.

Since the name of the object is forced to be version.group (see validation rules), you can't end up with two trying to run the same API version.

Sure, you could do it that way, but I don't think we should force people to do it that way.

I think that the consistent rule of "one groupversion, one APIServer resource" is very easy to describe, enforce, prioritize, and avoid conflicts for. When you start combining them, you open the API enough to be self-contradictory. I don't think that avoiding granular objects is worth it. I don't think its likely to be very onerous and prescriptiveness can help tooling built around it.

As long as it's explicitly acceptable to point multiple of these records at the same backing service, I guess I'm OK with this. It does mean there'll be more configuration work involved.

(I was actually expecting to split only by group. I guess it does seem valuable to host different versions separately in some situations, but is it really necessary?)

(I was actually expecting to split only by group. I guess it does seem valuable to host different versions separately in some situations, but is it really necessary?)

This came out of my first encounter with the enemy as I was dealing with an openshift project API that had one version hosted in openshift and another hosted in a third server that works against bare kube. It got me thinking about what I wanted to do in an upgrade scenario and I realized I wanted to avoid risking my v1 server.

This actually (potentially) represents a APIServer replica pool in HA setups. Name is misleading.

So maybe APIService{Spec, Status} is a better name.

So maybe APIService{Spec, Status} is a better name.

doing

If it's a Host:Port pair, call it an address.

But why present it like this? Why not make this take a service name?

But why present it like this? Why not make this take a service name?

I wasn't prepared to say that it must point to a particular service. An address allows an external reference more easily and some of the first that I wanted to federate weren't in-cluster to start.

That's a fair point, but you can represent such things as a service by adding the endpoint(s) yourself. I think referencing a service will let us understand the topology in a way that a single address will not.

That's a fair point, but you can represent such things as a service by adding the endpoint(s) yourself. I think referencing a service will let us understand the topology in a way that a single address will not.

So you'd like a service reference plus a port?

If you think it's too limiting to ask everyone to use 443, then I guess we need a port.

If you think it's too limiting to ask everyone to use 443, then I guess we need a port.

As a service, I guess there's not a good reason to allow anything else.

But why present it like this? Why not make this take a service name?

doing

Why does this and the next field need to be in this type?

Why does this and the next field need to be in this type?

As a summarizer, the discovery server accepts the remote TLS connection on your behalf.

Why would that be configurable per-apiserver?

Why would that be configurable per-apiserver?

You could have different custom CAs per-apiserver. It's less likely, but still possible that you have a few black sheep (unverifiable) amongst the rest.

Anyway, I would say that the summarizer "owns" the discovery paths.

Oh, I see. And you can't do it the ordinary way because you want it to be dynamic?

What do you think about handling that with a separate resource entirely, or even omitting it in the first draft. I expect for most people this will be unnecessary, and those who do have multiple CAs will still have only a few--CAs won't be 1:1 with apiservers.

CAs won't be 1:1 with apiservers

in the simplest case (each one self-signs a cert), they are

Oh, I see. And you can't do it the ordinary way because you want it to be dynamic?

Right, and potentially per-api server if someone is creating "burn after signing" keys (instead of self-signing).

What do you think about handling that with a separate resource entirely, or even omitting it in the first draft. I expect for most people this will be unnecessary, and those who do have multiple CAs will still have only a few--CAs won't be 1:1 with apiservers.

In openshift, I'd agree since we have a service serving cert signer infrastructure, but we don't have that here and I don't know that I think the same signing key will be used for every service. I hit this almost immediately when I tried to use this API with a "simple" server I built from the generic API server, though I will admit that was using the self-signed startup flow.

How about making the field optional and the summarizer/proxier could also accept a CA bundle? Insecure trumps (I think I still want this per server), apiserver.spec.CA next, CA bundle, then system CAs.

The only use case I can see for that is a short dev/test cycle for apiservers themselves. I think in any real cluster, the sysadmin is going to want everything signed by the same CA? Is there really a good use case for multiple CAs?

I mean, aren't we adding a certificates resource, so it should be easy to get a cert signed by the main apiserver?

I mean, aren't we adding a certificates resource, so it should be easy to get a cert signed by the main apiserver?

Not in the short term. Signing a serving certificate, even one for service, requires having strict rules about re-use of namespaces and the like. I wouldn't make plans like this based on it yet. In addition, it itself would be an extension API server.

Starting with a way to describe, "this is how you trust me" seems very reasonable. I'm also willing to provide federation server level CA options. When we move out of alpha we could decide if its been useful.

This seems like an administration nightmare. I would prefer to remove this. We can add it in later if it is really necessary. We can hardcode a "first-class citizen" group list for now.

This seems like an administration nightmare. I would prefer to remove this. We can add it in later if it is really necessary. We can hardcode a "first-class citizen" group list for now.

Leaving it empty just means that secondary priorities take over. We hit this immediately with openshift types laid on top of kube types. Names collide and we need to pick priorities. I don't think that the winners should be hardcoded and impossible to change. Strong default resource manifests seem reasonable (kube resources come in as 1 by default), but I need to be able to change them.

Allowing sysadmins to adjust it means kubectl will function differently on different clusters. Big user surprise. I really think we should come up with a better system.

Allowing sysadmins to adjust it means kubectl will function differently on different clusters. Big user surprise. I really think we should come up with a better system.

Pushing it to the client means that a cluster-admin can't choose which groups he wants to have priority in his cluster. Imagining two groups with Roles. I don't think that a client should choose which one of the Roles takes priority by default. As an explicit choice certainly (we support this today), but by default I think that's unexpected. I think its a better idea for a cluster-admin wants to control resource preference in his cluster and a client can choose to be explicit (best practice recommended today).

I didn't say we should push it to the client, I agree, that makes it worse. I do think the resolution order should be a fixed property of the GVKs involved, not a property of the configuration in the discovery merger.

I didn't say we should push it to the client, I agree, that makes it worse. I do think the resolution order should be a fixed property of the GVKs involved, not a property of the configuration in the discovery merger.

I don't wouldn't expect that priority to be consistent across all clusters, so we can't bake it into code. If we bake it into code, everyone will choose negative infinity or whatever the highest priority is. Where would it live if not here and not in code?

I'm still thinking about this. I agree there's a problem, but I don't like any solution so far. The solution in this PR is high-cognitive overhead for cluster administrators. I would prefer a solution that "just works" but without hard coding something or a central registry somewhere, I haven't thought of anything yet.

I'm still thinking about this. I agree there's a problem, but I don't like any solution so far. The solution in this PR is high-cognitive overhead for cluster administrators. I would prefer a solution that "just works" but without hard coding something or a central registry somewhere, I haven't thought of anything yet.

I think that this, in combination with the bootstrapping described here: #37650 means that the default scenario results in no additional work for a cluster-admin, but the power is here when you need it and we have scenarios for it already.

I mean, even the base kube resources rely on priority to work since resources are/were cross registered under extensions and other groups like autoscaling or batch. Those only work today because it sorts right alphabetically.