Add Kubernetes-specific headers in apiserver proxy. #4440
Conversation
For example, when accessing the status page of a Kubernetes-aware server, that server may want to link to a particular pod by linking through the same proxy. To do that, the server needs to know: 1) That it's being accessed through apiserver proxy, by looking for the presence of the Kubernetes-specific header(s). 2) The proxy prefix to add to URLs. Note that this should not include the namespace, in case the server wants to link to a different namespace. 3) The namespace of the server currently being accessed, in case it wants to link to another entity in the same namespace. In addition, the apiserver proxy needs to avoid rewriting links that already begin with the proxy prefix that it advertised to the target server through the X-Kubernetes-Proxy-Prefix header.
|
@enisoc Could you please explain more about your use case? In general, it isn't going to be possible to link across namespaces. Note that this also relates to #386. Hitting a pod through the proxy seems like a backdoor method of getting info about itself and about the cluster. I see the point about not rewriting links that are already prefixed, but it seems like there must be a general solution to that. It sounds like we just haven't implemented our reverse proxy correctly. It seems like we should be able to detect pretty easily whether the path is already prefixed or not. |
|
In general our proxy is probably too broad. I don't know that we want a generic http proxy being in common use for use cases that reflect the cluster itself as Brian noted. Why shouldn't the server running on Kube get those settings injected by its own pod setup and then return them? |
|
If it doesn't make sense to link across namespaces, I can simplify this to just one new header: The overall problem I'm trying to solve is this:
So my admin server needs to rewrite this:
to this:
The question then is where does the admin server get all those values (apiserver, version, namespace). I'm open to any suggestions for the right way to do that. For the record, here is the rabbit hole that led me to writing this pull request: Suppose I get {apiserver} from an env var. Then my links are absolute: But, is 1.2.3.4 an externally visible address? It's useless if not. What if the user is accessing apiserver through Conclusion: Omit the host and just write the links as Now suppose we get {version} and {namespace} from env vars and our admin server outputs: But I'm accessing the admin server through the apiserver proxy too, by navigating to:
How does the apiserver proxy know not to re-rewrite the link to: The proxy needs to be smart about looking for any of the valid API prefixes. And my admin server had to decide which version and namespace to use. What if the user accessed the admin server through a different version (say, v1beta4)? Then some of the links on the page (those that were rewritten by apiserver proxy) will use one prefix, and some of the links (those written by admin server itself) will use another prefix. The user will be punted around to different proxy prefixes as they click links, which is not a great experience. In order to minimize the things that both the apiserver and my admin server need to know, and to make the most fluid end-user experience, I proposed that the proxy should just straight-up tell the admin server what prefix it should add if it wants to make other links through the same proxy. Then the proxy only has to worry about avoiding rewrites of that exact prefix. Having the header be Kubernetes-specific also allows the admin server to fall back to direct, non-proxied links if it detects that it's not being accessed through the proxy at all. |
|
As I wrote in #3996, /proxy and /redirect aren't really part of the API, and as discussed #156 (comment), they should really be provided by a separate server. We should change ResourceLocation to a proper polymorphic subresource, not unlike the proposed Once we had that, /proxy, /redirect, /bastion, etc. could be built as a separate service. Rather than sticking /proxy and /redirect in the middle of the API paths, they should be completely separate. I'll think more about how to make the proxying simpler. Is there a problem with always making the console go through the proxy, even when theoretically unnecessary? If within the network, it should be able to redirect rather than proxy. |
|
|
Actually, duh: We have a ResourceLocation resource already: Endpoints. Pods and Nodes should produce Endpoints, just the same as Services. |
|
It seems to me that the issue of where the proxy should live is orthogonal to the purpose of this PR. Any proxy that requires adding a prefix to the URL will have this problem: How can the thing you're proxying to write more links that explicitly go through the proxy? As it stands now, the only possible way is to figure out the external IP of the proxy server (apiserver in this case), because the URL must include an explicit host in order to avoid being rewritten. Figuring out the external IP is tricky, especially if you want to allow things like Once this prefix is sent to the target servers in a header, you can change the prefix however you like without breaking them. |
|
Let's break this into 2 problems:
|
|
Closing since this won't be merged in its current form. |
|
@enisoc I suggest we schedule a GVC to discuss the problems and possible solutions. |
|
@bgrant0607 Sure, feel free to send an invite to my calendar; it's pretty empty. |
For example, when accessing the status page of a Kubernetes-aware
server, that server may want to link to a particular pod by linking
through the same proxy. To do that, the server needs to know:
presence of the Kubernetes-specific header(s).
the namespace, in case the server wants to link to a different
namespace.
wants to link to another entity in the same namespace.
In addition, the apiserver proxy needs to avoid rewriting links that
already begin with the proxy prefix that it advertised to the target
server through the X-Kubernetes-Proxy-Prefix header.