@smarterclayton thoughts?

Interesting. @ncdc's work with http upgrade to TCP via some protocol would eventually let that and normal proxy talk TCP through the bastion and still do security. I don't see a downside other than naming and trying to figure out whether you have access to the bastion (since we can't do a reverse lookup of IP to pod to namespace today). Maybe just replace IP with name of pod and make pods support proxy?

Maybe just replace IP with name of pod and make pods support proxy?

This already works today, actually.

What about port selection?

On Jan 14, 2015, at 5:55 PM, Daniel Smith notifications@github.com wrote:

Maybe just replace IP with name of pod and make pods support proxy?

This already works today, actually.

—
Reply to this email directly or view it on GitHub.

.../proxy/pods/podname:port/... actually works!

I love it when a plan comes together

On Jan 14, 2015, at 6:54 PM, Daniel Smith notifications@github.com wrote:

.../proxy/pods/podname:port/... actually works!

—
Reply to this email directly or view it on GitHub.

The original reason I asked about proxying by IP was for the case of a non-k8s-aware app that generates links to other pods by their IPs, but doesn't know how to refer to pods by pod name. As an alternative, I can adapt the app to be k8s-aware enough to link through the apiserver proxy by pod name. However, then I came upon the need for something like #3484.

Yeah, and the more I think about it the more I think that's a valid use case.

How does the non k8s-aware app find the pod ips?

On Jan 14, 2015, at 7:28 PM, Anthony Yeh notifications@github.com wrote:

The original reason I asked about proxying by IP was for the case of a non-k8s-aware app that generates links to other pods by their IPs, but doesn't know how to refer to pods by pod name. As an alternative, I can adapt the app to be k8s-aware enough to link through the apiserver proxy by pod name. However, then I came upon the need for something like #3484.

—
Reply to this email directly or view it on GitHub.

Each server is given the address of an etcd service on the command line. The server resolves its own IP and registers itself on the specified etcd service. Then the management interface reads the IPs out of etcd and links to them.

I guess we eventually need to support efficient pod access by currentIP, but it certainly complicates the authz story.

@erictune this is a case where the resource identifier is insufficient to properly scope a request, so the indirection has to happen inside RESTStorage before the authz check can happen.

Also, what are the performance implications of doing this lookup and that sort of check on every single http request? We can't easily cache the acl decision above the storage layer in this case, so we'd have to cache both lookup and decision.

I can see the value in the scenario, but definitely is complex in implications.

On Jan 14, 2015, at 8:42 PM, Anthony Yeh notifications@github.com wrote:

Each server is given the address of an etcd service on the command line. The server resolves its own IP and registers itself on the specified etcd service. Then the management interface reads the IPs out of etcd and links to them.

—
Reply to this email directly or view it on GitHub.

Why do we need this? Pods and Services are supported. Stop.

This broad issue was previously discussed in #1863, and more recently in #3996.

Probably we should make this one about reverse lookup by IP address.

There are definitely authz and sharding implications for reverse lookups across namespaces.

Reverse lookup in APIserver makes sense. It is just a query against the data in the apiserver.

Proxying, while possible to do in apiserver, I think belongs in a separate process.

• having a separate bastion process allows removing a cyclic dependency between kubelet and master, which may improve system layering. See Remove need for the apiserver to contact kubelet for current container state #156 (comment)
• a more sophisticated bastion than we may want in apiserver will be needed for SSH in many cases.
• in general, this seems like an area where organizations may want to do a lot of customization of bastions. Better to keep that level of complexity and customization out of apiserver. For example, a bastion seems like a natural place to put in detailed auditing and deep packet inspection for organizations that care about that.
• Having the bastion as a separate process gives flexibility in custom configuring networking around it.

+1

----- Original Message -----

Reverse lookup in APIserver makes sense. It is just a query against the data
in the apiserver.

Proxying, while possible to do in apiserver, I think belongs in a separate
process.

• having a separate bastion process allows removing a cyclic dependency
  between kubelet and master, which may improve system layering. See
  Remove need for the apiserver to contact kubelet for current container state #156 (comment)
• a more sophisticated bastion that we may want in apiserver will be needed
  for SSH in many cases.
• in general, this seems like an area where organizations may want to do a
  lot of customization of bastions. Better to keep that level of complexity
  and customization out of apiserver. For example, a bastion seems like a
  natural place to put in detailed auditing and deep packet inspection for
  organizations that care about that.
• Having the bastion as a separate process gives flexibility in custom
  configuring networking around it.

---

Reply to this email directly or view it on GitHub:
#3481 (comment)

+1 to separate process.

Discussion #4440 re. proxy and redirect is relevant.

We have started discussing this (@liggitt has incoming recommendations to the securing the kubelet) and have some recommendations. Concrete proposal:

• A master can be a bastion while still functioning as a master
  • It should be possible to run the master as a bastion but without any other logic (until the masters are stateless or have leaders)
  • A bastion authenticates requests the same way that the API requests results, so....
• Endpoints that are bastioned should be exposed as api resources
  • Pod logs should be GET /api/v1beta3/namespaces/default/pods/logs with support for the same arguments that the kubelet exposes
  • Remote execution should be POST /api/v1beta3/namespaces/default/pods/exec with support for the options on exec
  • They could also be exposed as new resources, but I think these are natural operations on pods.
  • I don't see value in inventing a new API convention just to serve these.
• We should not allow naked proxies for anything except admins (/proxy/* is an admin level access)
• In the short term, the bastion would authenticate to the kubelet using a high level identity. In the future, we'd like to do delegation.

There would be a new API REST helper that handles proxying, supports connection upgrade, can do security delegation. Coincidentally, we have one of these in OpenShift we use to talk to Kuberetes, so we can back port it.

/logs and /exec are extremely common, so specialized API subresources seem fine for those.

Internally, it's also very common for app admins/ops and devs (when debugging problems in production) to access status, stats, profiling, etc. URLs on application instances. Some apps also have control interfaces accessible via HTTP. Those ports/paths shouldn't be available via the normal public/external IP:ports.

On Mar 6, 2015, at 1:38 AM, Brian Grant notifications@github.com wrote:

/logs and /exec are extremely common, so specialized API subresources seem fine for those.

Internally, it's also very common for app admins/ops and devs (when debugging problems in production) to access status, stats, profiling, etc. URLs on application instances. Some apps also have control interfaces accessible via HTTP. Those ports/paths shouldn't be available via the normal public/external IP:ports.

/pods/foo/proxy?port=80

Proxy as a subresources for pods is strictly better than the global verb I think.

—
Reply to this email directly or view it on GitHub.

@csrwng is going to start working on this.

#5763 is setting the stage for this work.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale