"kubectl exec" does not work through "kubectl proxy" #32026

bronger · 2016-09-03T10:22:55Z

I use kubectl v1.3.5 and v1.2.0 in the k8s cluster. If I proxy to the cluster with

$ kubectl proxy --port=8888

I get the following error when trying to execute a command in a container:

$ kubectl --server=http://localhost:8888 exec -ti test-pod bash
error: unable to upgrade connection: <h3>Unauthorized</h3>

Other commands like get pods work fine. The server is contacted by the poxy with https://kubmaster.****.de:443 and a bearer token.

The text was updated successfully, but these errors were encountered:

nhlfr · 2016-09-06T15:20:44Z

It's an expected behavior - by default, we don't allow to attach or exec via kubectl proxy.
You may use kubectl proxy --disable-filter=true.

However, that brings an another bug:

$ kubectl --server=http://localhost:8001 exec -ti example bash
Error from server: Upgrade request required

@kubernetes/sig-api-machinery do we care about this?

smarterclayton · 2016-09-06T15:49:26Z

Probably. What's the failure cause?

On Tue, Sep 6, 2016 at 11:20 AM, Michal Rostecki notifications@github.com
wrote:

It's an expected behavior - we don't allow to attach or exec via kubectl
proxy.
You may use kubectl proxy --disable-filter=true.

However, that brings an another bug:

$ kubectl --server=http://localhost:8001 exec -ti example bash
Error from server: Upgrade request required

@kubernetes/sig-api-machinery
https://github.com/orgs/kubernetes/teams/sig-api-machinery do we care
about this?

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#32026 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p2JAimUgwQPQi5Xt4wqoR3eEWMshks5qnYTbgaJpZM4J0Rh_
.

ncdc · 2016-09-06T16:04:05Z

Last time I checked, I think the code that backs kubectl proxy isn't
upgrade aware.

On Tue, Sep 6, 2016 at 11:50 AM, Clayton Coleman notifications@github.com
wrote:

Probably. What's the failure cause?

On Tue, Sep 6, 2016 at 11:20 AM, Michal Rostecki <notifications@github.com

wrote:

It's an expected behavior - we don't allow to attach or exec via kubectl
proxy.
You may use kubectl proxy --disable-filter=true.

However, that brings an another bug:

$ kubectl --server=http://localhost:8001 exec -ti example bash
Error from server: Upgrade request required

@kubernetes/sig-api-machinery
https://github.com/orgs/kubernetes/teams/sig-api-machinery do we care
about this?

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/kubernetes/kubernetes/issues/
32026#issuecomment-244986068>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABG_
p2JAimUgwQPQi5Xt4wqoR3eEWMshks5qnYTbgaJpZM4J0Rh_>
.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#32026 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAABYkXSy3eJzGt9xMPaXHvofheVRu3Pks5qnYu8gaJpZM4J0Rh_
.

smarterclayton · 2016-09-06T17:06:10Z

We have an upgrade aware proxy in some places. If proxy isn't upgrade
aware then we can definitely move that code over.

On Sep 6, 2016, at 12:05 PM, Andy Goldstein notifications@github.com
wrote:

Last time I checked, I think the code that backs kubectl proxy isn't
upgrade aware.

On Tue, Sep 6, 2016 at 11:50 AM, Clayton Coleman notifications@github.com
wrote:

Probably. What's the failure cause?

On Tue, Sep 6, 2016 at 11:20 AM, Michal Rostecki <notifications@github.com

wrote:

It's an expected behavior - we don't allow to attach or exec via kubectl
proxy.
You may use kubectl proxy --disable-filter=true.

However, that brings an another bug:

$ kubectl --server=http://localhost:8001 exec -ti example bash
Error from server: Upgrade request required

@kubernetes/sig-api-machinery
https://github.com/orgs/kubernetes/teams/sig-api-machinery do we care
about this?

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/kubernetes/kubernetes/issues/
32026#issuecomment-244986068>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABG_
p2JAimUgwQPQi5Xt4wqoR3eEWMshks5qnYTbgaJpZM4J0Rh_>
.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
<
#32026 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAABYkXSy3eJzGt9xMPaXHvofheVRu3Pks5qnYu8gaJpZM4J0Rh_

.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#32026 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p96PeSjJCMacQapq6PiJpGggUrvRks5qnY8FgaJpZM4J0Rh_
.

nhlfr · 2016-09-06T21:41:53Z

I didn't have time to debug that properly today, I'll do it tomorrow. But yes, it seems that proxy isn't upgrade aware in exec.

nhlfr · 2016-09-07T06:44:19Z

I think that answer is here:
https://github.com/golang/go/blob/master/src/net/http/httputil/reverseproxy.go#L110-L122
https://github.com/golang/go/blob/master/src/net/http/httputil/reverseproxy.go#L199-L201

ReverseProxy is removing the "Upgrade" header.

So, the question is still whether we care about this. If yes, it seems that we have to write our proxy from scratch... because patching Go stdlibs isn't even possible, is it?

ncdc · 2016-09-07T12:37:58Z

We do have our own upgrade aware proxy in-tree. I can point you to it when
I'm at my computer.

On Wednesday, September 7, 2016, Michal Rostecki notifications@github.com
wrote:

I think that answer is here:
https://github.com/golang/go/blob/master/src/net/http/
httputil/reverseproxy.go#L110-L122
https://github.com/golang/go/blob/master/src/net/http/
httputil/reverseproxy.go#L199-L201

ReverseProxy is removing the "Upgrade" header.

So, the question is still whether we care about this. If yes, it seems
that we have to write our proxy from scratch... because patching Go stdlibs
isn't even possible, is it?

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#32026 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAABYu5xHLpLcJJbOb6SQ8DN8dPSukfeks5qnl12gaJpZM4J0Rh_
.

smarterclayton · 2016-09-07T23:33:38Z

We should already have one - @liggitt, where is that?

smarterclayton · 2016-09-07T23:34:03Z

Nm saw Andy's response too late

ncdc · 2016-09-08T13:15:45Z

We actually seem to have 2:

kubernetes/pkg/apiserver/proxy.go

Line 209 in b974c09

    
           func (r *ProxyHandler) tryUpgrade(w http.ResponseWriter, req, newReq *http.Request, location *url.URL, transport http.RoundTripper, gv unversioned.GroupVersion) bool {

and

kubernetes/pkg/registry/generic/rest/proxy.go

Line 128 in b974c09

// tryUpgrade returns true if the request was handled.

nhlfr · 2016-09-08T14:00:47Z

@ncdc thanks for the links. Will look at them.

nhlfr · 2016-09-08T15:15:53Z

It seems that those two proxies may be potentially merged, but for sure one of them has to be rewritten a little bit - at least I don't see any way how to reuse them for kubectl proxy in the current shape. Looking again..

samsieber · 2016-11-04T21:32:55Z

@nhlfr @pwittrock Is there a potential timeline for this? I'd like to use Kubernetes where I work, but this is a show stopper for us. If it's not soon, could we update the documentation on services to specify that upgrades to websockets aren't supported? I was able to get most things working on kubernetes until I hit this road block. It was a nasty surprise.

Thanks for getting to the bottom of this though - it seems like its been a long unsolved issue:

I was a little disheartened until I finally came across this issue where the cause seems to be known. So thank you for figuring this out, even if the solution doesn't come soon.

nhlfr · 2016-11-05T11:34:36Z

Well, the unification and needed re-implementation of proxies may be too big change to get it done in 1.5 release. I'm going to work on this soon (in the next week). Let's assume that the worst case will be 1.6 release.

ianlewis · 2016-12-01T08:18:07Z

I'm having issues with this developing an exec into pod feature for the UI as well since our local development environment depends on kubectl proxy. I'd like to see this implemented soonish as it is pretty annoying to work around.

Before this change, kubernetes had two proxy server implementations and also used a proxy server from Go stdlib (which is not upgrade-aware). This commit aims to unify them. Fixes kubernetes#32026

travikk · 2016-12-18T21:48:08Z

@nhlfr any update on potential release for a fix for this?

kaleocheng · 2017-05-23T09:14:29Z

@nhlfr any progress updates on this?

smarterclayton · 2017-07-07T20:28:58Z

Unifying the proxy in #48631

smarterclayton · 2017-07-25T03:57:17Z

Fix in #49534,

Automatic merge from submit-queue Support exec/attach/portforward in `kubectl proxy` Use the UpgradeAwareProxy shared code in kubectl proxy. Provide a separate transport for those requests that does not have HTTP/2 enabled. Refactor the code to be a bit cleaner in places and to better separate changes. Fixes #32026 ```release-note `kubectl proxy` will now correctly handle the `exec`, `attach`, and `portforward` commands. You must pass `--disable-filter` to the command in order to allow these endpoints. ```

dolftax · 2017-12-11T08:19:37Z

In which k8s release is this supposed to go? To use 'kubectl exec' via kubectl proxy returns exit status 1.

liggitt · 2017-12-11T14:11:06Z

This is kubectl proxy, not kube-proxy

dolftax · 2017-12-12T05:36:01Z

Updated my comment. Also found the same in 1.8 changelog,

"kubectl proxy now correctly handles the exec, attach, and portforward commands. You must pass --disable-filter to the command to allow these commands."

Thanks.

bronger changed the title ~~"kubectl exec" does not work though "kubectl proxy"~~ "kubectl exec" does not work through "kubectl proxy" Sep 3, 2016

k8s-github-robot added area/kubectl team/ux labels Sep 3, 2016

janetkuo added the 0 - Backlog label Sep 7, 2016

ncdc mentioned this issue Sep 21, 2016

kubectl proxy doesn't support websockets #33151

Closed

pwittrock added the priority/backlog Higher priority than priority/awaiting-more-evidence. label Sep 28, 2016

ianlewis mentioned this issue Dec 1, 2016

Exec into Pod kubernetes/dashboard#1345

Closed

4 tasks

bgrant0607 removed the 0 - Backlog label Dec 8, 2016

lenartj mentioned this issue May 15, 2017

Pod attach/exec proto kubernetes/dashboard#1939

Merged

18 tasks

0xmichalis added sig/cli Categorizes an issue or PR as relevant to SIG CLI. and removed team/ux (deprecated - do not use) labels May 23, 2017

smarterclayton self-assigned this Jul 7, 2017

smarterclayton added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Jul 7, 2017

smarterclayton added this to the v1.8 milestone Jul 7, 2017

smarterclayton mentioned this issue Jul 7, 2017

Unify generic proxy code in apimachinery #48631

Merged

smarterclayton mentioned this issue Jul 7, 2017

kubectl should use websockets for exec, port forward, logs, and attach #48633

Closed

smarterclayton mentioned this issue Jul 25, 2017

Support exec/attach/portforward in kubectl proxy #49534

Merged

k8s-github-robot closed this as completed in #49534 Aug 9, 2017

rmohr mentioned this issue Jan 8, 2018

virtctl does not work with kubectl proxy kubevirt/kubevirt#632

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"kubectl exec" does not work through "kubectl proxy" #32026

"kubectl exec" does not work through "kubectl proxy" #32026

bronger commented Sep 3, 2016 •

edited

Loading

nhlfr commented Sep 6, 2016 •

edited

Loading

smarterclayton commented Sep 6, 2016

ncdc commented Sep 6, 2016

smarterclayton commented Sep 6, 2016

nhlfr commented Sep 6, 2016

nhlfr commented Sep 7, 2016

ncdc commented Sep 7, 2016

smarterclayton commented Sep 7, 2016 via email

smarterclayton commented Sep 7, 2016 via email

ncdc commented Sep 8, 2016

nhlfr commented Sep 8, 2016

nhlfr commented Sep 8, 2016

samsieber commented Nov 4, 2016

nhlfr commented Nov 5, 2016

ianlewis commented Dec 1, 2016

travikk commented Dec 18, 2016

kaleocheng commented May 23, 2017

smarterclayton commented Jul 7, 2017

smarterclayton commented Jul 25, 2017

dolftax commented Dec 11, 2017 •

edited

Loading

liggitt commented Dec 11, 2017

dolftax commented Dec 12, 2017 •

edited

Loading

"kubectl exec" does not work through "kubectl proxy" #32026

"kubectl exec" does not work through "kubectl proxy" #32026

Comments

bronger commented Sep 3, 2016 • edited Loading

nhlfr commented Sep 6, 2016 • edited Loading

smarterclayton commented Sep 6, 2016

ncdc commented Sep 6, 2016

smarterclayton commented Sep 6, 2016

nhlfr commented Sep 6, 2016

nhlfr commented Sep 7, 2016

ncdc commented Sep 7, 2016

smarterclayton commented Sep 7, 2016 via email

smarterclayton commented Sep 7, 2016 via email

ncdc commented Sep 8, 2016

nhlfr commented Sep 8, 2016

nhlfr commented Sep 8, 2016

samsieber commented Nov 4, 2016

nhlfr commented Nov 5, 2016

ianlewis commented Dec 1, 2016

travikk commented Dec 18, 2016

kaleocheng commented May 23, 2017

smarterclayton commented Jul 7, 2017

smarterclayton commented Jul 25, 2017

dolftax commented Dec 11, 2017 • edited Loading

liggitt commented Dec 11, 2017

dolftax commented Dec 12, 2017 • edited Loading

bronger commented Sep 3, 2016 •

edited

Loading

nhlfr commented Sep 6, 2016 •

edited

Loading

dolftax commented Dec 11, 2017 •

edited

Loading

dolftax commented Dec 12, 2017 •

edited

Loading