[e2e test failure] Simple pod should support exec through kubectl proxy #50466

ericchiang · 2017-08-10T18:47:07Z

/cc @kubernetes/sig-cli-bugs

This test has started failing on GKE and GCI-GKI e2e tests:

https://k8s-testgrid.appspot.com/release-master-blocking#gci-gke
https://k8s-testgrid.appspot.com/release-master-blocking#gke

Example failure

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/12920

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:457
Expected error:
    <exec.CodeExitError>: {
        Err: {
            s: "error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://35.188.213.188 --kubeconfig=/tmp/gke-kubecfg386178168 --server=http://127.0.0.1:45529 --namespace=e2e-tests-kubectl-nlzj2 exec nginx echo running in container] []  <nil>  W0810 18:25:44.244920    4112 http.go:342] Error reading backend response: unexpected EOF\nerror: error sending request: Post http://127.0.0.1:45529/api/v1/namespaces/e2e-tests-kubectl-nlzj2/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: unexpected EOF\n [] <nil> 0xc4211d3830 exit status 1 <nil> <nil> true [0xc4203e6828 0xc4203e6840 0xc4203e6858] [0xc4203e6828 0xc4203e6840 0xc4203e6858] [0xc4203e6838 0xc4203e6850] [0x11bf750 0x11bf750] 0xc420f2db60 <nil>}:\nCommand stdout:\n\nstderr:\nW0810 18:25:44.244920    4112 http.go:342] Error reading backend response: unexpected EOF\nerror: error sending request: Post http://127.0.0.1:45529/api/v1/namespaces/e2e-tests-kubectl-nlzj2/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: unexpected EOF\n\nerror:\nexit status 1\n",
        },
        Code: 1,
    }
    error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://35.188.213.188 --kubeconfig=/tmp/gke-kubecfg386178168 --server=http://127.0.0.1:45529 --namespace=e2e-tests-kubectl-nlzj2 exec nginx echo running in container] []  <nil>  W0810 18:25:44.244920    4112 http.go:342] Error reading backend response: unexpected EOF
    error: error sending request: Post http://127.0.0.1:45529/api/v1/namespaces/e2e-tests-kubectl-nlzj2/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: unexpected EOF
     [] <nil> 0xc4211d3830 exit status 1 <nil> <nil> true [0xc4203e6828 0xc4203e6840 0xc4203e6858] [0xc4203e6828 0xc4203e6840 0xc4203e6858] [0xc4203e6838 0xc4203e6850] [0x11bf750 0x11bf750] 0xc420f2db60 <nil>}:
    Command stdout:
    
    stderr:
    W0810 18:25:44.244920    4112 http.go:342] Error reading backend response: unexpected EOF
    error: error sending request: Post http://127.0.0.1:45529/api/v1/namespaces/e2e-tests-kubectl-nlzj2/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: unexpected EOF
    
    error:
    exit status 1
    
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2066```

The text was updated successfully, but these errors were encountered:

mengqiy · 2017-08-11T18:37:50Z

Not sure if it is related to recent change in #49534.

pwittrock · 2017-08-15T00:26:47Z

@bowei Looks like this is only failing on GKE. @liggitt suggested this might be due to the way the master / node network is setup on GKE.

pwittrock · 2017-08-15T00:27:04Z

cc
@kubernetes/sig-network-bugs
@kubernetes/sig-cli-bugs

pwittrock · 2017-08-15T00:27:46Z

/priority critical-urgent

jagosan · 2017-08-15T17:19:26Z

/assign @bowei

jagosan · 2017-08-15T17:20:17Z

@bowei can you please take a look at @pwittrock's comment and update / reassign if appropriate? Thanks!

bowei · 2017-08-15T17:26:34Z

I'll take a look

bowei · 2017-08-15T17:31:55Z

From a cursory glance: if networking is at fault, it would result in all of the tests that traverse the SSH tunnel path to fail with a timeout, which I'm not seeing here.

bowei · 2017-08-15T20:40:45Z

This test does not seem to have ever passed, at least not in the testgrid history.

pwittrock · 2017-08-15T20:52:41Z

@liggitt

Thanks for your suggestion of looking into the networking on GKE. My understanding is that you didn't think the failure we are seeing on was related to kubectl itself.

The kubectl test appears to be the only one breaking. Should I expect to see other test failures if the failure is not related to the request sent by kubectl, but instead how it is processed by the apiserver?

liggitt · 2017-08-15T21:07:07Z

Should I expect to see other test failures if the failure is not related to the request sent by kubectl, but instead how it is processed by the apiserver?

I'm not sure we test any other websocket requests in e2e, so maybe not?

jagosan · 2017-08-15T21:51:42Z

/unassign @bowei
/assign @mengqiy
/remove-sig network

Thanks for looking @bowei. @mengqiy - any more insight to add here?

pwittrock · 2017-08-15T22:09:49Z

@liggitt

Should we be adding those tests, or is it not something that makes sense to do?

mengqiy · 2017-08-16T20:11:26Z

/assign @apelisse
/unassign
@apelisse is taking over this issue and have potential fix.

smarterclayton · 2017-08-17T01:37:36Z

The core bug here is that gke uses a different transport wrapper than normal gce installs and unwrapping the transport is required to upgrade passthrough proxies.

apelisse · 2017-08-17T02:40:34Z

And I think that's what I've addressed in #50775 (making the GKE transport unwrappable), but as we can see, it fails for another reason. Any idea?

liggitt · 2017-08-17T04:23:47Z

#50775 is necessary but not sufficient.

copied from #50775 (comment)

the issue is with

kubernetes/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go

Lines 269 to 282 in 4bfe9b1

    
           // dial dials the backend at req.URL and writes req to it. 
        
           func dial(req *http.Request, transport http.RoundTripper) (net.Conn, error) { 
        
           	conn, err := DialURL(req.URL, transport) 
        
           	if err != nil { 
        
           		return nil, fmt.Errorf("error dialing backend: %v", err) 
        
           	} 
        
           	if err = req.Write(conn); err != nil { 
        
           		conn.Close() 
        
           		return nil, fmt.Errorf("error sending request: %v", err) 
        
           	} 
        
           	return conn, err 
        
           }

that does not actually use the transport to handle the request, only to dial. That means that TLS-authentication methods like client certs work, and header authentication methods like bearer tokens do not.

can recreate this by running kubectl proxy using bearer token authentication and not cert auth, then try kubectl exec ... --server=http://localhost:8001

smarterclayton · 2017-08-17T05:32:50Z

The transport that is used for upgrade needs to be invoked on the request we use to dial the backend. In theory we can use a capturing RoundTripper to see the mutation (we use that in a few other places, and I needed it for the websocket code).

…

On Thu, Aug 17, 2017 at 12:23 AM, Jordan Liggitt ***@***.***> wrote: #50775 <#50775> is necessary but not sufficient. copied from #50775 (comment) <#50775 (comment)> the issue is with https://github.com/kubernetes/kubernetes/blob/ 4bfe9b1/staging/src/k8s.io/ apimachinery/pkg/util/proxy/upgradeaware.go#L269-L282 that does not actually use the transport to handle the request, only to dial. That means that TLS-authentication methods like client certs work, and header authentication methods like bearer tokens do not. can recreate this by running kubectl proxy using bearer token authentication and not cert auth, then try kubectl exec ... --server= http://localhost:8001 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#50466 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_py-FaRZl2Cs0alIEykUqQ5CIm3nPks5sY8BdgaJpZM4Oz1AO> .

As reported in kubernetes#50466, this test doesn't work in GKE because the transport layer doesn't work with dialing. As the feature that is broken in GKE is new and didn't work before, it is safe to juste ignore the test and consider the feature as "still not working" in GKE.

pwittrock · 2017-08-21T20:40:36Z

/assign @caesarxuchao Are you a good person to assign this to, or is there someone better?

As reported in kubernetes#50466, this test doesn't work in GKE because the transport layer doesn't work with dialing. As the feature that is broken in GKE is new and didn't work before, it is safe to juste ignore the test and consider the feature as "still not working" in GKE.

jagosan · 2017-08-22T23:04:45Z

/assign @cheftako

Automatic merge from submit-queue Skip "Simple pod should support exec through kubectl proxy" test As reported in #50466, this test doesn't work in GKE because it uses a bearer token and the feature only works with client certs. As the feature that is broken in GKE is new and didn't work before, it is safe to juste ignore the test and consider the feature as "still not working" in GKE. **What this PR does / why we need it**: Fixes the broken test in https://k8s-testgrid.appspot.com/release-master-blocking#gke **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: works-around #50466 **Special notes for your reviewer**: **Release note**: ```release-note NONE ```

@liggitt

…_wrap Automatic merge from submit-queue (batch tested with PRs 50775, 51397, 51168, 51465, 51536) Allow bearer requests to be proxied by kubectl proxy Use a fake transport to capture changes to the request and then surface them back to the end user. Fixes kubernetes#50466 @liggitt no tests yet, but works locally

k8s-ci-robot added sig/cli Categorizes an issue or PR as relevant to SIG CLI. kind/bug Categorizes issue or PR as related to a bug. labels Aug 10, 2017

ericchiang added this to the v1.8 milestone Aug 10, 2017

k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. kind/bug Categorizes issue or PR as related to a bug. labels Aug 15, 2017

k8s-ci-robot added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Aug 15, 2017

k8s-ci-robot assigned bowei Aug 15, 2017

pwittrock mentioned this issue Aug 15, 2017

Support exec/attach/portforward in kubectl proxy #49534

Merged

k8s-ci-robot assigned mengqiy and unassigned bowei Aug 15, 2017

k8s-ci-robot removed the sig/network Categorizes an issue or PR as relevant to SIG Network. label Aug 15, 2017

k8s-ci-robot assigned apelisse and unassigned mengqiy Aug 16, 2017

ericchiang added the kind/e2e-test-failure label Aug 17, 2017

apelisse mentioned this issue Aug 21, 2017

Skip "Simple pod should support exec through kubectl proxy" test #51047

Merged

apelisse added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Aug 21, 2017

k8s-ci-robot assigned cheftako Aug 22, 2017

smarterclayton mentioned this issue Aug 23, 2017

Allow bearer requests to be proxied by kubectl proxy #51168

Merged

k8s-github-robot closed this as completed in #51168 Aug 30, 2017

caesarxuchao mentioned this issue Mar 5, 2020

[DO NOT MERGE][TEST ONLY] use 0.0.9, httpConnect #88812

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[e2e test failure] Simple pod should support exec through kubectl proxy #50466

[e2e test failure] Simple pod should support exec through kubectl proxy #50466

ericchiang commented Aug 10, 2017

mengqiy commented Aug 11, 2017

pwittrock commented Aug 15, 2017

pwittrock commented Aug 15, 2017

pwittrock commented Aug 15, 2017

jagosan commented Aug 15, 2017

jagosan commented Aug 15, 2017

bowei commented Aug 15, 2017

bowei commented Aug 15, 2017 •

edited

Loading

bowei commented Aug 15, 2017

pwittrock commented Aug 15, 2017

liggitt commented Aug 15, 2017

jagosan commented Aug 15, 2017

pwittrock commented Aug 15, 2017 •

edited

Loading

mengqiy commented Aug 16, 2017

smarterclayton commented Aug 17, 2017

apelisse commented Aug 17, 2017 •

edited

Loading

liggitt commented Aug 17, 2017

smarterclayton commented Aug 17, 2017 via email

pwittrock commented Aug 21, 2017

jagosan commented Aug 22, 2017

[e2e test failure] Simple pod should support exec through kubectl proxy #50466

[e2e test failure] Simple pod should support exec through kubectl proxy #50466

Comments

ericchiang commented Aug 10, 2017

mengqiy commented Aug 11, 2017

pwittrock commented Aug 15, 2017

pwittrock commented Aug 15, 2017

pwittrock commented Aug 15, 2017

jagosan commented Aug 15, 2017

jagosan commented Aug 15, 2017

bowei commented Aug 15, 2017

bowei commented Aug 15, 2017 • edited Loading

bowei commented Aug 15, 2017

pwittrock commented Aug 15, 2017

liggitt commented Aug 15, 2017

jagosan commented Aug 15, 2017

pwittrock commented Aug 15, 2017 • edited Loading

mengqiy commented Aug 16, 2017

smarterclayton commented Aug 17, 2017

apelisse commented Aug 17, 2017 • edited Loading

liggitt commented Aug 17, 2017

smarterclayton commented Aug 17, 2017 via email

pwittrock commented Aug 21, 2017

jagosan commented Aug 22, 2017

bowei commented Aug 15, 2017 •

edited

Loading

pwittrock commented Aug 15, 2017 •

edited

Loading

apelisse commented Aug 17, 2017 •

edited

Loading