Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[k8s.io] Kubectl client [k8s.io] Simple pod should support exec through an HTTP proxy {Kubernetes e2e suite} #30489

Closed
k8s-github-robot opened this issue Aug 12, 2016 · 15 comments
Assignees
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gke/8167/

Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support exec through an HTTP proxy {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:277
Expected
    <bool>: false
to be true
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:276

Previous issues for this test: #27156 #28979

@k8s-github-robot k8s-github-robot added priority/backlog Higher priority than priority/awaiting-more-evidence. kind/flake Categorizes issue or PR as related to a flaky test. labels Aug 12, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30814/kubernetes-pull-build-test-e2e-gce/54259/

Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support exec through an HTTP proxy {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:349
Aug 17 19:18:06.500: Unexpected kubectl exec output. Wanted "running in container\n", got  ""
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:338

Happened on a presubmit run in #30814.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/7469/

Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support exec through an HTTP proxy {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:348
Expected error:
    <*errors.errorString | 0xc820bb08b0>: {
        s: "Error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.154.233.4 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-oicod exec nginx echo running in container] [ZONE=us-central1-f BUILD_URL=http://goto.google.com/k8s-test/job/kubernetes-e2e-gke-staging-parallel/7469/ JENKINS_AWS_SSH_PUBLIC_KEY_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/95c6ce69-fa1a-4cbf-b741-6c01855ddcf9/kube_aws_rsa.pub.txt GOLANG_VERSION=1.6.3 HOSTNAME=432c700ab4e0 CLOUDSDK_COMPONENT_MANAGER_DISABLE_UPDATE_CHECK=true ROOT_BUILD_CAUSE_TIMERTRIGGER=true TERM=xterm HUDSON_SERVER_COOKIE=02143c9ae5889f5c SHELL=/bin/bash PROJECT=k8s-e2e-gke-staging-parallel KUBE_GCE_INSTANCE_PREFIX=jenkins-e2e JENKINS_AWS_SSH_PRIVATE_KEY_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/f6aecaa7-deca-45cb-b2aa-93297f75630f/kube_aws_rsa.txt SSH_CLIENT=10.240.0.37 50821 22 KUBE_CONFIG_FILE=config-test.sh GOOGLE_APPLICATION_CREDENTIALS=/service-account.json BUILD_TAG=jenkins-kubernetes-e2e-gke-staging-parallel-7469 E2E_UP=true CLUSTER_NAME=jenkins-e2e ROOT_BUILD_CAUSE=TIMERTRIGGER CLOUDSDK_COMPONENT_MANAGER_SNAPSHOT_URL=file:///workspace/repo/components-2.json WORKSPACE=/workspace JOB_URL=http://goto.google.com/k8s-test/job/kubernetes-e2e-gke-staging-parallel/ JENKINS_AWS_CREDENTIALS_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/9e846514-4509-4bdc-bb7c-3c3ce9b0a269/aws_credentials.txt USER=jenkins KUBE_GCE_NETWORK=jenkins-e2e CLOUDSDK_CONFIG=/var/lib/jenkins/.config/gcloud E2E_REPORT_DIR=/workspace/_artifacts INSTANCE_PREFIX=jenkins-e2e JENKINS_GCE_SSH_PRIVATE_KEY_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/c926eb6c-c4a4-4db9-b78b-ea978a7efa4c/google_compute_engine.txt REPO_DIR=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel GINKGO_TEST_ARGS=--ginkgo.skip=\\[Slow\\]|\\[Serial\\]|\\[Disruptive\\]|\\[Flaky\\]|\\[Feature:.+\\] NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat JENKINS_HOME=/var/lib/jenkins CLOUDSDK_CORE_DISABLE_PROMPTS=1 JENKINS_USE_SERVER_VERSION=y MAIL=/var/mail/jenkins PATH=/workspace/kubernetes/platforms/linux/amd64:/workspace/cloudsdk/google-cloud-sdk/bin://google-cloud-sdk/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin E2E_OPT=--check_version_skew=false PWD=/workspace/kubernetes CLOUDSDK_BUCKET=gs://cloud-sdk-testing/rc HUDSON_URL=http://goto.google.com/k8s-test/ LANG=en_US.UTF-8 E2E_TEST=true JOB_NAME=kubernetes-e2e-gke-staging-parallel KUBECTL=./cluster/kubectl.sh HOST_ARTIFACTS_DIR=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel/_artifacts BUILD_DISPLAY_NAME=#7469 XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt KUBERNETES_PROVIDER=gke JENKINS_URL=http://goto.google.com/k8s-test/ BUILD_CAUSE=TIMERTRIGGER BUILD_ID=7469 GOLANG_DOWNLOAD_SHA256=cdde5e08530c0579255d6153b08fdb3b8e47caabbe717bc7bcd7561275a87aeb KUBE_GKE_NETWORK=jenkins-e2e CLOUDSDK_API_ENDPOINT_OVERRIDES_CONTAINER=https://staging-container.sandbox.googleapis.com/ JOB_BASE_NAME=kubernetes-e2e-gke-staging-parallel E2E_MIN_STARTUP_PODS=8 HOME=/workspace SHLVL=6 no_proxy=127.0.0.1,localhost JENKINS_SERVER_COOKIE=02143c9ae5889f5c EXECUTOR_NUMBER=0 KUBE_GCE_ZONE=us-central1-f JENKINS_GCE_SSH_PUBLIC_KEY_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/720d4d04-da89-4d85-9548-bb7284574659/google_compute_engine.pub.txt NODE_LABELS=agent-light-4 e2e node GINKGO_PARALLEL=y LOGNAME=jenkins HUDSON_HOME=/var/lib/jenkins SSH_CONNECTION=10.240.0.37 50821 10.240.0.57 22 NODE_NAME=agent-light-4 BUILD_CAUSE_TIMERTRIGGER=true E2E_PUBLISH_GREEN_VERSION=false GOPATH=/go KUBE_AWS_INSTANCE_PREFIX=jenkins-e2e BUILD_NUMBER=7469 HUDSON_COOKIE=460abfeb-ddd6-4c36-87e5-d44a239d8a20 FAIL_ON_GCP_RESOURCE_LEAK=true E2E_DOWN=true GOLANG_DOWNLOAD_URL=https://golang.org/dl/go1.6.3.linux-amd64.tar.gz _=/workspace/kubernetes/platforms/linux/amd64/ginkgo HTTPS_PROXY=http://127.0.0.1:45606]  <nil>  Error from server: ssh: unexpected packet in response to channel open: <nil>\n [] <nil> 0xc82097f260 exit status 1 <nil> true [0xc82029e588 0xc82029e5a0 0xc82029e5b8] [0xc82029e588 0xc82029e5a0 0xc82029e5b8] [0xc82029e598 0xc82029e5b0] [0xa96280 0xa96280] 0xc820a71500}:\nCommand stdout:\n\nstderr:\nError from server: ssh: unexpected packet in response to channel open: <nil>\n\nerror:\nexit status 1\n",
    }
    Error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.154.233.4 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-oicod exec nginx echo running in container] [ZONE=us-central1-f BUILD_URL=http://goto.google.com/k8s-test/job/kubernetes-e2e-gke-staging-parallel/7469/ JENKINS_AWS_SSH_PUBLIC_KEY_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/95c6ce69-fa1a-4cbf-b741-6c01855ddcf9/kube_aws_rsa.pub.txt GOLANG_VERSION=1.6.3 HOSTNAME=432c700ab4e0 CLOUDSDK_COMPONENT_MANAGER_DISABLE_UPDATE_CHECK=true ROOT_BUILD_CAUSE_TIMERTRIGGER=true TERM=xterm HUDSON_SERVER_COOKIE=02143c9ae5889f5c SHELL=/bin/bash PROJECT=k8s-e2e-gke-staging-parallel KUBE_GCE_INSTANCE_PREFIX=jenkins-e2e JENKINS_AWS_SSH_PRIVATE_KEY_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/f6aecaa7-deca-45cb-b2aa-93297f75630f/kube_aws_rsa.txt SSH_CLIENT=10.240.0.37 50821 22 KUBE_CONFIG_FILE=config-test.sh GOOGLE_APPLICATION_CREDENTIALS=/service-account.json BUILD_TAG=jenkins-kubernetes-e2e-gke-staging-parallel-7469 E2E_UP=true CLUSTER_NAME=jenkins-e2e ROOT_BUILD_CAUSE=TIMERTRIGGER CLOUDSDK_COMPONENT_MANAGER_SNAPSHOT_URL=file:///workspace/repo/components-2.json WORKSPACE=/workspace JOB_URL=http://goto.google.com/k8s-test/job/kubernetes-e2e-gke-staging-parallel/ JENKINS_AWS_CREDENTIALS_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/9e846514-4509-4bdc-bb7c-3c3ce9b0a269/aws_credentials.txt USER=jenkins KUBE_GCE_NETWORK=jenkins-e2e CLOUDSDK_CONFIG=/var/lib/jenkins/.config/gcloud E2E_REPORT_DIR=/workspace/_artifacts INSTANCE_PREFIX=jenkins-e2e JENKINS_GCE_SSH_PRIVATE_KEY_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/c926eb6c-c4a4-4db9-b78b-ea978a7efa4c/google_compute_engine.txt REPO_DIR=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel GINKGO_TEST_ARGS=--ginkgo.skip=\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\] NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat JENKINS_HOME=/var/lib/jenkins CLOUDSDK_CORE_DISABLE_PROMPTS=1 JENKINS_USE_SERVER_VERSION=y MAIL=/var/mail/jenkins PATH=/workspace/kubernetes/platforms/linux/amd64:/workspace/cloudsdk/google-cloud-sdk/bin://google-cloud-sdk/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin E2E_OPT=--check_version_skew=false PWD=/workspace/kubernetes CLOUDSDK_BUCKET=gs://cloud-sdk-testing/rc HUDSON_URL=http://goto.google.com/k8s-test/ LANG=en_US.UTF-8 E2E_TEST=true JOB_NAME=kubernetes-e2e-gke-staging-parallel KUBECTL=./cluster/kubectl.sh HOST_ARTIFACTS_DIR=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel/_artifacts BUILD_DISPLAY_NAME=#7469 XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt KUBERNETES_PROVIDER=gke JENKINS_URL=http://goto.google.com/k8s-test/ BUILD_CAUSE=TIMERTRIGGER BUILD_ID=7469 GOLANG_DOWNLOAD_SHA256=cdde5e08530c0579255d6153b08fdb3b8e47caabbe717bc7bcd7561275a87aeb KUBE_GKE_NETWORK=jenkins-e2e CLOUDSDK_API_ENDPOINT_OVERRIDES_CONTAINER=https://staging-container.sandbox.googleapis.com/ JOB_BASE_NAME=kubernetes-e2e-gke-staging-parallel E2E_MIN_STARTUP_PODS=8 HOME=/workspace SHLVL=6 no_proxy=127.0.0.1,localhost JENKINS_SERVER_COOKIE=02143c9ae5889f5c EXECUTOR_NUMBER=0 KUBE_GCE_ZONE=us-central1-f JENKINS_GCE_SSH_PUBLIC_KEY_FILE=/var/lib/jenkins/workspace/kubernetes-e2e-gke-staging-parallel@tmp/secretFiles/720d4d04-da89-4d85-9548-bb7284574659/google_compute_engine.pub.txt NODE_LABELS=agent-light-4 e2e node GINKGO_PARALLEL=y LOGNAME=jenkins HUDSON_HOME=/var/lib/jenkins SSH_CONNECTION=10.240.0.37 50821 10.240.0.57 22 NODE_NAME=agent-light-4 BUILD_CAUSE_TIMERTRIGGER=true E2E_PUBLISH_GREEN_VERSION=false GOPATH=/go KUBE_AWS_INSTANCE_PREFIX=jenkins-e2e BUILD_NUMBER=7469 HUDSON_COOKIE=460abfeb-ddd6-4c36-87e5-d44a239d8a20 FAIL_ON_GCP_RESOURCE_LEAK=true E2E_DOWN=true GOLANG_DOWNLOAD_URL=https://golang.org/dl/go1.6.3.linux-amd64.tar.gz _=/workspace/kubernetes/platforms/linux/amd64/ginkgo HTTPS_PROXY=http://127.0.0.1:45606]  <nil>  Error from server: ssh: unexpected packet in response to channel open: <nil>
     [] <nil> 0xc82097f260 exit status 1 <nil> true [0xc82029e588 0xc82029e5a0 0xc82029e5b8] [0xc82029e588 0xc82029e5a0 0xc82029e5b8] [0xc82029e598 0xc82029e5b0] [0xa96280 0xa96280] 0xc820a71500}:
    Command stdout:

    stderr:
    Error from server: ssh: unexpected packet in response to channel open: <nil>

    error:
    exit status 1

not to have occurred

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Aug 22, 2016
@ncdc
Copy link
Member

ncdc commented Aug 25, 2016

The initial comment (#30489 (comment)) wasn't related to exec at all - BeforeEach failed trying to run a pod. I'm unable to locate the appropriate data in the log files to triage.

The error from the second comment (Unexpected kubectl exec output. Wanted "running in container\n", got "") is a legit flake, but I'm having trouble figuring out why it failed. The exec request made its way from kubectl to apiserver to kubelet. Kubelet finished running it quickly. The apiserver returned a response to kubectl quickly (albeit the empty string) at 02:18:06.500. The apiserver shows Error proxying data from client to backend: read tcp 10.240.0.2:443->146.148.49.195:59778: read: connection reset by peer at 02:18:33.703557. It's unclear if this is related to this exec call, however. Finally, at 02:19:44.427668, the apiserver logs that the exec call finished, 1m38s after kubectl issued it.

For the last comment, I assume that Error from server: ssh: unexpected packet in response to channel open: <nil> is related to the ssh connection from apiserver to node?

@pwittrock I don't know how to resolve this flake. What do you recommend doing here?

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @ncdc

This flaky-test issue would love to have more attention...

@ncdc
Copy link
Member

ncdc commented Sep 1, 2016

@pwittrock bump if you have a moment

@pwittrock
Copy link
Member

@bprashanth Would you be able to help @ncdc on this, or find someone who can?

@pwittrock
Copy link
Member

@ncdc Apologies for my unresponsiveness. Thank you for following up on this.

@bprashanth
Copy link
Contributor

I'm not That familiar with exec, ignore that last one till I or someone has time to dig since it's GKE (@cjcullen), the previous failures should be standard kubectl exec (it still uses spdy right?). connection reset by peer is usually rst, or remote end has dropped the connection, suggest digging into node readiness/kubelet health if the exec actually got its connection dropped.

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @ncdc

This flaky-test issue would love to have more attention.

@ncdc
Copy link
Member

ncdc commented Sep 7, 2016

@bprashanth it turns out that Error proxying data from client to backend: read tcp 10.240.0.2:443->146.148.49.195:59778: read: connection reset by peer was related to a port-forwarding call, and not the exec one that failed.

If you look at the apiserver log, you see these entries one right after the other:

I0818 02:19:44.427522       5 handlers.go:164] POST /api/v1/namespaces/e2e-tests-kubectl-z139p/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: (1m20.970723096s) hijacked [[kubectl/v1.4.0 (linux/amd64) kubernetes/b9f294d] 146.148.49.195:59752]
I0818 02:19:44.427668       5 handlers.go:164] POST /api/v1/namespaces/e2e-tests-kubectl-k1h1c/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: (1m38.126429034s) hijacked [[kubectl/v1.4.0 (linux/amd64) kubernetes/b9f294d] 146.148.49.195:59710]
I0818 02:19:44.427912       5 handlers.go:164] POST /api/v1/namespaces/e2e-tests-kubectl-z139p/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: (1m20.72111347s) hijacked [[kubectl/v1.4.0 (linux/amd64) kubernetes/b9f294d] 146.148.49.195:59756]

There's something strange going on:

  1. 19:18:23.325, start exec for namespace e2e-tests-kubectl-z139p
    1. Build log reports results at 19:18:23.575
    2. Apiserver log reports hijacking ended at 19:19:44.427522
  2. 19:18:06.170, start exec for namespace e2e-tests-kubectl-k1h1c
  3. Build log reports results at 19:18:06.500
  4. Apiserver log reports hijacking ended at 19:19:44.427668
  5. 19:18:23.955, start exec for namespace e2e-tests-kubectl-z139p
    1. Build log reports results at 19:18:23.829
    2. Apiserver log reports hijacking ended at 19:19:44.427912

I'm not sure why there is a gap of over 1 minute between the time kubectl got a response from the apiserver and when the apiserver considered the request finished. I'm also still not sure why one of these exec requests didn't get any data. Maybe the latter is similar to whatever is going on in #27680.

@smarterclayton any ideas why the apiserver has a huge gap in logging the hijacking?

@smarterclayton
Copy link
Contributor

smarterclayton commented Sep 7, 2016 via email

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @ncdc

This flaky-test issue would love to have more attention.

2 similar comments
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @ncdc

This flaky-test issue would love to have more attention.

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @ncdc

This flaky-test issue would love to have more attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Development

No branches or pull requests

5 participants