Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubernetes-e2e-gke-staging: broken test run #30962

Closed
k8s-github-robot opened this issue Aug 19, 2016 · 21 comments
Closed

kubernetes-e2e-gke-staging: broken test run #30962

k8s-github-robot opened this issue Aug 19, 2016 · 21 comments
Assignees
@rmmh
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.

Comments

@k8s-github-robot
Copy link

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6333/

Run so broken it didn't make JUnit output!
@k8s-github-robot k8s-github-robot added priority/backlog Higher priority than priority/awaiting-more-evidence. area/test-infra kind/flake Categorizes issue or PR as related to a flaky test. labels Aug 19, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6334/

Multiple broken tests:

Failed: [k8s.io] MetricsGrabber should grab all metrics from a Kubelet. {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200d40b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #27295

Failed: [k8s.io] Services should check NodePort out-of-range {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200d40b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Failed: [k8s.io] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:203
Expected error:
    <*errors.errorString | 0xc8200d40b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #28283

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl patch should add annotations for pods in rc [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200d40b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #26126 #30653

Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support inline execution and attach {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200d40b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #26324 #27715 #28845

Failed: [k8s.io] Pods should not start app containers if init containers fail on a RestartAlways pod {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200d40b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #29954

Failed: [k8s.io] Services should be able to up and down services {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200d40b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #26128 #26685

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6337/

Run so broken it didn't make JUnit output!

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Aug 20, 2016
@fejta fejta unassigned spxtr Aug 22, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6352/

Multiple broken tests:

Failed: [k8s.io] Pods should cap back-off at MaxContainerBackOff [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:1428
getting pod back-off-cap
Expected error:
    <*errors.StatusError | 0xc820f22100>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "the server has asked for the client to provide credentials (get pods back-off-cap)",
            Reason: "Unauthorized",
            Details: {
                Name: "back-off-cap",
                Group: "",
                Kind: "pods",
                Causes: [
                    {
                        Type: "UnexpectedServerResponse",
                        Message: "Unauthorized",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 401,
        },
    }
    the server has asked for the client to provide credentials (get pods back-off-cap)
not to have occurred

Issues about this test specifically: #27703

Failed: [k8s.io] ResourceQuota should create a ResourceQuota and capture the life of a secret. {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200c5060>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Failed: [k8s.io] Services should release NodePorts on delete {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200c5060>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Failed: [k8s.io] Pods should be restarted with a /healthz http liveness probe [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200c5060>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Failed: [k8s.io] EmptyDir volumes should support (non-root,0666,tmpfs) [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200c5060>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Failed: [k8s.io] SchedulerPredicates [Serial] validates that required NodeAffinity setting is respected if matching {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:702
Expected error:
    <*errors.StatusError | 0xc8215fd600>: {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Operation cannot be fulfilled on nodes \"gke-jenkins-e2e-default-pool-c517f6f5-a9ec\": the object has been modified; please apply your changes to the latest version and try again",
            Reason: "Conflict",
            Details: {
                Name: "gke-jenkins-e2e-default-pool-c517f6f5-a9ec",
                Group: "",
                Kind: "nodes",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 409,
        },
    }
    Operation cannot be fulfilled on nodes "gke-jenkins-e2e-default-pool-c517f6f5-a9ec": the object has been modified; please apply your changes to the latest version and try again
not to have occurred

Issues about this test specifically: #28071

@k8s-github-robot k8s-github-robot added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Aug 22, 2016
@j3ffml
Copy link
Contributor

j3ffml commented Aug 25, 2016

@cjcullen can you look into this? Looks like auth problems.

@cjcullen
Copy link
Member

We don't have any logs left from the staging jobs. I'll watch for further flakes.

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6363/

Run so broken it didn't make JUnit output!

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6366/

Run so broken it didn't make JUnit output!

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6369/

Run so broken it didn't make JUnit output!

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6370/

Run so broken it didn't make JUnit output!

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6372/

Run so broken it didn't make JUnit output!

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @cjcullen @jlowdermilk

This flaky-test issue would love to have more attention...

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6376/

Run so broken it didn't make JUnit output!

@j3ffml j3ffml removed their assignment Aug 29, 2016
@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6379/

Run so broken it didn't make JUnit output!

@cjcullen
Copy link
Member

Recent flakes appear to be jenkins weirdness: deleting clusters while they are being tested. I have @rmmh helping me look into it.

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6384/

Run so broken it didn't make JUnit output!

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6380/

Run so broken it didn't make JUnit output!

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6383/

Run so broken it didn't make JUnit output!

@mikedanese mikedanese assigned rmmh and unassigned cjcullen Aug 31, 2016
@mikedanese mikedanese mentioned this issue Aug 31, 2016
Closed
rmmh pushed a commit to rmmh/kubernetes-test-infra that referenced this issue Aug 31, 2016
Make dockerized-e2e-runner.sh kill the container on timeout.
cc5a6a3 
This was tested manually with the small script:

    #!/bin/bash
    # test with "timeout -k4 2 ./leak-test.sh"
    # observe that the trap properly cleans up the container.

    CONTAINER_NAME="leak-$$"

    echo "container: $CONTAINER_NAME"
    trap "docker kill ${CONTAINER_NAME}" EXIT

    docker run --rm --name="${CONTAINER_NAME}" ubuntu sleep 1

    trap '' EXIT

This should fix flakes associated with leaked containers:
kubernetes/kubernetes#30962 and kubernetes/kubernetes#31213
rmmh pushed a commit to rmmh/kubernetes-test-infra that referenced this issue Aug 31, 2016
Make dockerized-e2e-runner.sh kill the container on timeout.
c7d697b 
This was tested manually with the small script:

    #!/bin/bash
    # test with "timeout -k4 2 ./leak-test.sh"
    # observe that the trap properly cleans up the container.

    CONTAINER_NAME="leak-$$"

    echo "container: $CONTAINER_NAME"
    trap "docker kill ${CONTAINER_NAME}" EXIT

    docker run --rm --name="${CONTAINER_NAME}" ubuntu sleep 600

    trap '' EXIT

This should fix flakes associated with leaked containers:
kubernetes/kubernetes#30962 and kubernetes/kubernetes#31213
@rmmh rmmh mentioned this issue Aug 31, 2016
Merged
rmmh pushed a commit to rmmh/kubernetes-test-infra that referenced this issue Aug 31, 2016
Make dockerized-e2e-runner.sh kill the container on timeout.
ba54b5f 
This was tested manually with the small script:

    #!/bin/bash
    # test with "timeout -k4 2 ./leak-test.sh"
    # observe that the trap properly cleans up the container.

    CONTAINER_NAME="leak-$$"

    echo "container: $CONTAINER_NAME"
    trap "docker stop ${CONTAINER_NAME}" EXIT

    docker run --rm --name="${CONTAINER_NAME}" ubuntu sleep 600

    trap '' EXIT

This should fix flakes associated with leaked containers:
kubernetes/kubernetes#30962 and kubernetes/kubernetes#31213
rmmh pushed a commit to rmmh/kubernetes-test-infra that referenced this issue Aug 31, 2016
Make dockerized-e2e-runner.sh kill the container on timeout.
c4ed1f3 
This was tested manually with the small script:

    #!/bin/bash
    # test with "timeout -k15 2 ./leak-test.sh"
    # observe that the trap properly cleans up the container.

    CONTAINER_NAME="leak-$$"

    echo "container: $CONTAINER_NAME"
    trap "docker stop ${CONTAINER_NAME}" EXIT

    docker run --rm --name="${CONTAINER_NAME}" ubuntu sleep 600

    trap '' EXIT

This should fix flakes associated with leaked containers:
kubernetes/kubernetes#30962 and kubernetes/kubernetes#31213
rmmh added a commit to kubernetes/test-infra that referenced this issue Aug 31, 2016
@rmmh
Make dockerized-e2e-runner.sh kill the container on timeout. (#483)
39182a4 
This was tested manually with the small script:

    #!/bin/bash
    # test with "timeout -k15 2 ./leak-test.sh"
    # observe that the trap properly cleans up the container.

    CONTAINER_NAME="leak-$$"

    echo "container: $CONTAINER_NAME"
    trap "docker stop ${CONTAINER_NAME}" EXIT

    docker run --rm --name="${CONTAINER_NAME}" ubuntu sleep 600

    trap '' EXIT

This should fix flakes associated with leaked containers:
kubernetes/kubernetes#30962 and kubernetes/kubernetes#31213
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6387/

Multiple broken tests:

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl patch should add annotations for pods in rc [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200af060>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #26126 #30653

Failed: [k8s.io] Pods should not start app containers if init containers fail on a RestartAlways pod {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200af060>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #29954

Failed: [k8s.io] DNS should provide DNS for services [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200af060>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Issues about this test specifically: #26168 #27450

Failed: [k8s.io] SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:299
Error waiting for 319 pods to be running - probably a timeout
Expected error:
    <*errors.errorString | 0xc822fc9450>: {
        s: "Timeout while waiting for pods with labels \"startPodsID=338c5783-703a-11e6-a16d-0242ac11000a\" to be running",
    }
    Timeout while waiting for pods with labels "startPodsID=338c5783-703a-11e6-a16d-0242ac11000a" to be running
not to have occurred

Issues about this test specifically: #27662 #29820

Failed: [k8s.io] Docker Containers should use the image defaults if command and args are blank [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132
Expected error:
    <*errors.errorString | 0xc8200af060>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @rmmh

This flaky-test issue would love to have more attention.

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6386/

Run so broken it didn't make JUnit output!

@rmmh
Copy link
Contributor

rmmh commented Sep 2, 2016

Closing as a dupe of #31213

@rmmh rmmh closed this as completed Sep 2, 2016
@k8s-github-robot k8s-github-robot mentioned this issue Oct 11, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rmmh rmmh

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@rmmh @fejta @j3ffml @cjcullen @spxtr @pwittrock @k8s-github-robot