Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite} #31066

Closed
k8s-github-robot opened this issue Aug 21, 2016 · 14 comments · Fixed by #32378
Closed

[k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite} #31066

k8s-github-robot opened this issue Aug 21, 2016 · 14 comments · Fixed by #32378
Assignees
@caesarxuchao @kevin-wangzefeng
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/21985/

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1243
Expected error:
    <*errors.errorString | 0xc820cd3450>: {
        s: "Error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.154.238.217 --kubeconfig=/workspace/.kube/config taint nodes jenkins-e2e-master kubernetes.io/e2e-taint-key-5c9b1cf3-6736-11e6-a811-0242ac110003=testing-taint-value:NoSchedule] []  <nil>  Error from server: Operation cannot be fulfilled on nodes \"jenkins-e2e-master\": <nil>\n [] <nil> 0xc820606540 exit status 1 <nil> true [0xc8200300b0 0xc8200301d0 0xc8200305f8] [0xc8200300b0 0xc8200301d0 0xc8200305f8] [0xc820030100 0xc820030410] [0xaaf610 0xaaf610] 0xc820b55020}:\nCommand stdout:\n\nstderr:\nError from server: Operation cannot be fulfilled on nodes \"jenkins-e2e-master\": <nil>\n\nerror:\nexit status 1\n",
    }
    Error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.154.238.217 --kubeconfig=/workspace/.kube/config taint nodes jenkins-e2e-master kubernetes.io/e2e-taint-key-5c9b1cf3-6736-11e6-a811-0242ac110003=testing-taint-value:NoSchedule] []  <nil>  Error from server: Operation cannot be fulfilled on nodes "jenkins-e2e-master": <nil>
     [] <nil> 0xc820606540 exit status 1 <nil> true [0xc8200300b0 0xc8200301d0 0xc8200305f8] [0xc8200300b0 0xc8200301d0 0xc8200305f8] [0xc820030100 0xc820030410] [0xaaf610 0xaaf610] 0xc820b55020}:
    Command stdout:

    stderr:
    Error from server: Operation cannot be fulfilled on nodes "jenkins-e2e-master": <nil>

    error:
    exit status 1

not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1974
@k8s-github-robot k8s-github-robot added priority/backlog Higher priority than priority/awaiting-more-evidence. kind/flake Categorizes issue or PR as related to a flaky test. labels Aug 21, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/13804/

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1287
Aug 21 16:32:16.664: Failed to find kubernetes.io/e2e-taint-key-7e49b5f3-67f7-11e6-81d0-0242ac110002=another-testing-taint-value:PreferNoSchedule in Name:          gke-jenkins-e2e-default-pool-6acbee6f-713p
Labels:         beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/instance-type=n1-standard-2
            beta.kubernetes.io/os=linux
            cloud.google.com/gke-nodepool=default-pool
            failure-domain.beta.kubernetes.io/region=us-central1
            failure-domain.beta.kubernetes.io/zone=us-central1-f
            kubernetes.io/hostname=gke-jenkins-e2e-default-pool-6acbee6f-713p
Taints:         kubernetes.io/e2e-taint-key-7e49b5f3-67f7-11e6-81d0-0242ac110002=testing-taint-value:NoSchedule
CreationTimestamp:  Sun, 21 Aug 2016 16:29:46 -0700
Phase:          
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  NetworkUnavailable    False   Sun, 21 Aug 2016 16:32:06 -0700     Sun, 21 Aug 2016 16:32:06 -0700     RouteCreated            RouteController created a route
  OutOfDisk         False   Sun, 21 Aug 2016 16:32:08 -0700     Sun, 21 Aug 2016 16:29:46 -0700     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Sun, 21 Aug 2016 16:32:08 -0700     Sun, 21 Aug 2016 16:29:46 -0700     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      False   Sun, 21 Aug 2016 16:32:08 -0700     Sun, 21 Aug 2016 16:29:46 -0700     KubeletHasNoDiskPressure    kubelet has no disk pressure
  Ready         True    Sun, 21 Aug 2016 16:32:08 -0700     Sun, 21 Aug 2016 16:30:37 -0700     KubeletReady            kubelet is posting ready status. WARNING: CPU hardcapping unsupported
Addresses:      10.240.0.6,104.198.55.51
Capacity:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   2
 memory:                7679820Ki
 pods:                  110
Allocatable:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   2
 memory:                7679820Ki
 pods:                  110
System Info:
 Machine ID:                
 System UUID:               A801BFF9-967D-80DE-5539-9195D83A9652
 Boot ID:               2e487a07-f832-4b5a-b4ca-5759ff5606c7
 Kernel Version:            3.16.0-4-amd64
 OS Image:              Debian GNU/Linux 7 (wheezy)
 Operating System:          linux
 Architecture:              amd64
 Container Runtime Version:     docker://1.11.2
 Kubelet Version:           v1.4.0-alpha.2.1418+1def4a0458825a
 Kube-Proxy Version:            v1.4.0-alpha.2.1418+1def4a0458825a
PodCIDR:                10.180.1.0/24
ExternalID:             367965509999101468
Non-terminated Pods:            (7 in total)
  Namespace             Name                                        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------             ----                                        ------------    ----------  --------------- -------------
  e2e-tests-job-9nay7           foo-138fz                                   0 (0%)      0 (0%)      0 (0%)      0 (0%)
  e2e-tests-volume-provisioning-rxcn9   pvc-volume-tester-rafz2                             0 (0%)      0 (0%)      0 (0%)      0 (0%)
  e2e-tests-volume-provisioning-yf4bi   pvc-volume-tester-y0e7s                             0 (0%)      0 (0%)      0 (0%)      0 (0%)
  kube-system               fluentd-cloud-logging-gke-jenkins-e2e-default-pool-6acbee6f-713p        100m (5%)   0 (0%)      200Mi (2%)  200Mi (2%)
  kube-system               heapster-v1.2.0-beta.1-632857166-68yi4                      138m (6%)   138m (6%)   301856Ki (3%)   301856Ki (3%)
  kube-system               kube-dns-v19-kie9g                              110m (5%)   110m (5%)   120Mi (1%)  220Mi (2%)
  kube-system               kube-proxy-gke-jenkins-e2e-default-pool-6acbee6f-713p               100m (5%)   0 (0%)      0 (0%)      0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  448m (22%)    248m (12%)  629536Ki (8%)   731936Ki (9%)
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                            -------------   --------    ------          -------
  2m        2m      1   {kubelet gke-jenkins-e2e-default-pool-6acbee6f-713p}            Normal      Starting        Starting kubelet.
  2m        2m      8   {kubelet gke-jenkins-e2e-default-pool-6acbee6f-713p}            Normal      NodeHasSufficientDisk   Node gke-jenkins-e2e-default-pool-6acbee6f-713p status is now: NodeHasSufficientDisk
  2m        2m      8   {kubelet gke-jenkins-e2e-default-pool-6acbee6f-713p}            Normal      NodeHasSufficientMemory Node gke-jenkins-e2e-default-pool-6acbee6f-713p status is now: NodeHasSufficientMemory
  2m        2m      8   {kubelet gke-jenkins-e2e-default-pool-6acbee6f-713p}            Normal      NodeHasNoDiskPressure   Node gke-jenkins-e2e-default-pool-6acbee6f-713p status is now: NodeHasNoDiskPressure
  1m        1m      1   {kube-proxy gke-jenkins-e2e-default-pool-6acbee6f-713p}         Normal      Starting        Starting kube-proxy.
  1m        1m      1   {kubelet gke-jenkins-e2e-default-pool-6acbee6f-713p}            Normal      NodeReady       Node gke-jenkins-e2e-default-pool-6acbee6f-713p status is now: NodeReady

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1370

@spxtr spxtr assigned kevin-wangzefeng and unassigned spxtr Aug 22, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30652/kubernetes-pull-build-test-e2e-gce/55375/

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1287
Expected error:
    <exec.CodeExitError>: {
        Err: {
            s: "error running &{/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.154.31.16 --kubeconfig=/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce/.kube/config taint nodes e2e-gce-agent-pr-22-0-master kubernetes.io/e2e-taint-key-53cb1156-68c3-11e6-98f6-42010af00043=testing-taint-value:NoSchedule] []  <nil>  Error from server: Operation cannot be fulfilled on nodes \"e2e-gce-agent-pr-22-0-master\": <nil>\n [] <nil> 0xc8208aca60 exit status 1 <nil> true [0xc820cbe008 0xc820cbe028 0xc820cbe040] [0xc820cbe008 0xc820cbe028 0xc820cbe040] [0xc820cbe020 0xc820cbe038] [0xab34b0 0xab34b0] 0xc8207b2fc0}:\nCommand stdout:\n\nstderr:\nError from server: Operation cannot be fulfilled on nodes \"e2e-gce-agent-pr-22-0-master\": <nil>\n\nerror:\nexit status 1\n",
        },
        Code: 1,
    }
    error running &{/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.154.31.16 --kubeconfig=/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce/.kube/config taint nodes e2e-gce-agent-pr-22-0-master kubernetes.io/e2e-taint-key-53cb1156-68c3-11e6-98f6-42010af00043=testing-taint-value:NoSchedule] []  <nil>  Error from server: Operation cannot be fulfilled on nodes "e2e-gce-agent-pr-22-0-master": <nil>
     [] <nil> 0xc8208aca60 exit status 1 <nil> true [0xc820cbe008 0xc820cbe028 0xc820cbe040] [0xc820cbe008 0xc820cbe028 0xc820cbe040] [0xc820cbe020 0xc820cbe038] [0xab34b0 0xab34b0] 0xc8207b2fc0}:
    Command stdout:

    stderr:
    Error from server: Operation cannot be fulfilled on nodes "e2e-gce-agent-pr-22-0-master": <nil>

    error:
    exit status 1

not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1976

Happened on a presubmit run in #30652.

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Aug 23, 2016
@k8s-github-robot k8s-github-robot mentioned this issue Aug 25, 2016
Closed
@caesarxuchao
Copy link
Member

Looks like the kubectl doesn't handle the node update conflict. @kevin-wangzefeng do you mind if I take the issue since you are not working on it?

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/31156/kubernetes-pull-build-test-e2e-gce/56251/

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1287
Aug 26 14:50:32.917: Failed to find kubernetes.io/e2e-taint-key-1b6042aa-6bd7-11e6-93a6-0242ac110002=another-testing-taint-value:PreferNoSchedule in Name:          e2e-gce-agent-pr-47-0-master
Labels:         agent-pr-47=
            beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/instance-type=n1-standard-1
            beta.kubernetes.io/os=linux
            failure-domain.beta.kubernetes.io/region=us-central1
            failure-domain.beta.kubernetes.io/zone=us-central1-f
            kubernetes.io/hostname=e2e-gce-agent-pr-47-0-master
Taints:         kubernetes.io/e2e-taint-key-1b6042aa-6bd7-11e6-93a6-0242ac110002=testing-taint-value:NoSchedule
CreationTimestamp:  Fri, 26 Aug 2016 14:45:45 -0700
Phase:          
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  NetworkUnavailable    False   Fri, 26 Aug 2016 14:50:31 -0700     Fri, 26 Aug 2016 14:50:31 -0700     RouteCreated            RouteController created a route
  OutOfDisk         False   Fri, 26 Aug 2016 14:50:27 -0700     Fri, 26 Aug 2016 14:45:45 -0700     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Fri, 26 Aug 2016 14:50:27 -0700     Fri, 26 Aug 2016 14:45:45 -0700     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      False   Fri, 26 Aug 2016 14:50:27 -0700     Fri, 26 Aug 2016 14:45:45 -0700     KubeletHasNoDiskPressure    kubelet has no disk pressure
  Ready         True    Fri, 26 Aug 2016 14:50:27 -0700     Fri, 26 Aug 2016 14:45:45 -0700     KubeletReady            kubelet is posting ready status
Addresses:      10.240.0.2,104.154.103.138
Capacity:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   1
 memory:                3787412Ki
 pods:                  110
Allocatable:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   1
 memory:                3787412Ki
 pods:                  110
System Info:
 Machine ID:            ec0c187f6c522664dafc164357c0b80a
 System UUID:           0A234A4D-924B-8E6D-D652-10FFD15F30C5
 Boot ID:           33abfe9a-eb0f-4840-9c5a-90766620d57d
 Kernel Version:        4.4.4+
 OS Image:          Google Container-VM Image
 Operating System:      linux
 Architecture:          amd64
 Container Runtime Version: docker://1.11.2
 Kubelet Version:       v1.4.0-alpha.3.116+95c1c2b965468c
 Kube-Proxy Version:        v1.4.0-alpha.3.116+95c1c2b965468c
PodCIDR:            10.180.0.0/24
ExternalID:         4145006453581482269
Non-terminated Pods:        (9 in total)
  Namespace         Name                                CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                                ------------    ----------  --------------- -------------
  kube-system           etcd-empty-dir-cleanup-e2e-gce-agent-pr-47-0-master     0 (0%)      0 (0%)      0 (0%)      0 (0%)
  kube-system           etcd-server-e2e-gce-agent-pr-47-0-master            200m (20%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           etcd-server-events-e2e-gce-agent-pr-47-0-master         100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-addon-manager-e2e-gce-agent-pr-47-0-master         5m (0%)     0 (0%)      50Mi (1%)   0 (0%)
  kube-system           kube-apiserver-e2e-gce-agent-pr-47-0-master         250m (25%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-controller-manager-e2e-gce-agent-pr-47-0-master        200m (20%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-scheduler-e2e-gce-agent-pr-47-0-master         100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           l7-lb-controller-v0.7.0-e2e-gce-agent-pr-47-0-master        10m (1%)    100m (10%)  50Mi (1%)   100Mi (2%)
  kube-system           node-problem-detector-v0.1-9mcsv                20m (2%)    200m (20%)  20Mi (0%)   100Mi (2%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  885m (88%)    300m (30%)  120Mi (3%)  200Mi (5%)
Events:
  FirstSeen LastSeen    Count   From                    SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                    -------------   --------    ------          -------
  5m        5m      1   {kubelet e2e-gce-agent-pr-47-0-master}          Normal      Starting        Starting kubelet.
  5m        5m      1   {kubelet e2e-gce-agent-pr-47-0-master}          Normal      NodeNotSchedulable  Node e2e-gce-agent-pr-47-0-master status is now: NodeNotSchedulable
  5m        4m      46  {kubelet e2e-gce-agent-pr-47-0-master}          Normal      NodeHasSufficientDisk   Node e2e-gce-agent-pr-47-0-master status is now: NodeHasSufficientDisk
  5m        4m      46  {kubelet e2e-gce-agent-pr-47-0-master}          Normal      NodeHasSufficientMemory Node e2e-gce-agent-pr-47-0-master status is now: NodeHasSufficientMemory
  5m        4m      46  {kubelet e2e-gce-agent-pr-47-0-master}          Normal      NodeHasNoDiskPressure   Node e2e-gce-agent-pr-47-0-master status is now: NodeHasNoDiskPressure

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1370

Happened on a presubmit run in #31156.

@k8s-github-robot k8s-github-robot added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Aug 26, 2016
@kevin-wangzefeng kevin-wangzefeng mentioned this issue Aug 27, 2016
Merged
@kevin-wangzefeng
Copy link
Member

@caesarxuchao sorry for the replying late, I just sent #31578 to fix this.

@caesarxuchao
Copy link
Member

Thanks, @kevin-wangzefeng! But there are two variations of the flake here. Will the retry fix this variation: #31066 (comment)? Do you have a theory why the taint is missing?

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/22386/

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1287
Aug 28 04:03:37.962: Failed to find kubernetes.io/e2e-taint-key-1145b6e7-6d0f-11e6-9ba4-0242ac110003=another-testing-taint-value:PreferNoSchedule in Name:          jenkins-e2e-master
Labels:         agent-light-2=
            beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/instance-type=n1-standard-1
            beta.kubernetes.io/os=linux
            failure-domain.beta.kubernetes.io/region=us-central1
            failure-domain.beta.kubernetes.io/zone=us-central1-f
            kubernetes.io/hostname=jenkins-e2e-master
Taints:         kubernetes.io/e2e-taint-key-1145b6e7-6d0f-11e6-9ba4-0242ac110003=testing-taint-value:NoSchedule
            kubernetes.io/e2e-taint-key-11d55714-6d0f-11e6-be40-0242ac110003=testing-taint-value:NoSchedule
CreationTimestamp:  Sun, 28 Aug 2016 03:58:56 -0700
Phase:          
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  NetworkUnavailable    False   Sun, 28 Aug 2016 04:03:31 -0700     Sun, 28 Aug 2016 04:03:31 -0700     RouteCreated            RouteController created a route
  OutOfDisk         False   Sun, 28 Aug 2016 04:03:36 -0700     Sun, 28 Aug 2016 03:58:56 -0700     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Sun, 28 Aug 2016 04:03:36 -0700     Sun, 28 Aug 2016 03:58:56 -0700     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      False   Sun, 28 Aug 2016 04:03:36 -0700     Sun, 28 Aug 2016 03:58:56 -0700     KubeletHasNoDiskPressure    kubelet has no disk pressure
  Ready         True    Sun, 28 Aug 2016 04:03:36 -0700     Sun, 28 Aug 2016 03:58:56 -0700     KubeletReady            kubelet is posting ready status
Addresses:      10.240.0.2,146.148.36.127
Capacity:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   1
 memory:                3787412Ki
 pods:                  110
Allocatable:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   1
 memory:                3787412Ki
 pods:                  110
System Info:
 Machine ID:            9fbccb94265b488e803032de57c2c373
 System UUID:           347BB69C-C713-7101-B12E-901FCB9F1E76
 Boot ID:           c1263c0e-aa06-4631-a518-76a84a625381
 Kernel Version:        4.4.4+
 OS Image:          Google Container-VM Image
 Operating System:      linux
 Architecture:          amd64
 Container Runtime Version: docker://1.11.2
 Kubelet Version:       v1.4.0-alpha.3.155+72fbb5193b839a
 Kube-Proxy Version:        v1.4.0-alpha.3.155+72fbb5193b839a
PodCIDR:            10.180.2.0/24
ExternalID:         5943064507410513332
Non-terminated Pods:        (9 in total)
  Namespace         Name                            CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                            ------------    ----------  --------------- -------------
  kube-system           etcd-empty-dir-cleanup-jenkins-e2e-master       0 (0%)      0 (0%)      0 (0%)      0 (0%)
  kube-system           etcd-server-events-jenkins-e2e-master           100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           etcd-server-jenkins-e2e-master              200m (20%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-addon-manager-jenkins-e2e-master           5m (0%)     0 (0%)      50Mi (1%)   0 (0%)
  kube-system           kube-apiserver-jenkins-e2e-master           250m (25%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-controller-manager-jenkins-e2e-master      200m (20%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-scheduler-jenkins-e2e-master           100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           l7-lb-controller-v0.7.0-jenkins-e2e-master      10m (1%)    100m (10%)  50Mi (1%)   100Mi (2%)
  kube-system           node-problem-detector-v0.1-uny8d            20m (2%)    200m (20%)  20Mi (0%)   100Mi (2%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  885m (88%)    300m (30%)  120Mi (3%)  200Mi (5%)
Events:
  FirstSeen LastSeen    Count   From                SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  5m        5m      1   {kubelet jenkins-e2e-master}            Normal      Starting        Starting kubelet.
  5m        5m      1   {kubelet jenkins-e2e-master}            Normal      NodeNotSchedulable  Node jenkins-e2e-master status is now: NodeNotSchedulable
  5m        4m      50  {kubelet jenkins-e2e-master}            Normal      NodeHasSufficientDisk   Node jenkins-e2e-master status is now: NodeHasSufficientDisk
  5m        4m      50  {kubelet jenkins-e2e-master}            Normal      NodeHasSufficientMemory Node jenkins-e2e-master status is now: NodeHasSufficientMemory
  5m        4m      50  {kubelet jenkins-e2e-master}            Normal      NodeHasNoDiskPressure   Node jenkins-e2e-master status is now: NodeHasNoDiskPressure

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1370

k8s-github-robot pushed a commit that referenced this issue Aug 28, 2016
Merge pull request #31578 from kevin-wangzefeng/add-retries-for-taint…
6dd8b97 
…s-e2e

Automatic merge from submit-queue

add retries for add/update/remove taints on node in taints e2e

fixes taint update conflict in taints e2e by adding retries for add/update/remove taints on node.
ref #27655 and #31066
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @caesarxuchao @kevin-wangzefeng

This flaky-test issue would love to have more attention...

@caesarxuchao
Copy link
Member

Thanks, @kevin-wangzefeng! But there are two variations of the flake here. Will the retry fix this variation: #31066 (comment)? Do you have a theory why the taint is missing?

@kevin-wangzefeng
Copy link
Member

@caesarxuchao Thanks for reminding that, I guess there is a mistake in the describe #L2577-L2590. I'm taking a look.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/14450/

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1287
Aug 31 02:51:01.516: Failed to find kubernetes.io/e2e-taint-key-6bc1a54b-6f60-11e6-92a5-0242ac110005=another-testing-taint-value:PreferNoSchedule in Name:          gke-jenkins-e2e-default-pool-f9623c04-5wro
Labels:         beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/instance-type=n1-standard-2
            beta.kubernetes.io/os=linux
            cloud.google.com/gke-nodepool=default-pool
            failure-domain.beta.kubernetes.io/region=us-central1
            failure-domain.beta.kubernetes.io/zone=us-central1-f
            kubernetes.io/hostname=gke-jenkins-e2e-default-pool-f9623c04-5wro
Taints:         kubernetes.io/e2e-taint-key-6bc1a54b-6f60-11e6-92a5-0242ac110005=testing-taint-value:NoSchedule
            kubernetes.io/e2e-taint-key-6c392ce8-6f60-11e6-95d6-0242ac110005=testing-taint-value:NoSchedule
CreationTimestamp:  Wed, 31 Aug 2016 02:49:08 -0700
Phase:          
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  NetworkUnavailable    False   Wed, 31 Aug 2016 02:51:00 -0700     Wed, 31 Aug 2016 02:51:00 -0700     RouteCreated            RouteController created a route
  OutOfDisk         False   Wed, 31 Aug 2016 02:50:53 -0700     Wed, 31 Aug 2016 02:49:08 -0700     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Wed, 31 Aug 2016 02:50:53 -0700     Wed, 31 Aug 2016 02:49:08 -0700     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      False   Wed, 31 Aug 2016 02:50:53 -0700     Wed, 31 Aug 2016 02:49:08 -0700     KubeletHasNoDiskPressure    kubelet has no disk pressure
  Ready         True    Wed, 31 Aug 2016 02:50:53 -0700     Wed, 31 Aug 2016 02:49:43 -0700     KubeletReady            kubelet is posting ready status. WARNING: CPU hardcapping unsupported
Addresses:      10.240.0.4,104.198.34.123
Capacity:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   2
 memory:                7679820Ki
 pods:                  110
Allocatable:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   2
 memory:                7679820Ki
 pods:                  110
System Info:
 Machine ID:            
 System UUID:           9C51539E-A7BF-D83C-499D-29E2D7AE7E14
 Boot ID:           c6a76c89-ac22-4c6f-b8e7-4303114c2b9c
 Kernel Version:        3.16.0-4-amd64
 OS Image:          Debian GNU/Linux 7 (wheezy)
 Operating System:      linux
 Architecture:          amd64
 Container Runtime Version: docker://1.11.2
 Kubelet Version:       v1.4.0-alpha.3.280+ebe733e46983b0
 Kube-Proxy Version:        v1.4.0-alpha.3.280+ebe733e46983b0
PodCIDR:            10.180.1.0/24
ExternalID:         7790909379752696131
Non-terminated Pods:        (4 in total)
  Namespace         Name                                        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                                        ------------    ----------  --------------- -------------
  kube-system           fluentd-cloud-logging-gke-jenkins-e2e-default-pool-f9623c04-5wro        100m (5%)   0 (0%)      200Mi (2%)  200Mi (2%)
  kube-system           heapster-v1.2.0-beta.1-52682022-9454j                       138m (6%)   138m (6%)   301856Ki (3%)   301856Ki (3%)
  kube-system           kube-dns-v19-0ht7w                              110m (5%)   110m (5%)   120Mi (1%)  220Mi (2%)
  kube-system           kube-proxy-gke-jenkins-e2e-default-pool-f9623c04-5wro               100m (5%)   0 (0%)      0 (0%)      0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  448m (22%)    248m (12%)  629536Ki (8%)   731936Ki (9%)
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                            -------------   --------    ------          -------
  1m        1m      1   {kubelet gke-jenkins-e2e-default-pool-f9623c04-5wro}            Normal      Starting        Starting kubelet.
  1m        1m      2   {kubelet gke-jenkins-e2e-default-pool-f9623c04-5wro}            Normal      NodeHasSufficientDisk   Node gke-jenkins-e2e-default-pool-f9623c04-5wro status is now: NodeHasSufficientDisk
  1m        1m      2   {kubelet gke-jenkins-e2e-default-pool-f9623c04-5wro}            Normal      NodeHasSufficientMemory Node gke-jenkins-e2e-default-pool-f9623c04-5wro status is now: NodeHasSufficientMemory
  1m        1m      2   {kubelet gke-jenkins-e2e-default-pool-f9623c04-5wro}            Normal      NodeHasNoDiskPressure   Node gke-jenkins-e2e-default-pool-f9623c04-5wro status is now: NodeHasNoDiskPressure
  1m        1m      1   {kubelet gke-jenkins-e2e-default-pool-f9623c04-5wro}            Normal      NodeReady       Node gke-jenkins-e2e-default-pool-f9623c04-5wro status is now: NodeReady
  1m        1m      1   {kube-proxy gke-jenkins-e2e-default-pool-f9623c04-5wro}         Normal      Starting        Starting kube-proxy.

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1370

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @caesarxuchao @kevin-wangzefeng

This flaky-test issue would love to have more attention.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/31600/kubernetes-pull-build-test-e2e-gce/57001/

Failed: [k8s.io] Kubectl client [k8s.io] Kubectl taint should remove all the taints with the same key off a node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1287
Expected error:
    <exec.CodeExitError>: {
        Err: {
            s: "error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.198.212.189 --kubeconfig=/workspace/.kube/config taint nodes e2e-gce-agent-pr-69-0-master kubernetes.io/e2e-taint-key-d918beae-7070-11e6-a8df-0242ac110002=another-testing-taint-value:PreferNoSchedule] []  <nil>  Error from server: client: etcd cluster is unavailable or misconfigured\n [] <nil> 0xc820ae2a40 exit status 1 <nil> true [0xc820040000 0xc820040030 0xc820040048] [0xc820040000 0xc820040030 0xc820040048] [0xc820040028 0xc820040040] [0xad92e0 0xad92e0] 0xc820dc0420}:\nCommand stdout:\n\nstderr:\nError from server: client: etcd cluster is unavailable or misconfigured\n\nerror:\nexit status 1\n",
        },
        Code: 1,
    }
    error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.198.212.189 --kubeconfig=/workspace/.kube/config taint nodes e2e-gce-agent-pr-69-0-master kubernetes.io/e2e-taint-key-d918beae-7070-11e6-a8df-0242ac110002=another-testing-taint-value:PreferNoSchedule] []  <nil>  Error from server: client: etcd cluster is unavailable or misconfigured
     [] <nil> 0xc820ae2a40 exit status 1 <nil> true [0xc820040000 0xc820040030 0xc820040048] [0xc820040000 0xc820040030 0xc820040048] [0xc820040028 0xc820040040] [0xad92e0 0xad92e0] 0xc820dc0420}:
    Command stdout:

    stderr:
    Error from server: client: etcd cluster is unavailable or misconfigured

    error:
    exit status 1

not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1976

Happened on a presubmit run in #31600.

This was referenced Dec 2, 2016
Closed
Closed
Closed
Closed
Closed
This was referenced Dec 9, 2016
Closed
Closed
This was referenced Dec 17, 2016
Closed
Closed
Closed
Closed
This was referenced Dec 28, 2016
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Jan 7, 2017
Closed
This was referenced Jan 21, 2017
Closed
Closed
Closed
This was referenced Jan 29, 2017
Closed
Closed
Closed
Closed
Closed
This was referenced Feb 24, 2017
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Mar 28, 2017
Closed
This was referenced Apr 9, 2017
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Apr 28, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@caesarxuchao caesarxuchao

@kevin-wangzefeng kevin-wangzefeng

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

update taints e2e, restrict taints operation with key, effect kevin-wangzefeng/kubernetes
6 participants
@caesarxuchao @kevin-wangzefeng @spxtr @pwittrock @bprashanth @k8s-github-robot