[k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite} #34654

k8s-github-robot · 2016-10-12T22:33:42Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-release-1.4/1244/

Failed: [k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:74
Oct 12 15:12:11.549: Err : timed out waiting for the condition
. Failed to remove deployment test-paused-deployment pods : &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1/namespaces/e2e-tests-deployment-i9o0y/pods ResourceVersion:2961} Items:[{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:test-paused-deployment-965846846-atlep GenerateName:test-paused-deployment-965846846- Namespace:e2e-tests-deployment-i9o0y SelfLink:/api/v1/namespaces/e2e-tests-deployment-i9o0y/pods/test-paused-deployment-965846846-atlep UID:c47240a0-90c8-11e6-b2f1-42010af0000a ResourceVersion:886 Generation:0 CreationTimestamp:2016-10-12 15:11:04 -0700 PDT DeletionTimestamp:2016-10-12 15:11:30 -0700 PDT DeletionGracePeriodSeconds:0xc82049b250 Labels:map[name:nginx pod-template-hash:965846846] Annotations:map[kubernetes.io/created-by:{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"e2e-tests-deployment-i9o0y","name":"test-paused-deployment-965846846","uid":"c4586158-90c8-11e6-b2f1-42010af0000a","apiVersion":"extensions","resourceVersion":"485"}}
] OwnerReferences:[] Finalizers:[] ClusterName:} Spec:{Volumes:[{Name:default-token-etqt7 VolumeSource:{HostPath:<nil> EmptyDir:<nil> GCEPersistentDisk:<nil> AWSElasticBlockStore:<nil> GitRepo:<nil> Secret:0xc820e0bd70 NFS:<nil> ISCSI:<nil> Glusterfs:<nil> PersistentVolumeClaim:<nil> RBD:<nil> Quobyte:<nil> FlexVolume:<nil> Cinder:<nil> CephFS:<nil> Flocker:<nil> DownwardAPI:<nil> FC:<nil> AzureFile:<nil> ConfigMap:<nil> VsphereVolume:<nil> AzureDisk:<nil>}}] InitContainers:[] Containers:[{Name:nginx Image:gcr.io/google_containers/nginx-slim:0.7 Command:[] Args:[] WorkingDir: Ports:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:default-token-etqt7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:<nil> ReadinessProbe:<nil> Lifecycle:<nil> TerminationMessagePath:/dev/termination-log ImagePullPolicy:IfNotPresent SecurityContext:<nil> Stdin:false StdinOnce:false TTY:false}] RestartPolicy:Always TerminationGracePeriodSeconds:0xc82049b3e0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName:default NodeName:gke-jenkins-e2e-default-pool-45b2bc27-h2ka SecurityContext:0xc820e10e40 ImagePullSecrets:[] Hostname: Subdomain:} Status:{Phase:Pending Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-12 15:11:04 -0700 PDT Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-12 15:11:04 -0700 PDT Reason:ContainersNotReady Message:containers with unready status: [nginx]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-12 15:11:04 -0700 PDT Reason: Message:}] Message: Reason: HostIP:10.240.0.4 PodIP: StartTime:2016-10-12 15:11:04 -0700 PDT InitContainerStatuses:[] ContainerStatuses:[{Name:nginx State:{Waiting:0xc820e52f00 Running:<nil> Terminated:<nil>} LastTerminationState:{Waiting:<nil> Running:<nil> Terminated:<nil>} Ready:false RestartCount:0 Image:gcr.io/google_containers/nginx-slim:0.7 ImageID: ContainerID:}]}}]}
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:249

Previous issues for this test: #28067 #28378 #32692 #33256

The text was updated successfully, but these errors were encountered:

k8s-github-robot · 2016-10-14T09:22:36Z

Builds:
kubernetes-e2e-gci-gke-release-1.4 1561
kubernetes-e2e-gci-gke-staging-parallel 996 1299

Failed: [k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:74
Oct 14 02:09:00.735: Err : timed out waiting for the condition
. Failed to remove deployment test-paused-deployment pods : &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1/namespaces/e2e-tests-deployment-cj6c5/pods ResourceVersion:2452} Items:[{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:test-paused-deployment-965846846-snn0g GenerateName:test-paused-deployment-965846846- Namespace:e2e-tests-deployment-cj6c5 SelfLink:/api/v1/namespaces/e2e-tests-deployment-cj6c5/pods/test-paused-deployment-965846846-snn0g UID:b07e0646-91ed-11e6-8701-42010af00008 ResourceVersion:2244 Generation:0 CreationTimestamp:2016-10-14 02:07:53 -0700 PDT DeletionTimestamp:2016-10-14 02:08:19 -0700 PDT DeletionGracePeriodSeconds:0xc820a5c510 Labels:map[name:nginx pod-template-hash:965846846] Annotations:map[kubernetes.io/created-by:{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"e2e-tests-deployment-cj6c5","name":"test-paused-deployment-965846846","uid":"b06ca0a0-91ed-11e6-8701-42010af00008","apiVersion":"extensions","resourceVersion":"426"}}
] OwnerReferences:[] Finalizers:[] ClusterName:} Spec:{Volumes:[{Name:default-token-hd7av VolumeSource:{HostPath:<nil> EmptyDir:<nil> GCEPersistentDisk:<nil> AWSElasticBlockStore:<nil> GitRepo:<nil> Secret:0xc820c24ae0 NFS:<nil> ISCSI:<nil> Glusterfs:<nil> PersistentVolumeClaim:<nil> RBD:<nil> Quobyte:<nil> FlexVolume:<nil> Cinder:<nil> CephFS:<nil> Flocker:<nil> DownwardAPI:<nil> FC:<nil> AzureFile:<nil> ConfigMap:<nil> VsphereVolume:<nil> AzureDisk:<nil>}}] InitContainers:[] Containers:[{Name:nginx Image:gcr.io/google_containers/nginx-slim:0.7 Command:[] Args:[] WorkingDir: Ports:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:default-token-hd7av ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:<nil> ReadinessProbe:<nil> Lifecycle:<nil> TerminationMessagePath:/dev/termination-log ImagePullPolicy:IfNotPresent SecurityContext:<nil> Stdin:false StdinOnce:false TTY:false}] RestartPolicy:Always TerminationGracePeriodSeconds:0xc820a5c6b0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName:default NodeName:gke-jenkins-e2e-default-pool-94977b5b-lnjk SecurityContext:0xc820e12c00 ImagePullSecrets:[] Hostname: Subdomain:} Status:{Phase:Running Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-14 02:07:53 -0700 PDT Reason: Message:} {Type:Ready Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-14 02:08:47 -0700 PDT Reason: Message:} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-14 02:07:53 -0700 PDT Reason: Message:}] Message: Reason: HostIP:10.240.0.2 PodIP:10.180.0.10 StartTime:2016-10-14 02:07:53 -0700 PDT InitContainerStatuses:[] ContainerStatuses:[{Name:nginx State:{Waiting:<nil> Running:0xc8205a08c0 Terminated:<nil>} LastTerminationState:{Waiting:<nil> Running:<nil> Terminated:<nil>} Ready:true RestartCount:0 Image:gcr.io/google_containers/nginx-slim:0.7 ImageID:docker://sha256:a9bfbd3ba7f240fb561895271c875fe21722d30560e56dbf9598ffde1615d4dc ContainerID:docker://8a534d52928be5369b0ef380f4291f290c3ce7d459b3e80a41633387bc730b69}]}}]}
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:249

k8s-github-robot · 2016-10-17T20:42:35Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-staging-parallel/1169/

Failed: [k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:74
Oct 17 13:10:16.261: Err : timed out waiting for the condition
. Failed to remove deployment test-paused-deployment pods : &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1/namespaces/e2e-tests-deployment-gfxrx/pods ResourceVersion:1845} Items:[{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:test-paused-deployment-965846846-f2o34 GenerateName:test-paused-deployment-965846846- Namespace:e2e-tests-deployment-gfxrx SelfLink:/api/v1/namespaces/e2e-tests-deployment-gfxrx/pods/test-paused-deployment-965846846-f2o34 UID:9058349d-94a5-11e6-b024-42010af00005 ResourceVersion:703 Generation:0 CreationTimestamp:2016-10-17 13:09:09 -0700 PDT DeletionTimestamp:2016-10-17 13:09:35 -0700 PDT DeletionGracePeriodSeconds:0xc820735460 Labels:map[pod-template-hash:965846846 name:nginx] Annotations:map[kubernetes.io/created-by:{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"e2e-tests-deployment-gfxrx","name":"test-paused-deployment-965846846","uid":"902f4dbc-94a5-11e6-b024-42010af00005","apiVersion":"extensions","resourceVersion":"497"}}
] OwnerReferences:[] Finalizers:[] ClusterName:} Spec:{Volumes:[{Name:default-token-zxu6n VolumeSource:{HostPath:<nil> EmptyDir:<nil> GCEPersistentDisk:<nil> AWSElasticBlockStore:<nil> GitRepo:<nil> Secret:0xc820bdcf30 NFS:<nil> ISCSI:<nil> Glusterfs:<nil> PersistentVolumeClaim:<nil> RBD:<nil> Quobyte:<nil> FlexVolume:<nil> Cinder:<nil> CephFS:<nil> Flocker:<nil> DownwardAPI:<nil> FC:<nil> AzureFile:<nil> ConfigMap:<nil> VsphereVolume:<nil> AzureDisk:<nil>}}] InitContainers:[] Containers:[{Name:nginx Image:gcr.io/google_containers/nginx-slim:0.7 Command:[] Args:[] WorkingDir: Ports:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:default-token-zxu6n ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:<nil> ReadinessProbe:<nil> Lifecycle:<nil> TerminationMessagePath:/dev/termination-log ImagePullPolicy:IfNotPresent SecurityContext:<nil> Stdin:false StdinOnce:false TTY:false}] RestartPolicy:Always TerminationGracePeriodSeconds:0xc820735560 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName:default NodeName:gke-jenkins-e2e-default-pool-14e204b9-4n51 SecurityContext:0xc820da4840 ImagePullSecrets:[] Hostname: Subdomain:} Status:{Phase:Pending Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-17 13:09:09 -0700 PDT Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-17 13:09:09 -0700 PDT Reason:ContainersNotReady Message:containers with unready status: [nginx]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-17 13:09:09 -0700 PDT Reason: Message:}] Message: Reason: HostIP:10.240.0.4 PodIP: StartTime:2016-10-17 13:09:09 -0700 PDT InitContainerStatuses:[] ContainerStatuses:[{Name:nginx State:{Waiting:0xc820918360 Running:<nil> Terminated:<nil>} LastTerminationState:{Waiting:<nil> Running:<nil> Terminated:<nil>} Ready:false RestartCount:0 Image:gcr.io/google_containers/nginx-slim:0.7 ImageID: ContainerID:}]}}]}
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:249

mtaufen · 2016-10-20T18:22:34Z

@yujuhong @Kargakis should we suspect the same image-prepull-related flake as in #28067?

yujuhong · 2016-10-20T18:55:45Z

@mtaufen yes, it looks like the image pulling was slower than usual in the one test failure I checked.

I1020 11:38:41.597636    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/mounttest:0.6": "Status: Downloaded newer image for gcr.io/google_containers/mounttest:0.6"
I1020 11:38:43.592997    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/test-webserver:e2e": "Status: Downloaded newer image for gcr.io/google_containers/test-webserver:e2e"
I1020 11:38:43.946608    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/busybox:1.24": "Status: Downloaded newer image for gcr.io/google_containers/busybox:1.24"
I1020 11:38:44.098222    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/busybox:1.24": "Status: Image is up to date for gcr.io/google_containers/busybox:1.24"
I1020 11:38:44.830752    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/serve_hostname:v1.4": "Status: Downloaded newer image for gcr.io/google_containers/serve_hostname:v1.4"
I1020 11:38:45.185037    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/mounttest-user:0.3": "Status: Downloaded newer image for gcr.io/google_containers/mounttest-user:0.3"
I1020 11:38:55.367215    1246 kube_docker_client.go:295] Pulling image "gcr.io/google_containers/redis:e2e": "a3ed95caeb02: Pull complete "
I1020 11:39:05.358979    1246 kube_docker_client.go:295] Pulling image "gcr.io/google_containers/redis:e2e": "43310c2277ff: Extracting [==================================================>]    439 B/439 B"
I1020 11:39:13.392151    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/redis:e2e": "Status: Downloaded newer image for gcr.io/google_containers/redis:e2e"
I1020 11:39:13.668903    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/update-demo:nautilus": "Status: Downloaded newer image for gcr.io/google_containers/update-demo:nautilus"
I1020 11:39:23.856828    1246 kube_docker_client.go:295] Pulling image "gcr.io/google-samples/gb-frontend:v4": "589a9d9a7c64: Extracting [====>                                              ] 5.571 MB/69.57 MB"
I1020 11:39:33.856891    1246 kube_docker_client.go:295] Pulling image "gcr.io/google-samples/gb-frontend:v4": "6fdfa3810627: Extracting [==================================================>]    325 B/325 B"
I1020 11:39:43.857404    1246 kube_docker_client.go:295] Pulling image "gcr.io/google-samples/gb-frontend:v4": "6ab8fdf783f1: Extracting [================================>                  ] 21.63 MB/33.44 MB"
I1020 11:39:53.866261    1246 kube_docker_client.go:295] Pulling image "gcr.io/google-samples/gb-frontend:v4": "58d34c85820d: Pull complete "
I1020 11:40:00.884468    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google-samples/gb-frontend:v4": "Status: Downloaded newer image for gcr.io/google-samples/gb-frontend:v4"
I1020 11:40:05.936712    1246 kube_docker_client.go:298] Stop pulling image "gcr.io/google_containers/nginx-slim:0.7": "Status: Downloaded newer image for gcr.io/google_containers/nginx-slim:0.7"

The test timed out around 11:40, after that, kubelet finally finised pulling "gcr.io/google_containers/nginx-slim:0.7 for the pod.

Quoting myself from earlier: "GKE uses a serialized image puller and doesn't have image prepulling enabled yet"
If this only happens on a few specific, deployment test, perhaps the tests can tolerate a longer latency? Of course, if someone is going to fix this properly for GKE, that'd be the best.
@dchen1107, can we turn off serialized image puller now, or is the docker bug still valid?

k8s-github-robot · 2016-10-21T06:25:18Z

Builds:
ci-kubernetes-e2e-gci-gke-prod-parallel 584 1204
ci-kubernetes-e2e-gci-gke-release-1.4 272 530 691 844
ci-kubernetes-e2e-gci-gke-staging-parallel 331 333 535 1090 1220 1224
kubernetes-e2e-gci-gke-prod-parallel 1716 2518 2560
kubernetes-e2e-gci-gke-release-1.4 1660 1925 1967 2028 2172 2423 2468 2771
kubernetes-e2e-gci-gke-staging-parallel 1845 2317 2641 2697 2736

Failed: [k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:74
Oct 20 22:39:32.069: Err : timed out waiting for the condition
. Failed to remove deployment test-paused-deployment pods : &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1/namespaces/e2e-tests-deployment-j9irf/pods ResourceVersion:3267} Items:[{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:test-paused-deployment-965846846-zbvdk GenerateName:test-paused-deployment-965846846- Namespace:e2e-tests-deployment-j9irf SelfLink:/api/v1/namespaces/e2e-tests-deployment-j9irf/pods/test-paused-deployment-965846846-zbvdk UID:95db8ae5-9750-11e6-abba-42010af0002d ResourceVersion:1332 Generation:0 CreationTimestamp:2016-10-20 22:38:24 -0700 PDT DeletionTimestamp:2016-10-20 22:38:51 -0700 PDT DeletionGracePeriodSeconds:0xc82077c7f0 Labels:map[name:nginx pod-template-hash:965846846] Annotations:map[kubernetes.io/created-by:{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"e2e-tests-deployment-j9irf","name":"test-paused-deployment-965846846","uid":"95d99ef3-9750-11e6-abba-42010af0002d","apiVersion":"extensions","resourceVersion":"1202"}}
] OwnerReferences:[] Finalizers:[] ClusterName:} Spec:{Volumes:[{Name:default-token-gxkse VolumeSource:{HostPath:<nil> EmptyDir:<nil> GCEPersistentDisk:<nil> AWSElasticBlockStore:<nil> GitRepo:<nil> Secret:0xc820560540 NFS:<nil> ISCSI:<nil> Glusterfs:<nil> PersistentVolumeClaim:<nil> RBD:<nil> Quobyte:<nil> FlexVolume:<nil> Cinder:<nil> CephFS:<nil> Flocker:<nil> DownwardAPI:<nil> FC:<nil> AzureFile:<nil> ConfigMap:<nil> VsphereVolume:<nil> AzureDisk:<nil>}}] InitContainers:[] Containers:[{Name:nginx Image:gcr.io/google_containers/nginx-slim:0.7 Command:[] Args:[] WorkingDir: Ports:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:default-token-gxkse ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:<nil> ReadinessProbe:<nil> Lifecycle:<nil> TerminationMessagePath:/dev/termination-log ImagePullPolicy:IfNotPresent SecurityContext:<nil> Stdin:false StdinOnce:false TTY:false}] RestartPolicy:Always TerminationGracePeriodSeconds:0xc82077c910 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName:default NodeName:gke-jenkins-e2e-default-pool-73fb840f-2i7z SecurityContext:0xc82083fb40 ImagePullSecrets:[] Hostname: Subdomain:} Status:{Phase:Pending Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-20 22:38:24 -0700 PDT Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-20 22:38:24 -0700 PDT Reason:ContainersNotReady Message:containers with unready status: [nginx]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-10-20 22:38:24 -0700 PDT Reason: Message:}] Message: Reason: HostIP:10.240.0.3 PodIP: StartTime:2016-10-20 22:38:24 -0700 PDT InitContainerStatuses:[] ContainerStatuses:[{Name:nginx State:{Waiting:0xc820603240 Running:<nil> Terminated:<nil>} LastTerminationState:{Waiting:<nil> Running:<nil> Terminated:<nil>} Ready:false RestartCount:0 Image:gcr.io/google_containers/nginx-slim:0.7 ImageID: ContainerID:}]}}]}
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:249

k8s-github-robot · 2016-10-23T15:29:07Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/18420/

Failed: [k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:146
Oct 23 08:14:04.796: Couldn't delete ns: "e2e-tests-deployment-vmapj": namespace e2e-tests-deployment-vmapj was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed (&errors.errorString{s:"namespace e2e-tests-deployment-vmapj was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed"})
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:360

k8s-github-robot · 2016-11-02T09:40:42Z

Builds:
ci-kubernetes-e2e-gci-gke-prod-parallel 497
ci-kubernetes-e2e-gci-gke-release-1.4 669
kubernetes-e2e-gci-gke-release-1.4 2257
kubernetes-e2e-gci-gke-staging-parallel 2479

Failed: [k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:74
Nov  2 02:07:50.574: Err : timed out waiting for the condition
. Failed to remove deployment test-paused-deployment pods : &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1/namespaces/e2e-tests-deployment-6gfrl/pods ResourceVersion:2836} Items:[{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:test-paused-deployment-965846846-0p2k5 GenerateName:test-paused-deployment-965846846- Namespace:e2e-tests-deployment-6gfrl SelfLink:/api/v1/namespaces/e2e-tests-deployment-6gfrl/pods/test-paused-deployment-965846846-0p2k5 UID:ac7a13c6-a0db-11e6-9553-42010af0002b ResourceVersion:2772 Generation:0 CreationTimestamp:2016-11-02 02:06:43 -0700 PDT DeletionTimestamp:2016-11-02 02:07:09 -0700 PDT DeletionGracePeriodSeconds:0xc820b83ef0 Labels:map[pod-template-hash:965846846 name:nginx] Annotations:map[kubernetes.io/created-by:{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"e2e-tests-deployment-6gfrl","name":"test-paused-deployment-965846846","uid":"ac78638d-a0db-11e6-9553-42010af0002b","apiVersion":"extensions","resourceVersion":"1416"}}
] OwnerReferences:[] Finalizers:[] ClusterName:} Spec:{Volumes:[{Name:default-token-sb2pf VolumeSource:{HostPath:<nil> EmptyDir:<nil> GCEPersistentDisk:<nil> AWSElasticBlockStore:<nil> GitRepo:<nil> Secret:0xc820b0bef0 NFS:<nil> ISCSI:<nil> Glusterfs:<nil> PersistentVolumeClaim:<nil> RBD:<nil> Quobyte:<nil> FlexVolume:<nil> Cinder:<nil> CephFS:<nil> Flocker:<nil> DownwardAPI:<nil> FC:<nil> AzureFile:<nil> ConfigMap:<nil> VsphereVolume:<nil> AzureDisk:<nil>}}] InitContainers:[] Containers:[{Name:nginx Image:gcr.io/google_containers/nginx-slim:0.7 Command:[] Args:[] WorkingDir: Ports:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:default-token-sb2pf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:<nil> ReadinessProbe:<nil> Lifecycle:<nil> TerminationMessagePath:/dev/termination-log ImagePullPolicy:IfNotPresent SecurityContext:<nil> Stdin:false StdinOnce:false TTY:false}] RestartPolicy:Always TerminationGracePeriodSeconds:0xc820930000 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName:default NodeName:gke-jenkins-e2e-default-pool-0f8ef722-t30h SecurityContext:0xc820529040 ImagePullSecrets:[] Hostname: Subdomain:} Status:{Phase:Running Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-11-02 02:06:43 -0700 PDT Reason: Message:} {Type:Ready Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-11-02 02:07:49 -0700 PDT Reason: Message:} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-11-02 02:06:43 -0700 PDT Reason: Message:}] Message: Reason: HostIP:10.240.0.3 PodIP:10.180.1.19 StartTime:2016-11-02 02:06:43 -0700 PDT InitContainerStatuses:[] ContainerStatuses:[{Name:nginx State:{Waiting:<nil> Running:0xc8205c9840 Terminated:<nil>} LastTerminationState:{Waiting:<nil> Running:<nil> Terminated:<nil>} Ready:true RestartCount:0 Image:gcr.io/google_containers/nginx-slim:0.7 ImageID:docker://sha256:a9bfbd3ba7f240fb561895271c875fe21722d30560e56dbf9598ffde1615d4dc ContainerID:docker://1b5f598ed07fb116a9d713f77664795b557b08e0b6de825026ef4f2bad34327d}]}}]}
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:249

0xmichalis · 2016-12-06T14:35:04Z

@derekwaynecarr is the namespace controller deleting resources in an arbitrary sequence or does it always delete Deployments before ReplicaSets or Pods?

0xmichalis · 2017-03-04T21:48:10Z

Failure modes:

The test framework broke before the test even started (on ns creation).
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-pre-release/996/#k8sio-deployment-paused-deployment-should-be-able-to-scale

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:125
Expected error:
    <*errors.errorString | 0xc4203abaf0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:194

Node never became ready.
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-subnet/1314/#k8sio-deployment-paused-deployment-should-be-ignored-by-the-controller

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142
Feb 28 02:14:35.557: All nodes should be ready after test, Not ready nodes: []*api.Node{(*api.Node)(0xc42145aa00)}
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438

3. This one seems like the only one related to the test - broke because the deployment wasn't observed.
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/4867/#k8sio-deployment-paused-deployment-should-be-ignored-by-the-controller

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:79
Expected error:
    <*errors.errorString | 0xc420415620>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:543

The test framework broke before the test even started.
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-release-1.4/4822/#k8sio-deployment-paused-deployment-should-be-ignored-by-the-controller

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133
Expected error:
    <*errors.StatusError | 0xc8203a7700>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "an error on the server (\"Internal Server Error: \\\"/api/v1/watch/namespaces/e2e-tests-deployment-inlzx/serviceaccounts?fieldSelector=metadata.name%3Ddefault\\\"\") has prevented the request from succeeding (get serviceAccounts)",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "serviceAccounts",
                Causes: [
                    {
                        Type: "UnexpectedServerResponse",
                        Message: "Internal Server Error: \"/api/v1/watch/namespaces/e2e-tests-deployment-inlzx/serviceaccounts?fieldSelector=metadata.name%3Ddefault\"",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
    an error on the server ("Internal Server Error: \"/api/v1/watch/namespaces/e2e-tests-deployment-inlzx/serviceaccounts?fieldSelector=metadata.name%3Ddefault\"") has prevented the request from succeeding (get serviceAccounts)
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:223

Namespace deletion failure. At this point the test is successful but a failure to cleanup the namespace, fails the test too.
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/4335/#k8sio-deployment-paused-deployment-should-be-ignored-by-the-controller

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142
Jan 27 03:12:08.875: Couldn't delete ns: "e2e-tests-deployment-sv6g6": namespace e2e-tests-deployment-sv6g6 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0 (&errors.errorString{s:"namespace e2e-tests-deployment-sv6g6 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0"})
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:353

0xmichalis · 2017-03-04T22:50:54Z

This one seems like the only one related to the test - broke because the deployment wasn't observed.

The reason behind this one: #42532

0xmichalis · 2017-03-09T11:24:15Z

Moving to 1.7 as none of the failure modes is related to the test

k8s-github-robot · 2017-06-05T07:51:54Z

This Issue hasn't been active in 87 days. It will be closed in 2 days (Jun 7, 2017).

cc @k8s-merge-robot @Kargakis

You can add 'keep-open' label to prevent this from happening, or add a comment to keep it open another 90 days

k8s-github-robot assigned 0xmichalis Oct 12, 2016

k8s-github-robot added kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. labels Oct 12, 2016

k8s-github-robot mentioned this issue Oct 17, 2016

kubernetes-e2e-gci-gke-prod-parallel: broken test run #34926

Closed

k8s-github-robot mentioned this issue Oct 21, 2016

kubernetes-e2e-gci-gke-release-1.4: broken test run #34682

Closed

This was referenced Oct 28, 2016

kubernetes-e2e-gci-gce-proto: broken test run #34863

Closed

kubernetes-e2e-gci-gke-multizone: broken test run #34670

Closed

k8s-github-robot mentioned this issue Nov 7, 2016

kubernetes-e2e-aws: broken test run #34673

Closed

This was referenced Nov 23, 2016

ci-kubernetes-e2e-gce-etcd3: broken test run #37213

Closed

ci-kubernetes-e2e-aws: broken test run #37256

Closed

ci-kubernetes-soak-gke-test: broken test run #37584

Closed

0xmichalis added the area/workload-api/deployment label Dec 5, 2016

k8s-github-robot mentioned this issue Dec 5, 2016

ci-kubernetes-e2e-kops-aws: broken test run #37891

Closed

k8s-github-robot mentioned this issue Jan 20, 2017

ci-kubernetes-soak-gke-test: broken test run #37898

Closed

This was referenced Jan 27, 2017

ci-kubernetes-e2e-gke-staging-parallel: broken test run #37993

Closed

ci-kubernetes-e2e-gke-gci-1.4-gci-1.5-upgrade-cluster-new: broken test run #38477

Closed

This was referenced Feb 4, 2017

ci-kubernetes-e2e-gci-gke-release-1.4: broken test run #37947

Closed

ci-kubernetes-e2e-gke-staging: broken test run #38100

Closed

k8s-github-robot mentioned this issue Feb 26, 2017

ci-kubernetes-e2e-gci-gce-proto: broken test run #42100

Closed

calebamiles added the sig/apps Categorizes an issue or PR as relevant to SIG Apps. label Feb 28, 2017

This was referenced Feb 28, 2017

ci-kubernetes-e2e-gci-gke-subnet: broken test run #39799

Closed

ci-kubernetes-e2e-gci-gke-pre-release: broken test run #37987

Closed

calebamiles modified the milestone: v1.6 Mar 3, 2017

This was referenced Mar 5, 2017

[k8s.io] Deployment deployment should support rollback when there's replica set with no revision {Kubernetes e2e suite} #38442

Closed

[k8s.io] Deployment deployment should support rollover {Kubernetes e2e suite} #38275

Closed

0xmichalis modified the milestones: v1.7, v1.6 Mar 9, 2017

This was referenced Mar 10, 2017

ci-kubernetes-e2e-gci-gce: broken test run #42159

Closed

ci-kubernetes-e2e-gci-gke-staging: broken test run #40676

Closed

ci-kubernetes-e2e-gci-gke-prod: broken test run #43036

Closed

This was referenced Mar 23, 2017

ci-kubernetes-e2e-gci-gke: broken test run #43507

Closed

ci-kubernetes-e2e-gci-gke-staging: broken test run #43037

Closed

ci-kubernetes-e2e-gci-gce-proto: broken test run #43454

Closed

k8s-github-robot mentioned this issue Apr 2, 2017

ci-kubernetes-e2e-gke-test: broken test run #43549

Closed

This was referenced Apr 24, 2017

ci-kubernetes-e2e-gci-gke-subnet: broken test run #43283

Closed

ci-kubernetes-soak-gke-gci-test: broken test run #38065

Closed

ci-kubernetes-e2e-gci-gke-test: broken test run #44699

Closed

0xmichalis closed this as completed Jun 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite} #34654

[k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite} #34654

k8s-github-robot commented Oct 12, 2016

k8s-github-robot commented Oct 14, 2016 •

edited by rmmh

Loading

k8s-github-robot commented Oct 17, 2016

mtaufen commented Oct 20, 2016 •

edited

Loading

yujuhong commented Oct 20, 2016

k8s-github-robot commented Oct 21, 2016 •

edited by rmmh

Loading

k8s-github-robot commented Oct 23, 2016

k8s-github-robot commented Nov 2, 2016 •

edited by rmmh

Loading

0xmichalis commented Dec 6, 2016

0xmichalis commented Mar 4, 2017 •

edited

Loading

0xmichalis commented Mar 4, 2017

0xmichalis commented Mar 9, 2017

k8s-github-robot commented Jun 5, 2017

[k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite} #34654

[k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite} #34654

Comments

k8s-github-robot commented Oct 12, 2016

k8s-github-robot commented Oct 14, 2016 • edited by rmmh Loading

k8s-github-robot commented Oct 17, 2016

mtaufen commented Oct 20, 2016 • edited Loading

yujuhong commented Oct 20, 2016

k8s-github-robot commented Oct 21, 2016 • edited by rmmh Loading

k8s-github-robot commented Oct 23, 2016

k8s-github-robot commented Nov 2, 2016 • edited by rmmh Loading

0xmichalis commented Dec 6, 2016

0xmichalis commented Mar 4, 2017 • edited Loading

0xmichalis commented Mar 4, 2017

0xmichalis commented Mar 9, 2017

k8s-github-robot commented Jun 5, 2017

k8s-github-robot commented Oct 14, 2016 •

edited by rmmh

Loading

mtaufen commented Oct 20, 2016 •

edited

Loading

k8s-github-robot commented Oct 21, 2016 •

edited by rmmh

Loading

k8s-github-robot commented Nov 2, 2016 •

edited by rmmh

Loading

0xmichalis commented Mar 4, 2017 •

edited

Loading