randomize pd test container name #27084

rootfs · 2016-06-08T21:29:24Z

Signed-off-by: Huamin Chen <hchen@redhat.com>

k8s-bot · 2016-06-08T22:13:13Z

GCE e2e build/test passed for commit acfcbff.

fejta · 2016-06-09T00:36:39Z

test/e2e/pd.go

@@ -431,9 +427,10 @@ func detachPD(hostName, pdName string) error {
 func testPDPod(diskNames []string, targetHost string, readOnly bool, numContainers int) *api.Pod {
 	containers := make([]api.Container, numContainers)
 	for i := range containers {
-		containers[i].Name = "mycontainer"
+		name := "mycontainer" + strconv.Itoa(mathrand.Int())


A monotonically increasing number unique to the program might work better here as it eliminates the chance of collision

fejta · 2016-06-09T00:38:14Z

LGTM with some comments. @saad-ali do you want to take a look?

saad-ali · 2016-06-09T00:45:41Z

LGTM

@rootfs did you run the tests to verify they are working. The merge bot doesn't run these tests.

rootfs · 2016-06-09T13:06:54Z

@saad-ali yes i ran tests overnight.

saad-ali · 2016-06-10T01:58:17Z

Taking another look at this PR. It actually won't change anything since the pod/container names are still generated once per test. Every iteration creating and tearing down the same pod will still end up with the same pod/container name. Am I missing something?

rootfs · 2016-06-10T13:04:24Z

@saad-ali before this change, the container name is always 'mycontainer'. This change appends a random number to the container name like the following

Jun  9 16:55:27.298: INFO: Running '/srv/dev/kubernetes/_output/local/bin/linux/amd64/kubectl exec --namespace=e2e-tests-pod-disks-lh723 pd-test-e8c5f20d-2e62-11e6-a1ff-b8ca3a62792c -c=mycontainer36928506542153397011 -- cat /testpd1/tracker2'

ncdc · 2016-06-10T13:53:13Z

From an exec standpoint, it's the pod name that needs to be unique, not the container name. It looks like that's already unique per pod (https://github.com/rootfs/kubernetes/blob/acfcbffd42ae92242c0b949f938152b3814de9c6/test/e2e/pd.go#L457)?

rootfs · 2016-06-10T14:21:28Z

thanks @ncdc for the insight.

The Pod name is created before the test starts, so even it is unique between tests, it is still deterministic in the same test https://github.com/rootfs/kubernetes/blob/acfcbffd42ae92242c0b949f938152b3814de9c6/test/e2e/pd.go#L180

ncdc · 2016-06-10T14:24:04Z

Ok, yeah, I didn't have a chance to read through the entire test file to find that. So yes, you need to create a new pod for each test case and test iteration, especially if you're deleting a pod and recreating.

rootfs · 2016-06-10T14:26:12Z

Let me check the failed the tests again.

rootfs · 2016-06-10T14:38:36Z

@ncdc @saad-ali the following log from doesn't convince me this fix can help. .In the failure, the container names were different (mycontainer3 and mycontainer4) and yet still failed

[k8s.io] Pod Disks 
  should schedule a pod w/ a RW PD shared between multiple containers, write to PD, delete pod, verify contents, and repeat in rapid succession [Slow] [Flaky]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:218
[BeforeEach] [k8s.io] Pod Disks
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120
STEP: Creating a kubernetes client
Jun 10 00:31:50.961: INFO: >>> kubeConfig: /workspace/.kube/config

STEP: Building a namespace api object
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [k8s.io] Pod Disks
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:69
[It] should schedule a pod w/ a RW PD shared between multiple containers, write to PD, delete pod, verify contents, and repeat in rapid succession [Slow] [Flaky]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:218
STEP: creating PD
Jun 10 00:31:54.923: INFO: Successfully created a new PD: "jenkins-e2e-65b25155-2edd-11e6-8aff-0242ac11000a".
Jun 10 00:31:54.923: INFO: PD Read/Writer Iteration #0
STEP: submitting host0Pod to kubernetes
W0610 00:31:54.935102    8734 request.go:347] Field selector: v1 - pods - metadata.name - pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a: need to check if this is versioned correctly.
STEP: writing a file in the container
Jun 10 00:32:04.921: INFO: Running '/workspace/kubernetes/platforms/linux/amd64/kubectl exec --namespace=e2e-tests-pod-disks-8bfcu pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a -c=mycontainer3 -- /bin/sh -c echo '7396730354903675317' > '/testpd1/tracker0''
Jun 10 00:32:05.261: INFO: Wrote value: "7396730354903675317" to PD "jenkins-e2e-65b25155-2edd-11e6-8aff-0242ac11000a" from pod "pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a" container "mycontainer3"
STEP: reading a file in the container
Jun 10 00:32:05.261: INFO: Running '/workspace/kubernetes/platforms/linux/amd64/kubectl exec --namespace=e2e-tests-pod-disks-8bfcu pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a -c=mycontainer4 -- cat /testpd1/tracker0'
Jun 10 00:32:05.487: INFO: Read file "/testpd1/tracker0" with content: 7396730354903675317

STEP: deleting host0Pod
Jun 10 00:32:05.525: INFO: PD Read/Writer Iteration #1
STEP: submitting host0Pod to kubernetes
W0610 00:32:05.532123    8734 request.go:347] Field selector: v1 - pods - metadata.name - pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a: need to check if this is versioned correctly.
STEP: reading a file in the container
Jun 10 00:32:07.055: INFO: Running '/workspace/kubernetes/platforms/linux/amd64/kubectl exec --namespace=e2e-tests-pod-disks-8bfcu pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a -c=mycontainer1 -- cat /testpd1/tracker0'
Jun 10 00:32:07.272: INFO: Read file "/testpd1/tracker0" with content: 7396730354903675317

STEP: writing a file in the container
Jun 10 00:32:07.273: INFO: Running '/workspace/kubernetes/platforms/linux/amd64/kubectl exec --namespace=e2e-tests-pod-disks-8bfcu pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a -c=mycontainer3 -- /bin/sh -c echo '2662366901720958409' > '/testpd1/tracker1''
Jun 10 00:32:07.509: INFO: Wrote value: "2662366901720958409" to PD "jenkins-e2e-65b25155-2edd-11e6-8aff-0242ac11000a" from pod "pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a" container "mycontainer3"
STEP: reading a file in the container
Jun 10 00:32:07.509: INFO: Running '/workspace/kubernetes/platforms/linux/amd64/kubectl exec --namespace=e2e-tests-pod-disks-8bfcu pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a -c=mycontainer4 -- cat /testpd1/tracker0'
Jun 10 00:32:07.732: INFO: Read file "/testpd1/tracker0" with content: 7396730354903675317

STEP: reading a file in the container
Jun 10 00:32:07.733: INFO: Running '/workspace/kubernetes/platforms/linux/amd64/kubectl exec --namespace=e2e-tests-pod-disks-8bfcu pd-test-6805af1f-2edd-11e6-8aff-0242ac11000a -c=mycontainer4 -- cat /testpd1/tracker1'
Jun 10 00:32:07.876: INFO: error running kubectl exec to read file: exit status 1
stdout=rpc error: code = 2 desc = "oci runtime error: exec failed: lstat /proc/10488/ns/ipc: no such file or directory"
stderr=error: error executing remote command: error executing command in container: Error executing in Docker Container: 126
)
Jun 10 00:32:07.876: INFO: Error reading file: exit status 1
Jun 10 00:32:07.876: INFO: Unexpected error occurred: exit status 1
STEP: cleaning up PD-RW test environment
STEP: Waiting for PD "jenkins-e2e-65b25155-2edd-11e6-8aff-0242ac11000a" to detach from "jenkins-e2e-minion-group-9hts"
Jun 10 00:32:14.756: INFO: GCE PD "jenkins-e2e-65b25155-2edd-11e6-8aff-0242ac11000a" appears to have successfully detached from "jenkins-e2e-minion-group-9hts".
STEP: Deleting PD "jenkins-e2e-65b25155-2edd-11e6-8aff-0242ac11000a"
Jun 10 00:32:18.587: INFO: Successfully deleted PD "jenkins-e2e-65b25155-2edd-11e6-8aff-0242ac11000a".

ncdc · 2016-06-10T14:43:16Z

Correct, this won't help. You can't do this reliably:

Create pod abcd
Delete pod abcd
Create pod abcd
kubectl exec abcd -- <commands>

The problem is that you have 2 pods with the same name but different UUIDs, and exec doesn't support UUIDs right now, so it picks the "wrong" (deleted) pod from steps 1/2.

randomize pd test pod name

acfcbff

Signed-off-by: Huamin Chen <hchen@redhat.com>

googlebot added the cla: yes label Jun 8, 2016

k8s-github-robot assigned fejta Jun 8, 2016

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-label-needed labels Jun 8, 2016

fejta reviewed Jun 9, 2016
View reviewed changes

fejta assigned saad-ali Jun 9, 2016

saad-ali added this to the v1.3 milestone Jun 9, 2016

saad-ali added lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Jun 9, 2016

rootfs closed this Jun 10, 2016

rootfs deleted the pd-fix branch June 10, 2016 15:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

randomize pd test container name #27084

randomize pd test container name #27084

rootfs commented Jun 8, 2016

k8s-bot commented Jun 8, 2016

fejta Jun 9, 2016 •

edited

Loading

fejta commented Jun 9, 2016

saad-ali commented Jun 9, 2016

rootfs commented Jun 9, 2016

saad-ali commented Jun 10, 2016

rootfs commented Jun 10, 2016

ncdc commented Jun 10, 2016

rootfs commented Jun 10, 2016

ncdc commented Jun 10, 2016

rootfs commented Jun 10, 2016

rootfs commented Jun 10, 2016

ncdc commented Jun 10, 2016

randomize pd test container name #27084

randomize pd test container name #27084

Conversation

rootfs commented Jun 8, 2016

k8s-bot commented Jun 8, 2016

fejta Jun 9, 2016 • edited Loading

Choose a reason for hiding this comment

fejta commented Jun 9, 2016

saad-ali commented Jun 9, 2016

rootfs commented Jun 9, 2016

saad-ali commented Jun 10, 2016

rootfs commented Jun 10, 2016

ncdc commented Jun 10, 2016

rootfs commented Jun 10, 2016

ncdc commented Jun 10, 2016

rootfs commented Jun 10, 2016

rootfs commented Jun 10, 2016

ncdc commented Jun 10, 2016

fejta Jun 9, 2016 •

edited

Loading