The PD does not appear to getting zone labels (failure-domain.beta.kubernetes.io/zone, etc.) applied to it. Description field from a GCE PD provisioned by the test:

{"kubernetes.io/created-for/pv/name":"pv-gce-j5swc","kubernetes.io/created-for/pvc/name":"pvc-nkmbw","kubernetes.io/created-for/pvc/namespace":"e2e-tests-volume-provisioning-9iwlu"}

CC @justinsb

The controller-manager failed to uniquely find the disk:

error getting labels for volume "gke-jenkins-e2e-109093fb-dynamic-pv-gce-j5swc": GCE persistent disk name was found in multiple zones: "gke-jenkins-e2e-109093fb-dynamic-pv-gce-j5swc"  

It might be because there are a ton of leaked disks around in the project. I'll delete them, but @saad-ali is there an existing issue around PD leaks from this test?

Ah, the leak is caused by the same underlying issue -- getDiskByNameUnknownZone thinking that it's finding the disk in multiple zones. So the issue is why getDiskByNameUnknownZone is finding it in multiple zones.

And there it is: gce.findDiskByName returns (nil, nil) if the disk is not found, but gce.getDiskByNameUnknownZone considers the disk to be found if the error returned by gce.FindDiskByName is nil.

I'll send out a fix.

I'm not sure how this test has been passing for the ubernetes-lite on GCE test suite so consistently, given that the same issue would appear at first glance to apply there as well.

Ah, it's one hell of a coincidence that it's been passing for the ubernetes-lite suite, but it does make sense.

It looks like GCE always returns the list of zones in a consistent order; this one, to be precise:

I0418 20:33:13.727509       7 gce.go:297] managing multiple zones: [us-central1-f us-central1-b us-central1-c us-central1-a]

The disk gets successfully found in the ubernetes-lite suite because the found = disk assignment in gce.getDiskByNameUnknownZone is harmless for the first three zones, because disk is nil. Then when us-central1-a comes around, the actual disk is assigned to found, which isn't checked again because us-central1-a happens to be last.

In the GKE test suite, this isn't the case because we're using us-central1-f as the master's zone, so when we check us-central1-b and don't get an error back from gce.findDiskByName, we bail.

Nice sleuthing Alex! What an embarassing bug. Thanks for fixing. I will review your fix in detail now to confirm.

I've confirmed that the tests are passing now that the fix is in.