This is a duplication of #26728, I'm looking into it.

This does not seem to be related to persistent volumes, the test is unable to delete a namespace:

Jun  1 22:01:48.966: INFO: Couldn't delete ns "e2e-tests-volume-provisioning-9pmxr": the server does not allow access to the requested resource (delete namespaces e2e-tests-volume-provisioning-9pmxr)

The namespace might contain a PersistentVolumeClaim or pods in various stages of termination, however it should be possible to delete it without errors.

Thanks @caesarxuchao for looking into it.

After consulting with Chao we are moving this to team CSI due to namespace deletion.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/18298/

Failed: [k8s.io] Dynamic provisioning [k8s.io] DynamicProvisioner should create and delete persistent volumes {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:123
Expected error:
    <*errors.errorString | 0xc820836090>: {
        s: "PersistentVolume pvc-ee236ef7-2cbe-11e6-9e3e-42010af00002 still exists within 20m0s",
    }
    PersistentVolume pvc-ee236ef7-2cbe-11e6-9e3e-42010af00002 still exists within 20m0s
not to have occurred

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/18298/

This is a different flake, volume controller can't delete provisioned GCE PD for 20 minutes:

I0607 15:00:16.246079       6 gce_util.go:58] Error deleting GCE PD volume jenkins-e2e-dynamic-pvc-ee236ef7-2cbe-11e6-9e3e-42010af00002: googleapi: Error 400: The disk resource 'jenkins-e2e-dynamic-pvc-ee236ef7-2cbe-11e6-9e3e-42010af00002' is already being used by 'jenkins-e2e-minion-group-zuq0', resourceInUseByAnotherResource

@saad-ali, is it a bug in #26351?

This is a different flake, volume controller can't delete provisioned GCE PD for 20 minutes:

Yes, 18298 hit a different issue than the original one being tracked by this bug. The persistent volume controller is unable to delete the provisioned volume because it is still attached after the test is complete. It remains attached because the attach_detach_controller apparently did not receive, or missed, the pod delete event.

#26406 is open to force the shared informer to resync. But resync wouldn't necessarily send out pod delete events, so it would be worth implementing a manual verifier that confirms each pod still exists periodically. Opened #26994 to track that.

because the attach_detach_controller apparently did not receive, or missed, the pod delete event

Missing delete event may be caused by #27004. Suggested work-around (#26994) should be implemented in addition to fixing that issue to prevent bugs in the informer from affecting the controller.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9472/

Failed: [k8s.io] Dynamic provisioning [k8s.io] DynamicProvisioner should create and delete persistent volumes {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:123
Expected error:
    <*errors.errorString | 0xc82056d6d0>: {
        s: "PersistentVolumeClaim pvc-6au2x not in phase Bound within 5m0s",
    }
    PersistentVolumeClaim pvc-6au2x not in phase Bound within 5m0s
not to have occurred

Looks like there are three different flavors of failure for this test:

1. server does not allow access to the requested resource
   • Status: @matchstick--"After consulting with Chao we are moving this to team CSI due to namespace deletion."
2. PersistentVolume pvc-ee236ef7-2cbe-11e6-9e3e-42010af00002 still exists within 20m0s
   • Resolution: Fix tracked by Race condition in Informer #27004. Work around tracked by Attach Detach Controller should verify pods exist periodically #26994.
3. PersistentVolumeClaim pvc-6au2x not in phase Bound within 5m0s

@jsafrane Could you take a look at 3 and see if the root cause is different from the others?

@jsafrane Could you take a look at 3

I'd love to, but I can't find any kube-controller-manager.log

#27004 should be fixed by #27435

but I can't find any kube-controller-manager.log

GKE had logging issues, which is fixed now. We can close this issue and wait for new occurrences. Is there any objection?

No objection. Closing.