Still getting random Pod stuck while mounting secret on the GKE 1.3.3.

@trane9991 Based on what you provided, you hit an issue where if a secret fails to unmount it will be unable to mount to any other pod. It was fixed by #28939.

The fix is in v1.3.4, which should be in GKE soon.

Closing this issue. Please re-open if you run into it in v1.3.4+

I'm experiencing this exact issue on 1.3.4 (albeit on AWS running jessie not GKE).

Has anyone successfully found a workaround that doesn't involve patching k8s itself?

We get random pods getting stuck in ContainerCreating (5-10%ish), which quite badly breaks our use case (spinning up containers on user request). We were previously on ~1.2.3 with no problems. I did downgrade to ~1.2.6 but had the same stalled container creation hang.

(Sorry for lack of precise version numbers, on a phone)

@boydgreenfield Based on what you've said you're experiencing this issue both with v1.2 and v1.3 which means it is likely not the issue described above. If you can provide more information I can help you debug: What version of master/node are you running. How are you deploying your pods? For the pods stuck in ContainerCreating, could you grab the /var/log/kubelet.log files from those nodes during the incident and the /var/log/kube-controller-manager.log file from the master.

@saad-ali – I was on a mobile device earlier but will respond with logs for 1.3 in the morning Pacific. Apologies about the less-than-complete error message this evening.

@boydgreenfield No problem! And I'd love to see the 1.2 logs as well if you're experiencing the issue on both and still have access to them.