If kubelet is unavailable, AttachDetachController fails to force detach on pod deletion #65392
Description
/kind bug
What you expected to happen:
When a pod using an attached volume is deleted (gracefully) but kubelet in the corresponding node is down, the AttachDetachController should assume the node is unrecoverable after a timeout (currently 6min) and forcefully detach the volume.
What happened:
The volume is never detached.
How to reproduce it (as minimally and precisely as possible):
- Create a pod with a volume using any of the attachable plugins (I used GCE PD to test).
- Stop kubelet inside the node where the pod is scheduled.
- Delete the pod.
- Wait for 6min+
- Check to see if the volume is still attached.
Anything else we need to know?:
This doesn't happen if the pod is force-deleted.
It's likely due to the last condition checked in this line. Once kubelet is down, the container status is no longer reported correctly. Inside the Attach Detach Controller, This function is called by the pod update informer handler, which sets whether the volume should be attached in the desired state of the world. On pod force deletion, the pod object is deleted immediately, and this triggers the pod delete informer handler, which doesn't call this function.
/sig storage
/cc @saad-ali @gnufied @jingxu97 @NickrenREN
/assign