cc @Random-Liu

@pwittrock @liangchenye
Offline discussed with @caesarxuchao, this is a known issue #19403 (comment).

For short, event though the pod is deleted on the apiserver, kubelet may still try to delete the pod again. If we create a pod with the same name and namespace soon after deleting the it, the new pod will be deleted by kubelet accidentally.

• As a quick fix, we should randomize the pod name for different test cases.
• As a better fix, we should start using the e2e test framework and use different namespace for different test cases.
• In the long run, we should fix kubelet to delete pod by UID, see Add UID to DeleteOptions, using it as a precondition in deletion operations. #22965

This is the same with #24937. This should have caused quite a few flakes, bump up the priority.

/cc @yujuhong

Great. Seems like we have a quick fix then.

For short, event though the pod is deleted on the apiserver, kubelet may still try to delete the pod again. If we create a pod with the same name and namespace soon after deleting the it, the new pod will be deleted by kubelet accidentally.

Don't we check the pod UID before deleting it?

@yujuhong But there is still a race between kubelet and apiserver.

For example:

• Kubelet gets the old pod, and updates the status.
• Apiserver deletes the pod because it exceeds the graceful timeout.
• Test creates the new pod with the same name and namespace.
• Kubelet deletes the new pod.

I'm not sure whether this is the main reason of the flake, but we haven't found any other possible reason yet. :)

Ah...yeah...that jogged my memory. Thanks for the explanation. Deleting by UID would be great.

Don't we check the pod UID before deleting it?

This is not enough, the pod may be deleted and recreated after the check in kubelet and before the deletion request reaches the API server. We need to use #22965, which technically is still "deleting by Name", but you can set the UID as a precondition.