This change is 

I need to get a test case for this tomorrow, but this came up in a scenario where we create 100s of namespaces, each namespace with a 100 pods, and we en-masse delete all namespaces. With about 10k pods, we would end up with ~70 or so pods that could never be deleted. Inspection of the state showed that the pod had a gracePeriodSeconds=0 and a deletionTimestamp=some_value . If a pod was in the state, forceful deletion would never occur. As for why the resource was in this state, is reason for another investigation yet to happen, but it appears that when a kubelet did a force-deletion, the deletion was given an OK response, but the object was not truly removed and instead had its local state updated as above. Once this happened, it was never able to be removed without direct access to etcd.

I think this is a release-blocker for 1.5, and I will need to cherry-pick to 1.4.x I think as well.

/cc @smarterclayton @ingvagabund @sjenning @caesarxuchao @deads2k @saad-ali @eparis

Logically this PR makes sense, should be a small change to pkg/api/rest/resttest to verify it.

In 1.7 once we move to etcd3 I will open a follow up item to set grace period to 0 and then delete the pod in a single transaction, which should avoid the potential race error here (we want consumers to see 0 and observe the deletion as discrete steps).

Marking do-not-merge pending unit tests

Jenkins GCE e2e failed for commit 1ec5411. Full PR test history.

The magic incantation to run this job again is @k8s-bot cvm gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Jenkins GCE etcd3 e2e failed for commit 1ec5411. Full PR test history.

The magic incantation to run this job again is @k8s-bot gce etcd3 e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@smarterclayton -- added test case.

The reason this happened is if the update succeeded, and the delete failed (due to etcd being temporarily unreachable, or having an internal error), we would find ourselves in a non-recoverable state.

Jenkins GCI GCE e2e failed for commit 1473084. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-bot gci gce e2e test this

Automatic merge from submit-queue

Commit found in the "release-1.5" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.