IMO it's not a problem. In Jenkins we do turn down clusters between runs, and when running manually it's all in hands of the person running.

In my case, when running tests I was always waiting for proper namespace deletion. Delete indeed returns straight away, but get namespace still returns namespace in question as "terminating". It's slightly faster than turning down the cluster.

In addition if you want to save the time you don't want to delete pods used to saturate the cluster (those first 30/node), as bringing them up again takes 7-8 minutes. Instead you just play with the test commenting out pieces you don't want to use, rebuild the test, which takes seconds and run it again.
Deleting additional 100 pods together with associated events is pretty quick.

I believe we can close this issue.

Instead you just play with the test commenting out pieces you don't want to use, rebuild the test, which takes seconds and run it again.
Deleting additional 100 pods together with associated events is pretty quick.

Sure, there's doing something like that, and theres leaving a clean environment for the next test by default so your tests can run in any order. We have iterations there as a feature, and I think we still run density as part of e2e, so anything that runs after it feels the after effects. It's just too easy to shoot yourself in the foot if you don't do proper cleanup in a test, assuming it's always giong to run alone in a fresh cluster doesn't sound like a good idea even if it happens to work today.

IIUC this (i.e. test isolation) is deeper issue, not only for the Density, but also Load test, and any other e2e that creates non-trivial number of resources.

We no longer run those tests as part of regular gce e2e test - see #9858

See #3954 (comment)
I don't mind making strong-cleanup a flag that the comment out crowd can tweak at will, but a test leaking state feels broken.

IIUC this (i.e. test isolation) is deeper issue, not only for the Density, but also Load test, and any other e2e that creates non-trivial number of resources.

We don't really have many of those in wide use today. Most e2es start 1,2 pods and don't care about latency, so the suite (on the whole) prioritizes throughput. Load tests like density are a special case because even their cleanup can disrupt the next run. But I agree we have a more general test cleanup problem :(

The events deletion at least is handled by #10175.

Yup. I believe we can close this bug.

Hmm, there's still the problem of cleaning up containers on the kubelet (delete just removes keys from etcd currently)

/cc @rrati

@bprashanth details?

kubectl delete pods doesn't wait till the kubelet has deleted containers on the node

Ohh yeah, that's been that way forever, it's non-blocking. It was my understanding that it was unlikely to change.

@quinton-hoole @wojtek-t is this still a thing?

Currently it's hacked - to provide a clear and fast solution we need to be able to quickly remove the whole namespace and to do it we need to be able to remove a collection of objects in a single step.
But I think we can close it now as a duplicate of #10380