Have you run e2e tests against this to confirm that it reduces flakyness?

1st round of e2e's passed. I'll run ~10 (up, Services-DNS-test, down) loops to see if I can coax any flakiness.

btw, doing up/test/down in a timely fashion is exactly what https://github.com/GoogleCloudPlatform/kubernetes/blob/master/hack/parallel-e2e.sh is for. :)

Between @zmerlynn and I, we've gotten ~15 up-DNS-down runs without a failure.

For what it's worth, I have run the tests 8 times locally now, without this fix, and none of them failed. So I'm not sure that ~15 local clean runs are a sure sign of improvement, but I'm happy to merge on the basis that it's unlikely to make things substantially worse.

I'm not sure what version you were testing against in #7453 when you managed to reproduce the failing DNS test, but I just noticed that since kubernetes-ete-gce run 5340 at 11:17am today, the DNS e2e test has not failed once (i.e. even without the fix in this PR).

I couldn't help noticing that the following went into the above successful builds:

Commit 07400f9 by Wojciech Tyczynski
Increase maxIdleConnection limit in etcd client.

I wonder whether that fixed the problem? In which case we can potentially roll back this roll back.

I think that commit can help a lot for any connection-related issue to etcd (in particular it reduced cpu-usage for performance tests in apiserver by ~35%). But didn't dig into this issue, so I have no idea whether it was connection or performance related.