Round robin load balancer can overwhelm first pod behind service #3640
Description
This is particularly easy to do with a new cluster.
Create a service S
and deploy N
pods behind it. Start up M
pods which create a persistent connection to service S
.
Because the round robin load balancer always starts at the 0th endpoint, and the endpoint list is consistent cluster-wide, you will see that you now have M
connections to Pod 1, and 0 connections to all other pods 2 through N.
Perhaps the the round robin load balancer should start off at a random index [0,N) when the list of endpoints changes.
(Concretely: we had 3 instances of twemproxy behind a "redis" service, fired up a bunch of worker pods (which happen to connect once, do all of their work, then exit), and each worker pod connected to the same instance of twemproxy, with the other 2 remaining completely idle.)