iptables proxy could prefer local containers #24300
Comments
|
ping. @thockin you might be interested in commenting on this based on your previous work in this area. |
|
I swear I opened a bug on this but I can't find it. I'm retitling this for clarity What does "prefer" mean? Always use? 90%? Even in the face of overload? Once you go down this route, you need a more concrete heuristic than "prefer". To say "There is some concern" about imbalance is being polite. This is fundamentally the main reason we DON'T implement this today. At least GCE doesn't have a notion of number of connections when it comes to L3 load-balancing, at least not that is visible or configurable. I can't say for sure whether it is smart enough to do something behind the scenes. The other reason we have not implemented it is because it changes the syncronization model between Kubernetes and the external load-balancers. Today that is pretty static - program all nodes and let the fast-updating kube-proxy handle updating itself. This necessarily means that the client IP gets hidden as we bounce traffic around. This is bad, and we know it. If we always use local backends, we don't need to hide the client IP anymore - win! But now we can ONLY route traffic to nodes with backends on them. This is an ever-changing set, which means we need to be programming the cloud APIs much more frequently, and those APIs are not known for being "fast". We could, perhaps, do as you suggest with port checks (pretty clever actually :), but that requires some thought in the face of UDP services. All that said, I am not against doing this - it's just a LOT of work and it's sort of a tradeoff - having load-balancers be balanced vs preserving client IP. We'd have to run some experiments before fully committing. It's a good project. I'd like to work on it, but I have no time right now. Is someone willing to work on it? Step 0 is a design doc. |
thockin
added
sig/network
thockin
changed the title
|
Dup of #19754 |
|
That is a much more delicate question because it is very likely to make for On Wed, Sep 21, 2016 at 3:51 PM, Min Ren notifications@github.com wrote:
|
|
@thockin, I guess it is more about NodePort service type. In our case, we have to connect to an external load balancer even with NodePort service. And we can not just set one node ip-port to external load balancer. Then there are two load balancing running, and kube-proxy load balancing real just give the request more latency. |
|
Yes, NodePort support will be a followup to the work that went in for 1.4 On Wed, Sep 21, 2016 at 9:41 PM, Min Ren notifications@github.com wrote:
|
Currently when kube-proxy sets up iptables rules it round robins requests for a specific service to all containers on all minion nodes in the cluster matching the selector for said service. This has many benefits to resiliency for the services and improved latency as described in issue #3760 .
A further improvement to this design would be to prioritize local containers over remote containers so requests don't need to hit the wire and traverse the private underlay network most of the time. I think this would have a significant improvement on the scalability of services when running on clusters with a larger number of minions as well.
There is some concern that by preferring local containers that this would create issues w/ external load balancers in cases where there are not an even spread of containers across the cluster and that we would then need to set the priority or weight correctly on the external load balancers. I believe it's too much of a rabbit hole to try to tackle that aspect of this at this time because of the different ways that each vendor handles this with various load balancing algorithms, not to mention the fact that an overloaded node with a bunch of containers could end up with too many requests and cause a performance issue. Instead this can be dealt with by leveraging algorithms on the load balancer that would auto balance traffic based on the performance of each node. Nodes that have additional containers running locally for the service would complete requests more quickly and therefore even a simple least-connections algorithm would send more requests to that node. Likewise, if it was overloaded and completing requests more slowly then the load balancer would send it less traffic.
One additional optional optimization here would be making the node port for the service inactive if there are no containers for the service on that node. This would allow the load balancer to remove that node from the pool completely until a container does exist on it with a simple port check.
The text was updated successfully, but these errors were encountered: