/cc @npinaeva @danwinship

nice - now compare with eBPF?

nice - now compare with eBPF?

It is not really about ebpf or nftables, is about the kernel path and how to avoid the "slow paths", I expect an eBPF implementation that shortcut the kernel to have the same performance. Both netfilter/flowtables and eBPF (at least in cilium) hook in the tc subsystem and redirect the traffic to the output interface directly, shortcutting the kernel, AFAIK they do the same. The problem with eBPF, for doing just Services, you need to maintain a considerable amount of code, duplicate existing kernel features, consume more resources, ... this is barely 20 lines of code

To recap, we had different slow paths in kube-proxy

1. Data plane programming, sending the whole blob of rules was slow, @danwinship implemented the partial syncs and I think there were some other improvements on the userspace @npinaeva helped to discover, IIRC we are now in a good state in this area ...

2. the Service match was a linear search, @danwinship fixed that using a map in nftables so we should be in O(1) vs O(n)

    map service-ips {
                type ipv4_addr . inet_proto . inet_service : verdict
                comment "ClusterIP, ExternalIP and LoadBalancer IP traffic"
                elements = { 10.96.0.10 . tcp . 53 : goto service-NWBZK7IH-kube-system/kube-dns/tcp/dns-tcp,
                             10.96.0.10 . udp . 53 : goto service-FY5PMXPG-kube-system/kube-dns/udp/dns,
                             10.96.117.116 . tcp . 80 : goto service-6THF4ISO-default/service/tcp/http,
                             10.96.221.118 . tcp . 80 : goto service-WRR5O5HH-default/test3/tcp/,
                             10.96.210.227 . tcp . 90 : goto service-6CKWZ3LT-default/test/tcp/,
                             10.96.117.116 . tcp . 5001 : goto service-LYULNVLC-default/service/tcp/iperf,
                             10.96.0.1 . tcp . 443 : goto service-2QRHZV4L-default/kubernetes/tcp/https,
                             10.96.0.10 . tcp . 9153 : goto service-AS2KJYAD-kube-system/kube-dns/tcp/metrics }
        }

3. The "performance" as throughput is fixed with this PR, is not likely you can go faster, bits need to travel from one place to another :)

Both netfilter/flowtables and eBPF (at least in cilium) hook in the tc subsystem and redirect the traffic to the output interface directly, shortcutting the kernel, AFAIK they do the same.

Much of Cilium's code is based on the tc hooks, but the standard eBPF shortcutting trick involves directly splicing two sockets together. In particular, that means the packet bypasses the network stack entirely, whereas the approach here only bypasses the network stack in the host netns. (The packets still traverse the network stack normally inside both containers.)

OTOH, the socket splicing trick, by definition, only works for traffic within a single node. Doing shortcutting for node-ingress/node-egress traffic requires a separate trick (which should be more-or-less equivalent to this one).

For completeness on what Dan said, the way of achieving performance in the datapath is just avoiding overhead processing the packets in the OS.

The most interesting case in network performance I've seen so far is on AI/ML workloads, those that don't even use TCP/IP because is "slow", and they use RDMA to bypass the OS and CPU, transferring the data between application memories.

But coming back to the TCP/IP world, AFAIK there are two common techniques to shortcut the path through the OS network stack by bypassing the kernel (adding some links there

• socket to socket with eBPF:
  • https://www.usenix.org/conference/srecon23emea/presentation/sitnicki
  • https://eunomia.dev/tutorials/29-sockops/
• interface to interface:
  • netfilter https://docs.kernel.org/networking/nf_flowtable.html
  • bpf redirect https://arthurchiao.art/blog/differentiate-bpf-redirects/

@aojea nice! I was actually just prototyping this in Calico last week as well, coincidentally.

Changes look rather similar, although I've scoped the Calico offload rule to include non-service traffic as well so long as it matches a Calico owned interface. My understanding is that I should just be able to adjust the Calico table's priority to be slightly after kube-proxy's, which would ensure that kube-proxy flow offload is checked first for Service traffic and Calico can still offload other non-service traffic.

The problem with eBPF, for doing just Services, you need to maintain a considerable amount of code, duplicate existing kernel features, consume more resources, ... this is barely 20 lines of code

🥳

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

(Related: the nftables flowtable rule does not depend on you using the nftables kube-proxy backend. It would work with iptables or ipvs too.)

you need nftables to program the dataplane AFAIK, so adding the nftables interface to the iptables proxy sounds a bit :/

It doesn't make sense to add it to the iptables proxy.Provider, but my point is that it doesn't make sense to add it to the nftables proxy.Provider either. The point of proxy.Provider is to watch for services and endpoints changing, and update the backend rules for those services and endpoints. The flowtable code has no dependencies on services or endpoints (or node topology labels). The nftables rules it adds do not interact with any of the other rules that the nftables proxier adds. The flowtable needs to be updated on an entirely separate schedule from when the service/endpoint rules need to be updated. Architecturally, it would make much more sense as a separate controller within kube-proxy (or outside of kube-proxy) that maintains its own table with its own rules and flowtable.

The flowtable code has no dependencies on services or endpoints (or node topology labels). The nftables rules it adds do not interact with any of the other rules that the nftables proxier adds

Let's take the angle of "Kubernetes Service acceleration", this feature is just "accelerating the Kubernetes Services traffic" and for that it need access to the ClusterIPs Set to avoid to interfere with any other network components on the nodes.

The kernel infrastructure only expose this functionality by using interface names, but I can see how it could have been exposed via connections and the neftables/netfilter/kenerl do the heavy lifting of identifying the network interfaces and adding the corresponding flows to the tables, as the skbuff IIRC will have both interfaces associated ... if this will have been exposed this way we'll have the same functionality but without different requirements ...

Anyway, I think is worth discussing, let's talk more in Kubecon and try to get more feedback ...

/triage accepted

This present some real interesting wins

Oh, so, not exactly the same thing, but our perf team here had done some testing with OVS offload to Mellanox NICs, and decided that it only makes sense for services with long-lived connections. If your service has lots of short-lived connections, then you end up spending more time configuring offload than you save by having offloaded the connection. (In the limiting case, if the connection closes immediately after you set up the offload, then you just completely wasted your time.) It's possible that this tradeoff applies more to hardware offload, where there are additional setup/cleanup steps required beyond what's required for the purely software offload case, but anyway, this feature could potentially benefit from having a hint on the Service that it has long-lived connections (and then it makes more sense at the kube-proxy level too...)

@aojea: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

PR needs rebase.