I researched and experimented a bit concerning this topic and these are my thoughts/findings.

Setup

Local Kind cluster with Cilium installed via the following command, alternatively using IPSec and WireGuard, in Overlay and Native routing mode.

# install cilium
~ ./cli install --wait --chart-directory=./install/kubernetes/cilium \
                --helm-set=encryption.enabled=true \
                --helm-set=encryption.type=wireguard \
                --helm-set=debug.enabled=true \
                --helm-set=debug.verbose=datapath \
                --helm-set=bpf.monitorAggregation=none \
                --helm-set=routingMode=native \
                --helm-set=ipv4NativeRoutingCIDR=10.244.0.0/16 \
                --helm-set=ipv6NativeRoutingCIDR=fd00:10:244::/56 \
                --helm-set=autoDirectNodeRoutes=true

The Cilium nodes in the cluster are as follows:

# nodes
~ kubectl get ciliumnode
NAME                           CILIUMINTERNALIP   INTERNALIP   AGE
cilium-testing-control-plane   10.244.0.173       172.18.0.2   29m
cilium-testing-worker          10.244.2.62        172.18.0.4   29m
cilium-testing-worker2         10.244.1.187       172.18.0.3   29m\

And two client-server pods in the default namespace pinging each other are deployed in 2 different nodes.

# pods
~ kubectl get pod
NAME                               IP             NODE
netshoot-client-56565c9798-g65s2   10.244.2.157   cilium-testing-worker
netshoot-server-7bb56f5756-tr9f5   10.244.1.230   cilium-testing-worker2

IPSec Overlay

1:<- endpoint 138 flow 0x0 , identity 5052->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
2:-> stack flow 0x0 , identity 5052->29976 state new ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
3:-> 0: 10.244.2.157 -> 10.244.1.230 EchoRequest
4:-<- stack encrypted  flow 0x0 , identity 5052->unknown state new ifindex lxc8646aff47552 orig-ip 0.0.0.0: 10.244.2.62 -> 10.244.1.187
5:-> overlay encrypted flow 0x0 , identity 5052->unknown state new ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.2.62 -> 10.244.1.187
---------------------------------------------
6:-<- network flow 0xa08b144 , identity unknown->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 172.18.0.3:53422 -> 172.18.0.4:8472 udp
7:<- overlay encrypted flow 0xa08b144 , identity 29976->unknown state new ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.1.187 -> 10.244.2.62 
8:-> stack encrypted flow 0xa08b144 , identity 29976->unknown state new ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.1.187 -> 10.244.2.62
9:<- stack flow 0xa08b144 , identity unknown->unknown state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.1.230 -> 10.244.2.157 EchoReply
10:-> endpoint 138 flow 0xa08b144 , identity 29976->5052 state reply ifindex lxc8646aff47552 orig-ip 10.244.1.230: 10.244.1.230 -> 10.244.2.157 EchoReply

TRACE_FROM_LXC (cil_from_container)
TRACE_TO_STACK (cil_from_container, handle_ipv4_from_lxc tailcall)
TRACE_TO_STACK (cil_to_host)
TRACE_FROM_STACK (cil_from_host, do_netdev)
TRACE_TO_OVERLAY (cil_from_host, __encap_with_nodeid)
TRACE_FROM_NETWORK (cil_from_netdev, do_netdev)
TRACE_FROM_OVERLAY (cil_from_overlay)
TRACE_TO_STACK (cil_from_overlay, handle_ipv4)
TRACE_FROM_STACK (cil_from_overlay)
TRACE_TO_LXC (cil_from_overlay, tail_ipv4_ct_ingress_policy_only, ipv4_local_delivery)

WireGuard Overlay

1:<- endpoint 675 flow 0x0 , identity 28163->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
2:-> overlay flow 0x0 , identity 28163->36107 state new ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
3:--> network flow 0x39904c3f , identity unknown->unknown state new ifindex eth0 orig-ip 0.0.0.0: 172.18.0.4:51871 -> 172.18.0.3:51871 udp
---------------------------------------------
4:-<- network flow 0x39904c3f , identity unknown->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 172.18.0.3:51871 -> 172.18.0.4:51871 udp
5:<- overlay flow 0x0 , identity 36107->unknown state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.1.230 -> 10.244.2.157 EchoReply
6:-> endpoint 675 flow 0x0 , identity 36107->28163 state reply ifindex lxcb8251079a7de orig-ip 10.244.1.230: 10.244.1.230 -> 10.244.2.157 EchoReply

TRACE_FROM_LXC (cil_from_container)
TRACE_TO_OVERLAY (cil_from_host, do_netdev, CILIUM_CALL_IPV4_FROM_NETDEV)
TRACE_TO_NETWORK (cil_to_netdev)
TRACE_FROM_NETWORK (cil_from_netdev, do_netdev)
TRACE_FROM_OVERLAY (cil_from_overlay)
TRACE_TO_LXC (cil_from_overlay, tail_ipv4_ct_ingress_policy_only, ipv4_local_delivery)

IPSec Native

1:<- endpoint 1224 flow 0x0 , identity 8899->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
2:-> stack flow 0x0 , identity 8899->26398 state new ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
3:-> 0: 10.244.2.157 -> 10.244.1.230 EchoRequest
4-<- stack encrypted  flow 0x0 , identity 8899->unknown state new ifindex lxc199cd20a470a orig-ip 0.0.0.0: 10.244.2.62 -> 10.244.1.187
5--> stack flow 0x0 , identity unknown->unknown state unknown ifindex cilium_host orig-ip 0.0.0.0: 10.244.2.62 -> 10.244.1.187
---------------------------------------------
6-<- network encrypted  flow 0x0 , identity unknown->unknown state new ifindex eth0 orig-ip 0.0.0.0: 10.244.1.187 -> 10.244.2.62
7--> stack encrypted  flow 0x0 , identity unknown->unknown state new ifindex eth0 orig-ip 0.0.0.0: 10.244.1.187 -> 10.244.2.62
8:<- stack flow 0x0 , identity unknown->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 10.244.1.230 -> 10.244.2.157 EchoReply
9:-> host from flow 0x0 , identity unknown->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 10.244.1.230 -> 10.244.2.157 EchoReply
10:<- host flow 0x0 , identity world->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 10.244.1.230 -> 10.244.2.157 EchoReply
11:-> endpoint 1224 flow 0x0 , identity 26398->8899 state reply ifindex lxc199cd20a470a orig-ip 10.244.1.230: 10.244.1.230 -> 10.244.2.157 EchoReply

WireGuard Native

1:<- endpoint 2255 flow 0x0 , identity 15205->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
2:-> stack flow 0x0 , identity 15205->33119 state new ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
3:-> 0: 10.244.2.157 -> 10.244.1.230 EchoRequest
4--> network flow 0x0 , identity host->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 172.18.0.4:51871 -> 172.18.0.3:51871 udp
---------------------------------------------
5-<- network flow 0x0 , identity unknown->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 172.18.0.3:51871 -> 172.18.0.4:51871 udp
6:<- host flow 0x0 , identity world->unknown state unknown ifindex cilium_wg0 orig-ip 0.0.0.0: 10.244.1.230 -> 10.244.2.157 EchoReply
7:-> endpoint 2255 flow 0x0 , identity 33119->15205 state reply ifindex lxc6b5542b90ce7 orig-ip 10.244.1.230: 10.244.1.230 -> 10.244.2.157 EchoReply

Expected Behavior

~~As for the IPSec traffic, we would also like to have the Wireguard traffic marked with TRACE_REASON_ENCRYPTED when entering/exiting a node, such as:~~

135:<- endpoint 2622 flow 0x0 , identity 59875->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.2.169 -> 10.244.1.195 EchoRequest
136:-> overlay flow 0x0 , identity 59875->46095 state new ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.2.169 -> 10.244.1.195 EchoRequest
137--> network encrypted  flow 0x5aa208c2 , identity unknown->unknown state new ifindex eth0 orig-ip 0.0.0.0: 172.18.0.3:51871 -> 172.18.0.4:51871 udp
139-<- network encrypted  flow 0x5aa208c2 , identity unknown->unknown state new ifindex eth0 orig-ip 0.0.0.0: 172.18.0.4:51871 -> 172.18.0.3:51871 udp
140:<- overlay flow 0x0 , identity 46095->unknown state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.1.195 -> 10.244.2.169 EchoReply
141:-> endpoint 2622 flow 0x0 , identity 46095->59875 state reply ifindex lxc0f25ac0b5d80 orig-ip 10.244.1.195: 10.244.1.195 -> 10.244.2.169 EchoReply

Edit This is not really what we would like to have. While being true, it would be more useful to provide the encrypted boolean value in Hubble/Cilium-dbg concerning the endpoint communication rather than the outer network packet. Hence, the expected behavior should be as follows.

After many discussions, we believe that the expected solution should leverage the TRACE_REASON_ENCRYPT on the inner packet. The only downsize, is that when using overlay we can trace the packet once being vxlan encapsulated, but this doesn't provide context about endpoints. Adding a new trace point TRACE_{FROM,TO}_CRYPTO_DEV will also help distinguishing between plain and encrypted packets TO/FROM host. These new trace points will help logging packet immediately going to that interface for encryption or right after leaving the WireGuard device after decryption. In the case of native routing, the output should look similar as follows:

1:<- endpoint 2255 flow 0x0 , identity 15205->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
2:-> stack flow 0x0 , identity 15205->33119 state new ifindex 0 orig-ip 0.0.0.0: 10.244.2.157 -> 10.244.1.230 EchoRequest
3:-> 0: 10.244.2.157 -> 10.244.1.230 EchoRequest
NEW:-> crypto_dev encrypted ... cilium_wg0
4--> network flow 0x0 , identity host->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 172.18.0.4:51871 -> 172.18.0.3:51871 udp
---------------------------------------------
5-<- network flow 0x0 , identity unknown->unknown state unknown ifindex eth0 orig-ip 0.0.0.0: 172.18.0.3:51871 -> 172.18.0.4:51871 udp
6:<- crypto_dev flow 0x0 , identity world->unknown state unknown ifindex cilium_wg0 orig-ip 0.0.0.0: 10.244.1.230 -> 10.244.2.157 EchoReply
7:-> endpoint 2255 flow 0x0 , identity 33119->15205 state reply ifindex lxc6b5542b90ce7 orig-ip 10.244.1.230: 10.244.1.230 -> 10.244.2.157 EchoReply

The NEW in the egress path and line 6 in ingress path should provide a minimum context for a user to understand that the packet (either ingress/egress) has passed through the crypto device. The cilium bpf metrics should be updated accordingly. In the native routing scenario this should be enough, but the overlay might need additional parsing logic, since the traced packet would be already vxlan encap-ed. However, this behavior is still coherent with IPSec.

How to tackle

A reasonable idea to recognize and distinguish Cilium-related Wireguard traffic would be to add a parsing logic to the programs attached to the network device (cil_to_netdev and cil_from_netdev). However, two asymmetric ways to distinguish the traffic are possible:

Egress (cil_to_netdev): here a bit of logic checking whether an sbk has MARK_MAGIC_WG_ENCRYPTED is already present. This means that the packet has already left the wireguard tunnel, thus is encrypted. In that case, we can easily change trace.reason = TRACE_REASON_ENCRYPT.
Ingress (cil_from_netdev): here IPSec matches and mark ESP packets with a valid NodeID. For WireGuard, we can insert a similar logic, matching and marking UDP packets with sport=dport=CILIUM_WG_PORT (51871), where WireGuard is listening.

Edit This solution would allow us to track the WireGuard encapsulated packet as encrypted, which is correct, but it wouldn't significantly benefit Hubble/Cilium-monitor. Adding parsing logic (L4 header ports) just for tracing could be an overkill, since we can achieve the same result in userspace. Then, to begin, the approach described in the related PR is followed #34958.

Doubts

Which mark should be used? Right now I would insert a new #define MARK_MAGIC_WG_DECRYPT 0x2D00 that overlaps with k8s space, but should be safe to use. However, my ideal solution would be to remove both MARK_MAGIC_WG_DECRYPT and MARK_MAGIC_WG_ENCRYPTED in favor of MARK_MAGIC_ENCRYPT and MARK_MAGIC_DECRYPT, which are currently used by IPSec only. By doing that, (a) we don't have to overlap with the k8s mark space; (b) have only two marks, which we know how to treat when using IPSec/Wireguard accordingly (IPSec right now needs to fwd to stack for encryption, in the WireGuard case the packet would already be encrypted); (c) and in future adoption of XFRM devices IPSec would have the same behavior as WireGuard, and would be nice to use same marks.
The encrypt_agg.cocci has been introduced to make sure that no aggregation is performed when tracing encrypted packet (IPSec). However, when applying the patch hence tracing Ingress Wireguard as encrypted, the cil_from_netdev program uses the TRACE_PAYLOAD_LEN macro.

Edit PR-related, moved in #34958.

Follow-up

Edit #28001 proposes to enable vxlan header parsing in Hubble (anche we can extend it also for cilium-dbg). This could help providing more context in the proposed solution for this issue regarding which pod-to-pod communications are encrypted in overlay mode (so far the context is lost after encap, we could then inspect the encap-ed packet but doing that in datapath just for tracing might not be the best).

wireguard: use TRACE_REASON_ENCRYPTED #34659

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions