Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incompatibility with Linux 6.11.{4,5} and 6.6.{57,58} due to ip6tables error #13863

Closed
philiptaron opened this issue Oct 19, 2024 · 59 comments
Closed
Labels
bug Bug

Comments

@philiptaron
Copy link

philiptaron commented Oct 19, 2024

What is the issue?

I updated to Linux 6.11.4 (on NixOS) and now tailscale status reports the following:

# Health check:
#     - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/nix/store/zpl4wlvc9a4ziq7b6ccrpxzn5mwc3frn-iptables-1.8.10/bin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).

Try `ip6tables -h' or 'ip6tables --help' for more information.

As a result, many things don't work correctly, in particular MagicDNS.

Steps to reproduce

  1. Make a NixOS system with kernel 6.11.4 (boot.kernelPackages = pkgs.linuxPackages_latest at time of issue filing). I assume other distros with 6.11.4 would also exhibit the issue, but I haven't checked myself.
  2. Install and configure tailscale (services.tailscale.enable = true, tailscale up etc)
  3. Run tailscale status

Observed:

# Health check:
#     - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/nix/store/zpl4wlvc9a4ziq7b6ccrpxzn5mwc3frn-iptables-1.8.10/bin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).

Try `ip6tables -h' or 'ip6tables --help' for more information.

Expected: No health check warnings.

This was the case with 6.11.3.

Are there any recent changes that introduced the issue?

The changelog for kernel 6.11.4 lists the following commits to the netfilter code.

$ git log v6.11.3..v6.11.4 --oneline | grep netfilter
339dc6c7266c netfilter: fib: check correct rtable in vrf setups
4cdc55ec6222 netfilter: xtables: avoid NFPROTO_UNSPEC where needed
915717e0bb98 netfilter: br_netfilter: fix panic with metadata_dst skb
9f5c115077d3 netfilter: nf_reject: Fix build warning when CONFIG_BRIDGE_NETFILTER=n
a2c6c487ed9c netfilter: nf_nat: don't try nat source port reallocation for reverse dir clash

Of these revisions, netfilter: xtables: avoid NFPROTO_UNSPEC where needed has been confirmed to be the source of this issue.

OS

Linux

OS version

NixOS at revision 0fe3416c7f455a1e7ca6dec3c0b2d1e2cd30d4f6

Tailscale version

1.76.1

Other software

My full NixOS configuration is at https://github.com/philiptaron/flock.nix. There's nothing fancy there, though; it's mostly a stock NixOS configuration that has run Tailscale reliably until this kernel upgrade.

Bug report

BUG-a941a9d4060b13bb02ff199d2b96deac67ccf9b681a2dfc2f5ee5576fb062a46-20241019151759Z-9ca281bdd508a563

@misuzu
Copy link

misuzu commented Oct 19, 2024

6.6.57 seems affected too

@raggi
Copy link
Member

raggi commented Oct 19, 2024

I guess we better move, even if this gets reverted just to avoid the trap. I believe some users have a light dependency on the number, I'd want to dig up some of those related bugs to refresh on the details - pretty sure they were here on the public tracker

@CoolnsX
Copy link

CoolnsX commented Oct 20, 2024

Happening the same for me in Arch Linux on same kernel.
This made the route advertising not working properly

@christian-heusel
Copy link

This is caused by 0bfcb7b71e73 ("netfilter: xtables: avoid NFPROTO_UNSPEC where needed") and a fix is already in the works: https://lore.kernel.org/all/20241019-xtables-typos-v2-1-6b8b1735dc8e@0upti.me/

For now downgrading the kernel or patching it with the above should fix the issue, although I'd expect the issue to be fixed with the next stable kernel 😊

@philiptaron
Copy link
Author

Thanks for the pointer, Christian (@christian-heusel). I applied the latest version of that patch and I can also confirm that it fixes this issue.

@mweinelt
Copy link

@crepererum
Copy link

For Arch Linux users: rolling back from kernel 6.11.4 to 6.11.3 fixes the issue, e.g. via:

$ sudo pacman -U /var/cache/pacman/pkg/linux-6.11.3.arch1-1-x86_64.pkg.tar.zst /var/cache/pacman/pkg/linux-headers-6.11.3.arch1-1-x86_64.pkg.tar.zst

@Moelf
Copy link

Moelf commented Oct 21, 2024

on arch this is fixed with 6.11.4-arch2-1

@hummeltech
Copy link

This is also happening with linux-lts-6.6.57-1-x86_64.pkg.tar.zst :(

@CoolnsX
Copy link

CoolnsX commented Oct 22, 2024

on arch this is fixed with 6.11.4-arch2-1

yeah, was gonna reply myself, but u r fast

@philiptaron
Copy link
Author

philiptaron commented Oct 22, 2024

Linux 6.11.5 and Linux 6.6.58, released today, do not contain the needed fix.

On NixOS, @K900 has cherry-picked the fix into the NixOS 6.11.5 -- see the following PR:

@CoolnsX
Copy link

CoolnsX commented Oct 23, 2024

Linux 6.11.5 and Linux 6.6.58, released today, do not contain the needed fix.

thanks, I was going to update it.

@BeiNacht
Copy link

Thanks for the pointer, Christian (@christian-heusel). I applied the latest version of that patch and I can also confirm that it fixes this issue.

Works with 6.12.x Kernel aswell

@VenomousSteam81
Copy link

I'm also experiencing this issue on Void linux with lts kernel 6.6.57_1.

@crazyuploader
Copy link

Experienced on Fedora as well, reverted and this works now -

Linux Fedora-DellG15 6.11.4-201.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Oct 10 22:31:19 UTC 2024 x86_64 GNU/Linux

@denisfalqueto
Copy link

I'm on NixOS and I had to downgrade to an 6.10.x kernel (pkgs.linuxPackages_6_10).

@philiptaron philiptaron changed the title Incompatibility with Linux 6.11.4 due to new ip6tables error Incompatibility with Linux 6.11.{4,5} and 6.6.{57,58} due to ip6tables error Oct 24, 2024
@hypoactiv
Copy link

Tracked in Fedora Bugzilla at https://bugzilla.redhat.com/show_bug.cgi?id=2321325

@ng8weji9vowdv
Copy link

ng8weji9vowdv commented Oct 26, 2024

apt build-dep linux linux-image-unsigned-$(uname -r)
apt-get install build-essential fakeroot linux-source bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev dwarves bison
apt-get install devscripts
apt-get build-dep linux
apt-get source linux
chmod 777 linux_6.11.4-1.dsc
chmod 777 linux_6.11.4.orig.tar.xz
chmod 777 linux_6.11.4-1.debian.tar.xz
rm -rf linux-6.11.4/
apt-get source linux'
cd linux-6.11.4/

apply patches from torvalds/linux@306ed17

cp /lib/modules/6.11.4-amd64/build/.config .
cp /lib/modules/6.11.4-amd64/build/Module.symvers .
make kernelversion scripts prepare modules_prepare
make modules M=net/netfilter -j22

after that u need simple manipulations with files (dont ask idk how debian distributes in compressed mode and couldn't really figure out the wiki is lacking info on what they do and i cba figuring out) such as copying in /usr/lib/modules/6.11.4-amd64/kernel/net/netfilter/ and renaming .ko.xz into .ko.xz.backup for example, then running depmod -a, modprobe -r xt_mark and modprobe xt_mark (if any process is using it you have to terminate it, or, stop the service), downside obviously annoyment with /etc/modules (which most likely wont load your module (lack of signs and it doesnt expect the module to be in /net/netfilter but in root of kernel folder seems like) etc.) so after reboot the dependent services might not initialize in a correct way, so you have to restart.

tl;dr; just wait until they push the fix into upstream

someone can shorten the apt sides, also, in case if you somehow deleted a module from the /usr/lib/modules/.../net/netfilter/ and apt install --reinstall linux-image-6.11.4-amd64 does not help to restore files, try apt reinstall linux-image-6.11.4-amd64 instead
xt_mark.ko.zip

@Funtimes909
Copy link

Happening on Arch Linux 6.11.4, taildrop between devices not working

@marek22k
Copy link

marek22k commented Oct 26, 2024

Same on Arch Linux 6.6.58-1-lts:

# Health check:
#     - Tailscale can't reach the configured DNS servers. Internet connectivity may be affected.
#     - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/usr/bin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).

Try `ip6tables -h' or 'ip6tables --help' for more information.

@lunarcontrol
Copy link

Experiencing this on a fully-updated Fedora Workstation 40 machine

# Health check:
#     - Tailscale can't reach the configured DNS servers. Internet connectivity may be affected.
#     - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/usr/sbin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).

Try `ip6tables -h' or 'ip6tables --help' for more information.
Operating System: Fedora Linux 40 (Workstation Edition)
Kernel: Linux 6.11.4-201.fc40.x86_64

@CoolnsX
Copy link

CoolnsX commented Nov 3, 2024

Can confirm, it works on arch with kernel 6.11.6-arch1-1

@byronogis
Copy link

working on nixos, 6.6.59

 $ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.59, NixOS, 24.11 (Vicuna), 24.11.20241102.7ffd9ae`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.9`
 - nixpkgs: `/nix/store/zq2axpgzd5kykk1v446rkffj3bxa2m2h-source`

@basncy
Copy link

basncy commented Nov 4, 2024

Archlinux works on 6.6.59-1-lts and 6.11.6-arch1-1

@philiptaron
Copy link
Author

As the issue is fixed upstream and released in Linux 6.11.6 and 6.6.59, I'll close this.

@Atomique
Copy link

Atomique commented Nov 7, 2024

Thanks a lot. In F40 my upgrade to kernel 6.11.6-200.fc40.x86_64 fixed this also!

@maxfridbe
Copy link

Thanks a lot! F41 and F40 Silverblue all fixed now!

@travisghansen
Copy link

Anyone else getting this on arch again with 6.11.8-arch1-2?

@mt190502
Copy link

Anyone else getting this on arch again with 6.11.8-arch1-2?

image

@masterwishx
Copy link

Have same on ubuntu in oracle cloud : 20.04.6 LTS (Focal Fossa)

After this update i think :

linux-headers-5.15.0-1071-oracle linux-image-5.15.0-1071-oracle,
linux-modules-5.15.0-1071-oracle linux-modules-extra-5.15.0-1071-oracle,
linux-oracle-5.15-headers-5.15.0-1071,
curl libcurl3-gnutls libcurl4 linux-headers-oracle linux-image-oracle,
linux-libc-dev linux-oracle

@travisghansen
Copy link

If it helps anyone mine was a case where the kernel had been updated but the system not rebooted. So literally the kernel modules for the running kernel were no longer on the fs.

@Atomique
Copy link

If it helps anyone mine was a case where the kernel had been updated but the system not rebooted. So literally the kernel modules for the running kernel were no longer on the fs.

You can check this with needrestart if needed. https://manpages.ubuntu.com/manpages/focal/man1/needrestart.1.html

@travisghansen
Copy link

I'm on arch, but that may be helpful for others for sure!

@masterwishx
Copy link

If it helps anyone mine was a case where the kernel had been updated but the system not rebooted. So literally the kernel modules for the running kernel were no longer on the fs.

You can check this with needrestart if needed. https://manpages.ubuntu.com/manpages/focal/man1/needrestart.1.html

For me this issue was seems after restart...

@Kemichal
Copy link

It looks like this bug has hit my Ubuntu 22.04 servers today.
Kernel 5.15.0-128-generic

@masterwishx
Copy link

It looks like this bug has hit my Ubuntu 22.04 servers today. Kernel 5.15.0-128-generic

no issues on Ubuntu 22.04 6.8.0-1018-oracle but still have issue on Ubuntu 20.04 5.15.0-1072-oracle

@christian-heusel
Copy link

no issues on Ubuntu 22.04 6.8.0-1018-oracle but still have issue on Ubuntu 20.04 5.15.0-1072-oracle

Yeah please get in touch with your kernel vendors (ubuntu or oracle) instead of bumping this issue

@steled
Copy link

steled commented Dec 20, 2024

For Ubuntu a new kernel version is released which fixed the problem: 5.15.0-130.140

@iluvatyr
Copy link

For Ubuntu a new kernel version is released which fixed the problem: 5.15.0-130.140

yes, it's fixed for me aswell!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug
Projects
None yet
Development

No branches or pull requests