OpenThread broken on newer Linux kernel versions #8523

agners · 2022-12-13T22:32:08Z

Describe the bug

Trying to start the otbr-agent on Linux 6.0, 6.1 (and possibly 5.19, maybe older kernels) seems not to work correctly.

Two things are observed:

Pressing Ctrl+C does not succeed (otbr-agent freezes hard, even a SIGKILL does not kill the process, according to ps the process is in state D, uninterruptible sleep).
Trying to form a network via Web interface fails as well. It seems that the factoryreset command tries to shutdown the interface, which causes the hang similar to 1.

In both cases mctp_unregister: BUG mctp_ptr set for unknown type 65535 errors start to appear.

To Reproduce Information to reproduce the behavior, including:

Git commit id e66c21c
IEEE 802.15.4 hardware platform: nRF52840 Dongle
Build steps: manually via ./script/cmake-build as well as container builds
Network topology

Expected behavior

Ctrl+C should be working.

Forming a network should succeed.

Console/log output

Dez 13 22:50:00 allenwind cc5da69851dd[5196]: Dec 13 21:49:52 cc5da69851dd otbr-agent[181]: [INFO]-BA------: Result of publish meshcop service OpenThread BorderRouter #177A._meshcop._udp.local: OK
Dez 13 22:50:00 allenwind cc5da69851dd[5196]: Dec 13 21:50:00 cc5da69851dd otbr-agent[181]: 00:00:09.220 [I] Platform------: Session socket is ready
Dez 13 22:50:00 allenwind cc5da69851dd[5196]: Dec 13 21:50:00 cc5da69851dd otbr-agent[181]: 00:00:09.220 [I] Platform------: > #012state
Dez 13 22:50:00 allenwind cc5da69851dd[5196]: Dec 13 21:50:00 cc5da69851dd otbr-agent[181]: 00:00:09.220 [D] Cli-----------: Input: state
Dez 13 22:50:00 allenwind cc5da69851dd[5196]: Dec 13 21:50:00 cc5da69851dd otbr-agent[181]: 00:00:09.220 [D] Cli-----------: Output: disabled
Dez 13 22:50:00 allenwind cc5da69851dd[5196]: Dec 13 21:50:00 cc5da69851dd otbr-agent[181]: 00:00:09.220 [D] Cli-----------: Output: Done
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: 00:00:12.160 [I] Platform------: Session socket is ready
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: 00:00:12.185 [I] Platform------: > #012factoryreset
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: 00:00:12.185 [D] Cli-----------: Input: factoryreset
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: 00:00:12.185 [I] Settings------: Wiped all info
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: [INFO]-BA------: Stop Thread Border Agent
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: [INFO]-ADPROXY-: Stopped
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: [INFO]-DPROXY--: Stopped
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: [INFO]-BA------: Unpublish meshcop service OpenThread BorderRouter #177A._meshcop._udp.local
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: [INFO]-MDNS----: Removing service OpenThread BorderRouter #177A._meshcop._udp
Dez 13 22:50:03 allenwind cc5da69851dd[5196]: Dec 13 21:50:03 cc5da69851dd otbr-agent[181]: [INFO]-BA------: Result of unpublish meshcop service OpenThread BorderRouter #177A._meshcop._udp.local: OK
Dez 13 22:50:04 allenwind kernel: wpan0 (unregistering): mctp_unregister: BUG mctp_ptr set for unknown type 65535
Dez 13 22:50:04 allenwind cc5da69851dd[5196]: Dec 13 21:50:04 cc5da69851dd kernel: [  307.224069] wpan0 (unregistering): mctp_unregister: BUG mctp_ptr set for unknown type 65535
Dez 13 22:50:05 allenwind kernel: wpan0 (unregistered): mctp_unregister: BUG mctp_ptr set for unknown type 65535

Additional context

Possibly related #8400.

The text was updated successfully, but these errors were encountered:

agners · 2022-12-13T22:39:18Z

Diving into it it seems that it is somehow related to mctp_unregister not being able to properly unregister, which causes the user space process to hang:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/mctp/device.c?h=v6.1#n434

Digging further, I looked into recent changes of that file. I then came across this change:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/net/mctp/device.c?h=v6.1&id=f364dd71d92fe6722fe5d47803be974dc0c40762

This seems to be a "larger than advertised" change. The unknown type above 65535 equals 0xFFFF which matches ARPHRD_VOID define. That seems no longer to be a valid type. It is however the type OpenThread is using:

openthread/src/posix/platform/netif.cpp

Line 1639 in 2db38b0

VerifyOrDie(ioctl(sTunFd, TUNSETLINK, ARPHRD_VOID) == 0, OT_EXIT_ERROR_ERRNO);

I guess this is strictly speaking a Linux kernel regression. However, maybe OpenThread should be using ARPHRD_NONE none here?

agners · 2022-12-13T22:45:48Z

Replacing ARPHRD_VOID with ARPHRD_NONE makes the OTBR work on Linux 6.1.

I probably don't understand the problem fully yet, but it seems to me that the type change via TUNSETLINK is causing issue for the mctp driver.

Linux kernels with CONFIG_MCTP enable (available in 5.16 or newer) seem to have an issue whne setting the link type to ARPHRD_VOID. There seem to be no good reason why ARPHRD_VOID is used. Stick to the default ARPHRD_NONE. Fixes: openthread#8523

The check could be incorrectly triggered if a netdev changes its type after register. That is possible for a tun device using TUNSETLINK ioctl, resulting in mctp unregister failing and the netdev unregister waiting forever. This was encountered by openthread/openthread#8523 The check is not required, it was added in an attempt to track down mctp_ptr being set unexpectedly, which should not happen in normal operation. Fixes: 7b1871a ("mctp: Warn if pointer is set for a wrong dev type") Signed-off-by: Matt Johnston <matt@codeconstruct.com.au>

mkj · 2022-12-14T06:28:09Z

The kernel MCTP code was assuming that the link type of a network device didn't change after creation - a wrong assumption for tun devices. Using ARPHDR_NONE seems a good workaround to avoid the link type changing.

I've proposed a fix for it https://lore.kernel.org/netdev/20221214061044.892446-1-matt@codeconstruct.com.au/T/#u that should get backported to stable kernels.

The unregister check could be incorrectly triggered if a netdev changes its type after register. That is possible for a tun device using TUNSETLINK ioctl, resulting in mctp unregister failing and the netdev unregister waiting forever. This was encountered by openthread/openthread#8523 Neither check at register or unregister is required. They were added in an attempt to track down mctp_ptr being set unexpectedly, which should not happen in normal operation. Fixes: 7b1871a ("mctp: Warn if pointer is set for a wrong dev type") Signed-off-by: Matt Johnston <matt@codeconstruct.com.au>

The unregister check could be incorrectly triggered if a netdev changes its type after register. That is possible for a tun device using TUNSETLINK ioctl, resulting in mctp unregister failing and the netdev unregister waiting forever. This was encountered by openthread/openthread#8523 Neither check at register or unregister is required. They were added in an attempt to track down mctp_ptr being set unexpectedly, which should not happen in normal operation. Fixes: 7b1871a ("mctp: Warn if pointer is set for a wrong dev type") Signed-off-by: Matt Johnston <matt@codeconstruct.com.au> Link: https://lore.kernel.org/r/20221215054933.2403401-1-matt@codeconstruct.com.au Signed-off-by: Jakub Kicinski <kuba@kernel.org>

[ Upstream commit b389a90 ] The unregister check could be incorrectly triggered if a netdev changes its type after register. That is possible for a tun device using TUNSETLINK ioctl, resulting in mctp unregister failing and the netdev unregister waiting forever. This was encountered by openthread/openthread#8523 Neither check at register or unregister is required. They were added in an attempt to track down mctp_ptr being set unexpectedly, which should not happen in normal operation. Fixes: 7b1871a ("mctp: Warn if pointer is set for a wrong dev type") Signed-off-by: Matt Johnston <matt@codeconstruct.com.au> Link: https://lore.kernel.org/r/20221215054933.2403401-1-matt@codeconstruct.com.au Signed-off-by: Jakub Kicinski <kuba@kernel.org> Signed-off-by: Sasha Levin <sashal@kernel.org>

agners · 2023-02-14T08:37:06Z

Since this is actually a kernel bug and fixed in 6.1.2 and newer I'd consider this closed. It is probably still worth not changing the link type if not really required and stick with ARPHDR_NONE.

The unregister check could be incorrectly triggered if a netdev changes its type after register. That is possible for a tun device using TUNSETLINK ioctl, resulting in mctp unregister failing and the netdev unregister waiting forever. This was encountered by openthread/openthread#8523 Neither check at register or unregister is required. They were added in an attempt to track down mctp_ptr being set unexpectedly, which should not happen in normal operation. Fixes: 7b1871a ("mctp: Warn if pointer is set for a wrong dev type") Signed-off-by: Matt Johnston <matt@codeconstruct.com.au> Link: https://lore.kernel.org/r/20221215054933.2403401-1-matt@codeconstruct.com.au Signed-off-by: Jakub Kicinski <kuba@kernel.org> (cherry picked from commit b389a90)

agners mentioned this issue Dec 13, 2022

[posix] set link type to none explicitly #8526

Merged

jk-ozlabs mentioned this issue Dec 14, 2022

Ubuntu 22.10 fails to create wpan interface #8400

Closed

agners closed this as completed Feb 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenThread broken on newer Linux kernel versions #8523

OpenThread broken on newer Linux kernel versions #8523

agners commented Dec 13, 2022

agners commented Dec 13, 2022

agners commented Dec 13, 2022

mkj commented Dec 14, 2022

agners commented Feb 14, 2023

OpenThread broken on newer Linux kernel versions #8523

OpenThread broken on newer Linux kernel versions #8523

Comments

agners commented Dec 13, 2022

agners commented Dec 13, 2022

agners commented Dec 13, 2022

mkj commented Dec 14, 2022

agners commented Feb 14, 2023