Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Elevated packet latency macOS → Linux, only on same LAN(?) #10356

Open
sartak opened this issue Nov 21, 2023 · 4 comments
Open

Elevated packet latency macOS → Linux, only on same LAN(?) #10356

sartak opened this issue Nov 21, 2023 · 4 comments
Labels
L1 Very few Likelihood needs-triage needs-triage-eng Ready for triage by Engineering team OS-macos Issues involving Tailscale for macOS P2 Aggravating Priority level T3 Performance/Debugging Issue type

Comments

@sartak
Copy link

sartak commented Nov 21, 2023 •
edited
Loading

What is the issue?

When connecting on a local network from macOS to Linux, I experience elevated packet loss, up to 15%, both for ping and ssh. Latency is not impaired--given the RTT, the packets are obviously not leaving my LAN.

The problem is specific to the Mac. When I have two Linux boxes connecting to each other, there is no packet loss. Similarly it's specific to Tailscale - while experiencing packet loss via Tailscale, concurrent connections to the local IPs are unaffected.

Steps to reproduce

I ran ping side by side in many terminals at the same time, to both the "true" local IP address and to the Tailscale interface.

From Mac to Linux:

--- 192.168.0.2 ping statistics ---
576 packets transmitted, 576 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.561/3.593/12.269/1.604 ms

--- mana-tailscale ping statistics ---
562 packets transmitted, 498 packets received, 11.4% packet loss
round-trip min/avg/max/stddev = 2.778/11.321/1149.472/61.257 ms

From Linux to Mac

--- 192.168.0.15 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 99143ms
rtt min/avg/max/mdev = 1.674/22.680/114.468/34.407 ms

--- hydra-tailscale ping statistics ---
100 packets transmitted, 94 received, 6% packet loss, time 99275ms
rtt min/avg/max/mdev = 3.353/8.414/101.229/16.486 ms

From a different Linux box to Linux

--- 192.168.0.2 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 846ms
rtt min/avg/max/mdev = 0.177/0.211/0.258/0.019 ms

--- mana-tailscale ping statistics ---
125 packets transmitted, 125 received, 0% packet loss, time 411ms
rtt min/avg/max/mdev = 0.714/2.038/108.693/9.581 ms

Are there any recent changes that introduced the issue?

This has been happening for a few weeks now. Took me a while to track it down to Tailscale specifically (I first suspected degraded network hardware, but the non-Tailscale pings rule out that explanation. That's very much a compliment to the quality of your software that it was so deep into the list of hypotheses 😄)

OS

No response

OS version

MacOS Sonoma 14.1.1, Raspbian buster, Ubuntu focal

Tailscale version

1.54.0 on all

Other software

This is over a wifi connection, though intuitively it feels like that shouldn't much matter given the concurrent connections are okay.

Bug report

BUG-b7bfb2cca2685b00ef26ca6f8ce36066dbdd6f96d71d6de050d4e9ac35058039-20231121224148Z-b9ba52d3cfe6e9f6
@sartak
Copy link
Author

sartak commented Dec 27, 2023

Here's another bugreport:

BUG-b7bfb2cca2685b00ef26ca6f8ce36066dbdd6f96d71d6de050d4e9ac35058039-20231227191124Z-83817bc46af2b155

It seems to happen for a while (a few hours?) after switching wifi networks. Even restarting tailscale on both my mac and my linux machine doesn't help.

@knyar knyar added OS-macos Issues involving Tailscale for macOS L1 Very few Likelihood P2 Aggravating Priority level T3 Performance/Debugging Issue type needs-triage-eng Ready for triage by Engineering team and removed bug Bug labels Dec 29, 2023
@sartak
Copy link
Author

sartak commented Jun 27, 2024 •
edited
Loading

Still happening for me. This is a Mac (Sonoma 14.5) pinging a Linux box (ubuntu jammy, linux 5.15.0-91-generic), both on the latest Tailscale, 1.68.1.

The lost packet is very noticeable here, but notice that packets 7 and 18 also had extremely high latency:

λ ping mana-tailscale
PING mana-tailscale (100.74.12.50): 56 data bytes
64 bytes from 100.74.12.50: icmp_seq=0 ttl=64 time=15.695 ms
64 bytes from 100.74.12.50: icmp_seq=1 ttl=64 time=8.315 ms
64 bytes from 100.74.12.50: icmp_seq=2 ttl=64 time=6.111 ms
64 bytes from 100.74.12.50: icmp_seq=3 ttl=64 time=4.321 ms
64 bytes from 100.74.12.50: icmp_seq=4 ttl=64 time=5.954 ms
64 bytes from 100.74.12.50: icmp_seq=5 ttl=64 time=7.019 ms
64 bytes from 100.74.12.50: icmp_seq=6 ttl=64 time=6.450 ms
64 bytes from 100.74.12.50: icmp_seq=7 ttl=64 time=910.145 ms
64 bytes from 100.74.12.50: icmp_seq=8 ttl=64 time=24.474 ms
64 bytes from 100.74.12.50: icmp_seq=9 ttl=64 time=85.266 ms
64 bytes from 100.74.12.50: icmp_seq=10 ttl=64 time=6.987 ms
64 bytes from 100.74.12.50: icmp_seq=11 ttl=64 time=6.796 ms
64 bytes from 100.74.12.50: icmp_seq=12 ttl=64 time=7.244 ms
64 bytes from 100.74.12.50: icmp_seq=13 ttl=64 time=3.987 ms
64 bytes from 100.74.12.50: icmp_seq=14 ttl=64 time=5.992 ms
64 bytes from 100.74.12.50: icmp_seq=15 ttl=64 time=5.294 ms
Request timeout for icmp_seq 16
64 bytes from 100.74.12.50: icmp_seq=17 ttl=64 time=42.827 ms
64 bytes from 100.74.12.50: icmp_seq=18 ttl=64 time=318.292 ms
64 bytes from 100.74.12.50: icmp_seq=19 ttl=64 time=26.196 ms
64 bytes from 100.74.12.50: icmp_seq=20 ttl=64 time=6.390 ms
64 bytes from 100.74.12.50: icmp_seq=21 ttl=64 time=6.711 ms
64 bytes from 100.74.12.50: icmp_seq=22 ttl=64 time=7.249 ms
^C
--- mana-tailscale ping statistics ---
23 packets transmitted, 22 packets received, 4.3% packet loss
round-trip min/avg/max/stddev = 3.987/68.987/910.145/194.962 ms

A concurrently-running ping on the non-tailscale LAN interface never exceeded 10ms:

302 packets transmitted, 302 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.570/3.786/8.163/1.483 ms

BUG-e2b6f53c6548bbac0b9b9211bd3ece7b87fe65ad62845cc6ce6bb84ff0af1cd9-20240627153055Z-95c1c05efcb97834

This also appears to affect me only when the two machines are on the same network. When I bring the Mac to a different network, my pings are much more consistent:

100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 17.485/29.285/79.669/10.621 ms

@sartak sartak changed the title Elevated packet loss from macOS Elevated packet latency macOS -> Linux, only on same LAN(?) Jun 27, 2024
@sartak sartak changed the title Elevated packet latency macOS -> Linux, only on same LAN(?) Elevated packet latency macOS → Linux, only on same LAN(?) Jun 27, 2024
@ThisIsCyreX
Copy link

Feels like the same issue I have. Android and Linux have a very high packet loss (not constant) on my local wifi network. Cellular is fine. #12662

@entrptaher
Copy link

Facing same issue, and this happens only on one device in the lan which is definitely weird.

image

It could be a mac specific issue though. I tried disabling location services without any result.

Both server 1 and server 2 is under same router, in same physical location.

Then For some weird reason the issue vanished when I start to ping at high rate from all connections. Occational random issue persisted though.

image

The problem is the problem is not consistent, so it's hard to figure out what is the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
L1 Very few Likelihood needs-triage needs-triage-eng Ready for triage by Engineering team OS-macos Issues involving Tailscale for macOS P2 Aggravating Priority level T3 Performance/Debugging Issue type
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sartak @knyar @entrptaher @ThisIsCyreX