Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High CPU usage by tailscaled (top reports 197%) #10954

Open
mrisher opened this issue Jan 26, 2024 · 25 comments
Open

High CPU usage by tailscaled (top reports 197%) #10954

mrisher opened this issue Jan 26, 2024 · 25 comments
Labels
bug Bug funnel Relating to Tailscale Funnel https://tailscale.com/blog/introducing-tailscale-funnel/ OS-linux

Comments

@mrisher
Copy link

mrisher commented Jan 26, 2024

What is the issue?

After briefly using tailscale serve (but turning it off) I am seeing CPU in excess of 100% on the Linux server (Google GCP virtual machine running Debian)

Steps to reproduce

tailscale serve localhost:2112
tailscale serve localhost:2112 off
tailscale serve status

mrisher@weaviate:~$ sudo tailscale serve status
No serve config
image image

Are there any recent changes that introduced the issue?

Turning serve on and off? I am using this machine as an Exit node, but for minimal traffic (just accessing a web-based code editor, Google Colab)

OS

Linux

OS version

Debian GNU/Linux 11 (bullseye)

Tailscale version

1.58.2 tailscale commit: 6a2a8457f0d62c775bfee3e350d2b34d85eda0f5 other commit: 8c7d4563e46a1856053912ac067fc5036c2d2b3e go version: go1.21.5

Other software

Weaviate 1.23.0 is running on the same server; top reports its CPU at 1.3%

Bug report

BUG-71075528b847dd4e833cbc922e417037397f623d21ad382a4496771cd49c8bfa-20240126143412Z-034be10ca854edd3

@mrisher mrisher changed the title High CPU usage High CPU usage by tailscaled (top reports 197%) Jan 26, 2024
@mrisher
Copy link
Author

mrisher commented Jan 28, 2024

Possibly related, I'm now seeing this error every few seconds
Jan 28 07:14:47 weaviate tailscaled[618595]: [unexpected] magicsock: derp-18 does not know about peer [ifFEx], removing route"

image

@catzkorn catzkorn added funnel Relating to Tailscale Funnel https://tailscale.com/blog/introducing-tailscale-funnel/ OS-linux and removed needs-triage labels Jan 31, 2024
@catzkorn
Copy link
Contributor

Hi,

Thanks for your report. I've flagged this with the relevant team internally.

@kurtraschke
Copy link

I've just hit what I believe is the same issue: 1600% CPU usage (i.e. every core pegged) with tailscale serve on; resetting the tailscale serve configuration and restarting tailscaled returned CPU usage to normal.

tailscale bugreport: BUG-d854060efdcc83069a313f42136afbc4ba857e6a833c7f8c03c1f84350722551-20240213002251Z-ed9ad910ca053428

@XGhozt
Copy link

XGhozt commented Feb 21, 2024

I may have run into this as well. The issue seems to stop when I disable "MagicDNS". I have a local AdGuard DNS server instead of public DNS servers on my network, not sure if that's related. If I don't disable MagicDNS then it just sits at 197% CPU usage for me as well.

@lucollab
Copy link

i experience exactly the same behavior. i run tailscale on a qnap 251+ nas and everything works really great. I have deployed several applications via Docker (Container Station) and can access them from anywhere outside the home network via tailscale. To deploy Vaultwarden via https in my tailnet, I ran the following command (i defined 2284 as the host port in the docker compose file of vaultwarden):

tailscale serve https:2284 / 127.0.0.1:2284

The CPU load of the entire NAS then increases to 97%. As soon as I stop and restart tailscaled, the service is back to a few percent.

i also run a pi.hole in my network, but on a standalone raspberry pi, which is not (yet) in the tailnet.

@mrisher
Copy link
Author

mrisher commented Apr 16, 2024 via email

@colaH16
Copy link

colaH16 commented Apr 16, 2024

That sounds very similar. So this thread has been open for awhile now; any suggestions on how to make progress?

On Tue, Apr 16, 2024 at 8:40 AM Lukas @.> wrote: i experience exactly the same behavior. i run tailscale on a qnap 251+ nas and everything works really great. I have deployed several applications via Docker (Container Station) and can access them from anywhere outside the home network via tailscale. To deploy Vaultwarden via https in my tailnet, I ran the following command (i defined 2284 as the host port in the docker compose file of vaultwarden): tailscale serve https:2284 / 127.0.0.1:2284 The CPU load of the entire NAS then increases to 97%. As soon as I stop and restart tailscaled, the service is back to a few percent. i also run a pi.hole in my network, but on a standalone raspberry pi, which is not (yet) in the tailnet. — Reply to this email directly, view it on GitHub <#10954 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABVHNR6FMXECD35T4O67KTY5TBVLAVCNFSM6AAAAABCMICVJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJYGM2DGMZVGQ . You are receiving this because you authored the thread.Message ID: @.>

I am having problems with two types of my servers.
One is an instance of Oracle cloud and the others are proxmox vm.

I run this commands to start tailscale
proxmox vm
sudo tailscale up --advertise-tags tag:home-rack-vm --ssh --accept-routes=false --accept-dns=false --hostname=vm-hostname | cut -f1 -d. --exit-node= --advertise-routes=10.0.0.0/14,192.168.10.0/23

OCI Instance
sudo tailscale up --ssh --advertise-connector --advertise-tags tag:hybrid-k8s-node --accept-routes=false --accept-dns=false --hostname=hybrid-k8s-node-hostname | cut -f1 -d. --advertise-exit-node --exit-node= --advertise-routes=10.123.23.0/24,169.254.169.254/32,10.123.23.0/28

On tailscale web, subnet routes are switched off for all machines

@lucollab
Copy link

lucollab commented Apr 16, 2024

@colaH16: Can you provide some more information - I'm currently at a loss as far as the information in your post is concerned. How exactly is your tailscale situation related to this problem?

addendum to my post:

after I executed the serve command, the CPU load increased to almost 100% as usual. I didn't deactivate the serve by "off" afterwards, but stopped the tailscale app on the qnap nas and started it again. Now tailscaled is running with "normal" CPU load (0.5-2%) and the serve is still active.

if you want me to provide any logs, please let me know. However, I currently don't know how exactly I can read the tailscale logs on the qnap device.

btw: I currently see no effects on the pihole, whose logs I view simultaneously

@mrisher
Copy link
Author

mrisher commented Apr 16, 2024 via email

@colaH16
Copy link

colaH16 commented Apr 16, 2024

@lucollab
I didn't use serve..
Just using tailscale up.

Pinging between tailscale servers sometimes results in a time out.
At this point, the usage of the tailscale process is close to 200%.


I ran the test as shown in this photo.
I actually run top and forward pings to the tailscale ip.
Then every 2-10 minutes the CPU usage increases and stops forwarding pings to the tailscale ip.

@dexterdy
Copy link

I have similar issues after using tailscale funnel

@batuozdemir
Copy link

I'm having a similar issue on my VPS (using funnel). Did anyone find a solution?

@colaH16
Copy link

colaH16 commented Jun 30, 2024

no. I just limit cpu by linux systemd service file
I'm still losing connections to Tailscale, but I've stopped other apps from slowing down due to tailscale’s CPU usage.

@dexterdy
Copy link

dexterdy commented Jul 1, 2024

I'm having a similar issue on my VPS (using funnel). Did anyone find a solution?

I switched to cloudflare tunnels. It's also free for individuals and besides not having this issue, they have some other advantages.

@batuozdemir
Copy link

I'm having a similar issue on my VPS (using funnel). Did anyone find a solution?

I switched to cloudflare tunnels. It's also free for individuals and besides not having this issue, they have some other advantages.

This works beautifully, much faster too. Thanks!

@TheJesusFish
Copy link

I’m also running into this issue. Maybe it has to do with another device on the tailnet being a subnet router? I couldn’t even connect to the LAN IP of my Qnap until I used the —accept-routes=false flag.

@vpant
Copy link

vpant commented Jul 5, 2024

I am having the same issue when I use sudo tailscale up --accept-routes .

sudo tailscale up does not actually cause this issue but I cannot ping devices with the advertised ip.

@hinayand
Copy link

hinayand commented Jul 11, 2024

I have met this problem, too.

I run alist on my OrangePi 5 Pro. To let me can access it from the Internet and don't need to install tailscale app, I ran tailscale funnel 5244 on my device.

And few hours later, my device shutted down because tailscale makes a high temprature.

When I found it was shutting down, I restarted it and ran tailscale funnel 5244 again. And htop reports tailscaled make a high CPU usage.

@leeaash
Copy link

leeaash commented Jul 25, 2024

I think I may meet the same issue on my Debian VPS, the CPU usage gets high once I switch network traffic to Tailscale
tunnel on this server, it will cause cpu get high, and restart tailscaled process won’t help until I do restart my VPS.

@joephill11
Copy link

also seeing this on one of my ubuntu vms running in virtualbox is there a way to get this investigated i can provide any logs or details needed.

@rafaelcpalmeida
Copy link

I'm also experiencing this on several VMs, including AWS EC2 instances. @catzkorn any updates from the team?

@benjamonnguyen
Copy link

Happened for me after using tailscale serve

@dpritt
Copy link

dpritt commented Oct 27, 2024

Similar issue here, I am running Tailscale on pfSense Plus and getting dismal throughput, noticed the application is hogging all the CPU
last pid: 90197; load averages: 1.38, 1.51, 1.33 up 0+18:43:39 08:00:34
184 threads: 5 running, 154 sleeping, 25 waiting
CPU: 3.4% user, 0.7% nice, 2.2% system, 1.0% interrupt, 92.7% idle
Mem: 269M Active, 216M Inact, 382M Wired, 206M Buf, 4035M Free
Swap: 1024M Total, 1024M Free

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
48094 root 110 0 1300M 113M RUN 4:28 48.97% /usr/local/bin/tailscaled -port 41641 -tun tailscale0 -statedir /usr/local/pkg/tailscale/state{tailscaled}
48094 root 36 0 1300M 113M RUN 1:46 12.99% /usr/local/bin/tailscaled -port 41641 -tun tailscale0 -statedir /usr/local/pkg/tailscale/state{tailscaled}

pfBlocker is off.

I out this on the firewall so I could contact IOT devices in my network remotely, so I am using the Gateway setting so local subnets are available.

Not sure what to do other than ditch Tailscale (which would be a shame as it has some nice features).

@superkeyor
Copy link

superkeyor commented Nov 22, 2024

I had both Tailscale and ZeroTier installed with routing enabled on the same computer. Somehow, both programs were using excessive CPU time. After disabling ZeroTier, Tailscale returned to normal :)

@dacort
Copy link

dacort commented Jan 1, 2025

Running tailscale in Alpine on Proxmox and maybe running into a similar issue.

I recently upgraded from 1.66.4-r2 -> 1.76.6-r4 (just using apk upgrade tailscale) and ever since then my CPU has been pegged. I got this message when I upgraded:

* If tailscale fails to connect after this upgrade, you may need to disable
* 'no_logs_no_support' in /etc/conf.d/tailscale.

Unsure why that's set...maybe default for Alpine? So I changed that and tried to /etc/init.d/tailscale restart but was told /usr/sbin/tailscaled is already running. OK, weird. So I just rebooted the container and it seems to be back to normal...for now. ¯_(ツ)_/¯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug funnel Relating to Tailscale Funnel https://tailscale.com/blog/introducing-tailscale-funnel/ OS-linux
Projects
None yet
Development

No branches or pull requests