Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

We can turn our cpu profile dumping to be crash-only (was: empty cpu profile (profiling haproxy on fedora)) #1475

Open
chipitsine opened this issue Jan 7, 2024 · 5 comments

Comments

@chipitsine
Copy link

chipitsine commented Jan 7, 2024

some repro steps:

git clone https://github.com/haproxy/haproxy
cd haproxy

make all ERR=1 TARGET=linux-glibc USE_OPENSSL=1 LDFLAGS="-lprofiler"

curl -fsSL https://github.com/summerwind/h2spec/releases/download/v2.6.0/h2spec_linux_amd64.tar.gz -o h2spec.tar.gz
tar xvf h2spec.tar.gz
chmod +x h2spec

CPUPROFILE=haproxy.prof ./haproxy -f .github/h2spec.config -d &
sleep 5
./h2spec -Svtk -h 127.0.0.1 -p 8443

killall  haproxy

file haproxy.prof appears, but it's empty

ilia@fedora:~/temp/haproxy-profile$ ls -l haproxy.prof 
-rw-r--r-- 1 ilia ilia 0 Jan  7 16:05 haproxy.prof
ilia@fedora:~/temp/haproxy-profile$ 

OS: Fedora 39
gperftools: gperftools-libs-2.9.1-6.fc39.x86_64

@chipitsine
Copy link
Author

I've just noticed that Fedora is using pretty old gperftools.
give me some time to try the latest release

@chipitsine
Copy link
Author

well, I tested on 2.15, the same zero size haproxy.prof

@alk
Copy link
Contributor

alk commented Jan 7, 2024

This is because profile is dumped on program shutdown. If it dies from signal, there is no profile flushing.

Have your program stop normally, or have it stop normally on signal. Or arrange profiling for specific duration and Stop it yourself.

As somewhat crude (and not 100% reliable) workaround people have contributed CPUPROFILESIGNAL. Which allows you to toggle profiling on/off by sending that signal. Feel free to look it up in the code.

Yes, we could turn our profile dumping design to be continuous (via mmap-ed files) and crash-only. But this is an enhancement.

Let me turn this ticket into such enhancement. If there is anyone willing to volunteer several days of work into this, I am willing to guide such person.

@alk alk changed the title empty cpu profile (profiling haproxy on fedora) We can turn our cpu profile dumping to be crash-only (was: empty cpu profile (profiling haproxy on fedora)) Jan 7, 2024
@alk alk added the enhancement label Jan 7, 2024
@chipitsine
Copy link
Author

chipitsine commented Jan 7, 2024

hello, @alk

thank you for you help. I previously tried CPUPROFILESIGNAL, but it also behaves in strange way
as far as I understand, I must send a signal twice: to begin collecting and to end collecting.

git clone https://github.com/haproxy/haproxy
cd haproxy

make all ERR=1 TARGET=linux-glibc USE_OPENSSL=1 LDFLAGS="-lprofiler"

curl -fsSL https://github.com/summerwind/h2spec/releases/download/v2.6.0/h2spec_linux_amd64.tar.gz -o h2spec.tar.gz
tar xvf h2spec.tar.gz
chmod +x h2spec

CPUPROFILE=haproxy.prof CPUPROFILESIGNAL=12 ./haproxy -f .github/h2spec.config -d &
sleep 5
killall -12 haproxy
./h2spec -Svtk -h 127.0.0.1 -p 8443
killall -12 haproxy
sleep 5

killall  haproxy

no haproxy.prof was created

ilia@fedora:~/temp/haproxy-profile$ ls -l haproxy.prof
ls: cannot access 'haproxy.prof': No such file or directory
ilia@fedora:~/temp/haproxy-profile$ 

@chipitsine
Copy link
Author

nevermind, I've managed with "31" signal. 12 was just handled by haproxy itself

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants