Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test FilterProjectReplayerTest.projectOnly in velox_tool_trace_test crashes #12071

Open
kevinwilfong opened this issue Jan 13, 2025 · 2 comments
Open

Flaky test FilterProjectReplayerTest.projectOnly in velox_tool_trace_test crashes #12071

kevinwilfong opened this issue Jan 13, 2025 · 2 comments
Assignees
@duanmeng
Labels
bug Something isn't working triage Newly created issue that needs attention.

Comments

@kevinwilfong
Copy link
Contributor

Bug description

I frequently see FilterProjectReplayerTest.projectOnly in velox_tool_trace_test in the "Build with GCC / Linux release with adapters" CI job crash with a SIGSEGV on PRs.

*** Signal 11 (SIGSEGV) (0x0) received by PID 83215 (pthread TID 0x7f475a704b80) (linux TID 83215) (code: address not mapped to object), stack trace: ***
(error retrieving stack trace)
Fatal signal handler. ThreadDebugInfo object not found.

E.g. (multiple PRs from multiple authors)
https://github.com/facebookincubator/velox/actions/runs/12745207168/job/35518716720
https://github.com/facebookincubator/velox/actions/runs/12741414722/job/35507977698
https://github.com/facebookincubator/velox/actions/runs/12719384670/job/35459508390

System information

"Build with GCC / Linux release with adapters" CI job

Relevant logs

No response
@kevinwilfong kevinwilfong added bug Something isn't working triage Newly created issue that needs attention. labels Jan 13, 2025
@kevinwilfong
Copy link
Contributor Author

I found this stack trace in one of them

*** Aborted at 1736561111 (Unix time, try 'date -d @1736561111') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 85813 (pthread TID 0x7f3da04f7340) (linux TID 85813) (code: address not mapped to object), stack trace: ***
@ 000000000cf0cd48 _ZN5folly10symbolizer12_GLOBAL__N_118innerSignalHandlerEiP9siginfo_tPv
/home/runner/work/velox/velox/velox/_build/debug/_deps/folly-src/folly/experimental/symbolizer/SignalHandler.cpp:453
@ 000000000cf0ce2e _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
/home/runner/work/velox/velox/velox/_build/debug/_deps/folly-src/folly/experimental/symbolizer/SignalHandler.cpp:474
@ 000000000004251f (unknown)
@ 000000000c34292a _ZNKSt12__shared_ptrIN8facebook5velox9RowVectorELN9__gnu_cxx12_Lock_policyE2EE3getEv
/usr/include/c++/11/bits/shared_ptr_base.h:1296
-> /home/runner/work/velox/velox/velox/velox/tool/trace/tests/AggregationReplayerTest.cpp
@ 000000000c33ecb5 _ZNKSt19__shared_ptr_accessIN8facebook5velox9RowVectorELN9__gnu_cxx12_Lock_policyE2ELb0ELb0EE6_M_getEv
/usr/include/c++/11/bits/shared_ptr_base.h:993
-> /home/runner/work/velox/velox/velox/velox/tool/trace/tests/AggregationReplayerTest.cpp
@ 000000000c33b2db _ZNKSt19__shared_ptr_accessIN8facebook5velox9RowVectorELN9__gnu_cxx12_Lock_policyE2ELb0ELb0EEptEv
/usr/include/c++/11/bits/shared_ptr_base.h:987
-> /home/runner/work/velox/velox/velox/velox/tool/trace/tests/AggregationReplayerTest.cpp
@ 000000000cc7dea9 _ZN8facebook5velox4tool5trace21TraceReplayTaskRunner4copyERKSt6vectorISt10shared_ptrINS0_9RowVectorEESaIS7_EE
/home/runner/work/velox/velox/velox/velox/tool/trace/TraceReplayTaskRunner.cpp:48
@ 000000000cc7dd13 _ZN8facebook5velox4tool5trace21TraceReplayTaskRunner3runEb
/home/runner/work/velox/velox/velox/velox/tool/trace/TraceReplayTaskRunner.cpp:35
@ 000000000cc590d4 _ZN8facebook5velox4tool5trace20OperatorReplayerBase3runEb
/home/runner/work/velox/velox/velox/velox/tool/trace/OperatorReplayerBase.cpp:87
@ 000000000c352b77 _ZN8facebook5velox4tool5trace4test42FilterProjectReplayerTest_projectOnly_Test8TestBodyEv
/home/runner/work/velox/velox/velox/velox/tool/trace/tests/FilterProjectReplayerTest.cpp:284
@ 000000001796619a _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2621
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 000000001795ff54 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2657
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 000000001793d32f _ZN7testing4Test3RunEv
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2696
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 000000001793de4a _ZN7testing8TestInfo3RunEv
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2845
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 000000001793e797 _ZN7testing9TestSuite3RunEv
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:3004
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 000000001794ec0f _ZN7testing8internal12UnitTestImpl11RunAllTestsEv
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:5890
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 00000000179670ab _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2621
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 000000001796100c _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2657
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 000000001794d1f6 _ZN7testing8UnitTest3RunEv
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:5455
-> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
@ 000000000c3c156f _Z13RUN_ALL_TESTSv
/home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/include/gtest/gtest.h:2314
-> /home/runner/work/velox/velox/velox/velox/tool/trace/tests/TableWriterReplayerTest.cpp
@ 000000000c3bc40a main
/home/runner/work/velox/velox/velox/velox/tool/trace/tests/TableWriterReplayerTest.cpp:524
@ 0000000000029d8f (unknown)
@ 0000000000029e3f __libc_start_main
@ 000000000c32ada4 _start
Fatal signal handler. ThreadDebugInfo object not found.

@kgpai
Copy link
Contributor

kgpai commented Jan 13, 2025

cc: @duanmeng

@duanmeng duanmeng self-assigned this Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@duanmeng duanmeng

Labels
bug Something isn't working triage Newly created issue that needs attention.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kgpai @kevinwilfong @duanmeng