Skip to content

velox_local_runner_test is flaky #11878

Open
@kagamiori

Description

Bug description

Example failure: https://github.com/facebookincubator/velox/actions/runs/12333843341/job/34423296887.

System information

CI

Relevant logs

2024-12-15T00:48:38.5214561Z 377/384 Test #382: velox_local_runner_test ..............................................Subprocess aborted***Exception:   9.40 sec
2024-12-15T00:48:38.5215885Z [==========] Running 3 tests from 1 test suite.
2024-12-15T00:48:38.5216613Z [----------] Global test environment set-up.
2024-12-15T00:48:38.5217136Z [----------] 3 tests from LocalRunnerTest
2024-12-15T00:48:38.5217560Z [ RUN      ] LocalRunnerTest.count
2024-12-15T00:48:38.5218254Z I1215 00:48:29.939764 94011 Compression.cpp:627] Initialized zstd compressor with compression level 7
2024-12-15T00:48:38.5218996Z [       OK ] LocalRunnerTest.count (2944 ms)
2024-12-15T00:48:38.5219425Z [ RUN      ] LocalRunnerTest.error
2024-12-15T00:48:38.5221297Z E20241215 00:48:33.879781 94011 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:308, Function:shutdown, Expression: participants_.size() == 0 (1 vs. 0) Unexpected alive participants on destruction, Source: RUNTIME, ErrorCode: INVALID_STATE
2024-12-15T00:48:38.5223445Z terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
2024-12-15T00:48:38.5224125Z   what():  Exception: VeloxRuntimeError
2024-12-15T00:48:38.5224525Z Error Source: RUNTIME
2024-12-15T00:48:38.5224845Z Error Code: INVALID_STATE
2024-12-15T00:48:38.5225295Z Reason: (1 vs. 0) Unexpected alive participants on destruction
2024-12-15T00:48:38.5225803Z Retriable: False
2024-12-15T00:48:38.5226129Z Expression: participants_.size() == 0
2024-12-15T00:48:38.5226523Z Function: shutdown
2024-12-15T00:48:38.5227093Z File: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp
2024-12-15T00:48:38.5227997Z Line: 308
2024-12-15T00:48:38.5228256Z Stack trace:
2024-12-15T00:48:38.5228591Z # 0  _ZN8facebook5velox7process10StackTraceC1Ei
2024-12-15T00:48:38.5229853Z # 1  _ZN8facebook5velox14VeloxException5State4makeIZNS1_C4EPKcmS5_St17basic_string_viewIcSt11char_traitsIcEES9_S9_S9_bNS1_4TypeES9_EUlRT_E_EESt10shared_ptrIKS2_ESA_SB_
2024-12-15T00:48:38.5231505Z # 2  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
2024-12-15T00:48:38.5232786Z # 3  _ZN8facebook5velox17VeloxRuntimeErrorC2EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bS7_
2024-12-15T00:48:38.5234401Z # 4  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
2024-12-15T00:48:38.5236285Z # 5  _ZN8facebook5velox6memory16SharedArbitrator8shutdownEv
2024-12-15T00:48:38.5326710Z # 6  _ZN8facebook5velox6memory13MemoryManagerD1Ev
2024-12-15T00:48:38.5327638Z # 7  _ZN8facebook5velox6memory13MemoryManager18testingSetInstanceERKNS1_20MemoryManagerOptionsE
2024-12-15T00:48:38.5328698Z # 8  _ZN8facebook5velox4exec4test16OperatorTestBase11setupMemoryElllllll
2024-12-15T00:48:38.5329549Z # 9  _ZN8facebook5velox4exec4test16OperatorTestBase11resetMemoryEv
2024-12-15T00:48:38.5330286Z # 10 _ZN8facebook5velox4exec4test16OperatorTestBase8TearDownEv
2024-12-15T00:48:38.5331029Z # 11 _ZN8facebook5velox4exec4test21HiveConnectorTestBase8TearDownEv
2024-12-15T00:48:38.5332029Z # 12 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5333211Z # 13 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5334004Z # 14 _ZN7testing4Test3RunEv
2024-12-15T00:48:38.5334363Z # 15 _ZN7testing8TestInfo3RunEv
2024-12-15T00:48:38.5334746Z # 16 _ZN7testing9TestSuite3RunEv
2024-12-15T00:48:38.5335270Z # 17 _ZN7testing8internal12UnitTestImpl11RunAllTestsEv
2024-12-15T00:48:38.5336234Z # 18 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5337865Z # 19 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5338734Z # 20 _ZN7testing8UnitTest3RunEv
2024-12-15T00:48:38.5339087Z # 21 _Z13RUN_ALL_TESTSv
2024-12-15T00:48:38.5339387Z # 22 main
2024-12-15T00:48:38.5339643Z # 23 0x0000000000029d8f
2024-12-15T00:48:38.5339936Z # 24 __libc_start_main
2024-12-15T00:48:38.5340232Z # 25 _start
2024-12-15T00:48:38.5340387Z 
2024-12-15T00:48:38.5340654Z *** Aborted at 1734223714 (Unix time, try 'date -d @1734223714') ***
2024-12-15T00:48:38.5341818Z *** Signal 6 (SIGABRT) (0x3e900016f3b) received by PID 94011 (pthread TID 0x7f5dca29a340) (linux TID 94011) (maybe from PID 94011, UID 1001) (code: -6), stack trace: ***
2024-12-15T00:48:38.5343165Z     @ 00000000175ccb9e _ZN5folly10symbolizer12_GLOBAL__N_118innerSignalHandlerEiP9siginfo_tPv
2024-12-15T00:48:38.5344386Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/folly-src/folly/experimental/symbolizer/SignalHandler.cpp:453
2024-12-15T00:48:38.5345560Z     @ 00000000175ccc84 _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
2024-12-15T00:48:38.5346725Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/folly-src/folly/experimental/symbolizer/SignalHandler.cpp:474
2024-12-15T00:48:38.5347817Z     @ 000000000004251f (unknown)
2024-12-15T00:48:38.5348177Z     @ 00000000000969fc pthread_kill
2024-12-15T00:48:38.5348535Z     @ 0000000000042475 raise
2024-12-15T00:48:38.5348861Z     @ 00000000000287f2 abort
2024-12-15T00:48:38.5349187Z     @ 00000000000a2b9d (unknown)
2024-12-15T00:48:38.5349534Z     @ 00000000000ae20b (unknown)
2024-12-15T00:48:38.5349876Z     @ 00000000000ad1e8 (unknown)
2024-12-15T00:48:38.5350238Z     @ 00000000000ad958 __gxx_personality_v0
2024-12-15T00:48:38.5350666Z     @ 000000000000afe8 __libunwind_Unwind_Resume
2024-12-15T00:48:38.5351307Z     @ 00000000173d1292 _ZN8facebook5velox6memory16SharedArbitrator8shutdownEv
2024-12-15T00:48:38.5352219Z                        /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:308
2024-12-15T00:48:38.5353061Z     @ 0000000017384df9 _ZN8facebook5velox6memory13MemoryManagerD1Ev
2024-12-15T00:48:38.5353823Z                        /home/runner/work/velox/velox/velox/velox/common/memory/Memory.cpp:137
2024-12-15T00:48:38.5354932Z     @ 0000000017385495 _ZN8facebook5velox6memory13MemoryManager18testingSetInstanceERKNS1_20MemoryManagerOptionsE
2024-12-15T00:48:38.5356030Z                        /home/runner/work/velox/velox/velox/velox/common/memory/Memory.cpp:204
2024-12-15T00:48:38.5357195Z     @ 000000000c306de9 _ZN8facebook5velox4exec4test16OperatorTestBase11setupMemoryElllllll
2024-12-15T00:48:38.5358231Z                        /home/runner/work/velox/velox/velox/velox/exec/tests/utils/OperatorTestBase.cpp:113
2024-12-15T00:48:38.5359211Z     @ 000000000c307172 _ZN8facebook5velox4exec4test16OperatorTestBase11resetMemoryEv
2024-12-15T00:48:38.5360184Z                        /home/runner/work/velox/velox/velox/velox/exec/tests/utils/OperatorTestBase.cpp:120
2024-12-15T00:48:38.5361132Z     @ 000000000c307352 _ZN8facebook5velox4exec4test16OperatorTestBase8TearDownEv
2024-12-15T00:48:38.5362075Z                        /home/runner/work/velox/velox/velox/velox/exec/tests/utils/OperatorTestBase.cpp:160
2024-12-15T00:48:38.5363056Z     @ 000000000c2c60a9 _ZN8facebook5velox4exec4test21HiveConnectorTestBase8TearDownEv
2024-12-15T00:48:38.5364085Z                        /home/runner/work/velox/velox/velox/velox/exec/tests/utils/HiveConnectorTestBase.cpp:69
2024-12-15T00:48:38.5365333Z     @ 000000000c5ca136 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5366564Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2621
2024-12-15T00:48:38.5367633Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5369025Z     @ 000000000c5c3302 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5370229Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2657
2024-12-15T00:48:38.5371283Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5372038Z     @ 000000000c59ed61 _ZN7testing4Test3RunEv
2024-12-15T00:48:38.5372776Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2704
2024-12-15T00:48:38.5373830Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5374599Z     @ 000000000c59f82a _ZN7testing8TestInfo3RunEv
2024-12-15T00:48:38.5375352Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2845
2024-12-15T00:48:38.5376417Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5377182Z     @ 000000000c5a0177 _ZN7testing9TestSuite3RunEv
2024-12-15T00:48:38.5377937Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:3004
2024-12-15T00:48:38.5378989Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5379903Z     @ 000000000c5b05ef _ZN7testing8internal12UnitTestImpl11RunAllTestsEv
2024-12-15T00:48:38.5380821Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:5890
2024-12-15T00:48:38.5381882Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5383185Z     @ 000000000c5cb047 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5384495Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2621
2024-12-15T00:48:38.5385547Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5386828Z     @ 000000000c5c43ba _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5388278Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2657
2024-12-15T00:48:38.5389535Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5390304Z     @ 000000000c5aebd6 _ZN7testing8UnitTest3RunEv
2024-12-15T00:48:38.5391059Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:5455
2024-12-15T00:48:38.5392121Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5392859Z     @ 000000000c2a515c _Z13RUN_ALL_TESTSv
2024-12-15T00:48:38.5393641Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/include/gtest/gtest.h:2314
2024-12-15T00:48:38.5394603Z                        -> /home/runner/work/velox/velox/velox/velox/runner/tests/Main.cpp
2024-12-15T00:48:38.5395168Z     @ 000000000c2a508d main
2024-12-15T00:48:38.5395683Z                        /home/runner/work/velox/velox/velox/velox/runner/tests/Main.cpp:28
2024-12-15T00:48:38.5396265Z     @ 0000000000029d8f (unknown)
2024-12-15T00:48:38.5396623Z     @ 0000000000029e3f __libc_start_main
2024-12-15T00:48:38.5397006Z     @ 000000000c283d04 _start
2024-12-15T00:48:38.5397435Z Fatal signal handler. ThreadDebugInfo object not found.

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriageNewly created issue that needs attention.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions