Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

thread_stress_test flaky (Linux dbg and opt) #13330

Closed
yang-g opened this issue Nov 9, 2017 · 10 comments
Closed

thread_stress_test flaky (Linux dbg and opt) #13330

yang-g opened this issue Nov 9, 2017 · 10 comments
Assignees
@vjpai
Labels
disposition/BUILDNURSE For all buildnurse related build/test failures and flakes lang/c++ platform/Linux

Comments

@yang-g
Copy link
Member

yang-g commented Nov 9, 2017 •
edited by jtattermusch
Loading 

[ RUN      ] AsyncClientEnd2endTest/0.ThreadStress
D1108 17:22:51.526708000 140735181389824 ev_posix.cc:135]              Using polling engine: poll
D1108 17:22:51.526754000 140735181389824 dns_resolver.cc:307]          Using native dns resolver
I1108 17:22:51.529232000 140735181389824 server_builder.cc:269]        Synchronous server. Num CQs: 8, Min pollers: 1, Max Pollers: 2, CQ timeout (msec): 10000
libc++abi.dylib: libc++abi.dylib: libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailableterminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailableterminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailable


libc++abi.dylib: libc++abi.dylib: libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailableterminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailableterminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailable
libc++abi.dylib: libc++abi.dylib: 

terminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailableterminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailable
@yang-g yang-g added the disposition/BUILDNURSE For all buildnurse related build/test failures and flakes label Nov 9, 2017
@yang-g yang-g mentioned this issue Nov 9, 2017
Closed
@yang-g yang-g mentioned this issue Nov 9, 2017
Merged
@dgquintas
Copy link
Contributor

Seen again: https://sponge.corp.google.com/target?id=15cc1ab7-bc2c-46be-a909-e874af69638f&target=github/grpc/c%2B%2B_macos_dbg_native&searchFor=&show=FAILED&sortBy=STATUS

@vjpai any idea how we could deal with this?

@dgquintas dgquintas mentioned this issue Nov 10, 2017
Merged
@dgquintas
Copy link
Contributor

Again: https://sponge.corp.google.com/target?id=96b4507d-db04-4f5f-b150-4284dd51ba57&target=github/grpc/c%2B%2B_macos_dbg_native&searchFor=&show=FAILED&sortBy=STATUS

@dgquintas dgquintas mentioned this issue Nov 10, 2017
Merged
@dgquintas
Copy link
Contributor

It seems to be happening consistently now: https://sponge.corp.google.com/target?id=3a35381a-015b-4a11-92e3-7f0b590b6a82&target=github/grpc/c%2B%2B_macos_opt_native&searchFor=&show=FAILED&sortBy=STATUS

@dgquintas dgquintas mentioned this issue Nov 10, 2017
Merged
@yang-g yang-g mentioned this issue Nov 10, 2017
Merged
@jtattermusch
Copy link
Contributor

Still happening for bins/opt/thread_stress_test --gtest_filter=AsyncClientEnd2endTest/0.ThreadStress GRPC_POLL_STRATEGY=poll on Mac.

@jtattermusch jtattermusch changed the title macos C++ test fails because of running out of threads MacOS thread_stress_test consistently fails (because of running out of threads?) Nov 13, 2017
@jtattermusch
Copy link
Contributor

Bumping to P1. @vjpai do you have cycles to actively investigate this or should be reassign?

@dgquintas dgquintas mentioned this issue Nov 14, 2017
Merged
@vjpai
Copy link
Member

vjpai commented Nov 14, 2017 •
edited
Loading

Since we are allergic to exceptions, I propose that we stop using std::thread in the C++ library. We should be using gpr_thread so that we can at least clean it up and return resources_exhausted. I'll write a PR to do this.

@vjpai
Copy link
Member

vjpai commented Nov 28, 2017

Still working on the sync server refactor that should help here

@sreecha sreecha mentioned this issue Dec 1, 2017
Merged
@jtattermusch
Copy link
Contributor

This also seems to be flaky with tsan and asan, as well as on linux dbg and opt.

@jtattermusch jtattermusch changed the title MacOS thread_stress_test consistently fails (because of running out of threads?) thread_stress_test flaky (Linux dbg and opt) Dec 14, 2017
@jtattermusch
Copy link
Contributor

This is now only happening on linux and with much lower frequency:

https://sponge.corp.google.com/target?id=ef6c1968-6021-4e12-beae-4e0d301a4d5c&target=github/grpc/c%2B%2B_linux_opt_native&searchFor=&show=FAILED&sortBy=STATUS

@mehrdada mehrdada mentioned this issue Jan 2, 2018
Merged
@yang-g yang-g mentioned this issue Jan 12, 2018
Merged
@jtattermusch
Copy link
Contributor

Duplicate of #13148.

@lock lock bot locked as resolved and limited conversation to collaborators Sep 30, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@vjpai vjpai

Labels
disposition/BUILDNURSE For all buildnurse related build/test failures and flakes lang/c++ platform/Linux
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dgquintas @vjpai @jtattermusch @yang-g @nathanielmanistaatgoogle