Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cloud_to_cloud:node:... very flaky (TIMEOUT) #13253

Closed
dgquintas opened this issue Nov 3, 2017 · 9 comments
Closed

cloud_to_cloud:node:... very flaky (TIMEOUT) #13253

dgquintas opened this issue Nov 3, 2017 · 9 comments
Assignees
@murgatroid99
Labels
disposition/BUILDNURSE For all buildnurse related build/test failures and flakes infra/New Failure priority/P1

Comments

@dgquintas
Copy link
Contributor

dgquintas commented Nov 3, 2017 •
edited
Loading

https://sponge.corp.google.com/target?id=80ee7ec6-5b64-4ed8-9704-e376ac09e159&target=github/grpc&searchFor=&show=ALL&sortBy=STATUS

For example:

cloud_to_cloud:node:csharpcoreclr_server:large_unary 
cloud_to_cloud:node:python_server:large_unary 
cloud_to_cloud:node:ruby_server:client_streaming
@dgquintas dgquintas mentioned this issue Nov 3, 2017
Merged
@adelez
Copy link
Contributor

adelez commented Nov 3, 2017

This dashboard shows how severe the issue is: https://performanceedit-dot-grpc-testing.appspot.com/explore?dashboard=5711255578869760: Node client is the most flaky one.

@y-zeng y-zeng mentioned this issue Nov 3, 2017
Merged
@markdroth markdroth mentioned this issue Nov 3, 2017
Merged
@jtattermusch jtattermusch added the disposition/BUILDNURSE For all buildnurse related build/test failures and flakes label Nov 6, 2017
@jtattermusch
Copy link
Contributor

A bit more analysis shows that the node client is behind virtually all cloud_to_cloud interop test failures in the last days. Most failures are large_unary, followed by client_streaming.


grpc/ubuntu/master/grpc_interop_tocloud | 88 | null | cloud_to_cloud:node:c++_server:custom_metadata | cloud_to_cloud | node | c++_server | custom_metadata | TIMEOUT | 2017-11-03 22:05:43 UTC | 181.35 |  
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
2 | grpc/ubuntu/master/grpc_interop_tocloud | 87 | null | cloud_to_cloud:node:c++_server:large_unary | cloud_to_cloud | node | c++_server | large_unary | TIMEOUT | 2017-11-03 17:48:06 UTC | 186.91 |  
3 | grpc/ubuntu/master/grpc_interop_tocloud | 86 | null | cloud_to_cloud:node:csharpcoreclr_server:large_unary | cloud_to_cloud | node | csharpcoreclr_server | large_unary | TIMEOUT | 2017-11-03 16:28:36 UTC | 187.81 |  
4 | grpc/ubuntu/master/grpc_interop_tocloud | 86 | null | cloud_to_cloud:node:c++_server:large_unary | cloud_to_cloud | node | c++_server | large_unary | TIMEOUT | 2017-11-03 16:28:43 UTC | 187.56 |  
5 | grpc/ubuntu/master/grpc_interop_tocloud | 85 | null | cloud_to_cloud:node:c++_server:large_unary | cloud_to_cloud | node | c++_server | large_unary | TIMEOUT | 2017-11-03 15:48:23 UTC | 185.57 |  
6 | grpc/ubuntu/master/grpc_interop_tocloud | 85 | null | cloud_to_cloud:node:python_server:large_unary | cloud_to_cloud | node | python_server | large_unary | TIMEOUT | 2017-11-03 15:47:48 UTC | 187.59 |  
7 | grpc/ubuntu/master/grpc_interop_tocloud | 85 | null | cloud_to_cloud:node:csharp_server:large_unary | cloud_to_cloud | node | csharp_server | large_unary | TIMEOUT | 2017-11-03 15:47:31 UTC | 180.36 |  
8 | grpc/ubuntu/master/grpc_interop_tocloud | 85 | null | cloud_to_cloud:node:python_server:client_streaming | cloud_to_cloud | node | python_server | client_streaming | TIMEOUT | 2017-11-03 15:48:30 UTC | 187.25 |  
9 | grpc/ubuntu/master/grpc_interop_tocloud | 84 | null | cloud_to_cloud:node:c++_server:large_unary | cloud_to_cloud | node | c++_server | large_unary | TIMEOUT | 2017-11-03 14:46:00 UTC | 181.13 |  
10 | grpc/ubuntu/master/grpc_interop_tocloud | 83 | null | cloud_to_cloud:node:ruby_server:large_unary | cloud_to_cloud | node | ruby_server | large_unary | TIMEOUT | 2017-11-03 09:45:30 UTC | 180.71 |  
11 | grpc/ubuntu/master/grpc_interop_tocloud | 82 | null | cloud_to_cloud:node:python_server:large_unary | cloud_to_cloud | node | python_server | large_unary | TIMEOUT | 2017-11-03 07:46:03 UTC | 188.04 |  
12 | grpc/ubuntu/master/grpc_interop_tocloud | 81 | null | cloud_to_cloud:node:python_server:client_streaming | cloud_to_cloud | node | python_server | client_streaming | TIMEOUT | 2017-11-03 02:23:04 UTC | 181.55 |  
13 | grpc/ubuntu/master/grpc_interop_tocloud | 80 | null | cloud_to_cloud:node:c++_server:client_streaming | cloud_to_cloud | node | c++_server | client_streaming | TIMEOUT | 2017-11-02 15:07:50 UTC | 186.11 |  
14 | grpc/ubuntu/master/grpc_interop_tocloud | 79 | null | cloud_to_cloud:node:c++_server:large_unary | cloud_to_cloud | node | c++_server | large_unary | TIMEOUT | 2017-11-02 13:31:24 UTC | 182.7 |  
15 | grpc/ubuntu/master/grpc_interop_tocloud | 79 | null | cloud_to_cloud:node:ruby_server:large_unary | cloud_to_cloud | node | ruby_server | large_unary | TIMEOUT | 2017-11-02 13:31:08 UTC | 180.46 |  
16 | grpc/ubuntu/master/grpc_interop_tocloud | 77 | null | cloud_to_cloud:node:python_server:client_streaming | cloud_to_cloud | node | python_server | client_streaming | TIMEOUT | 2017-11-02 03:11:00 UTC | 182.57 |  
17 | grpc/ubuntu/master/grpc_interop_tocloud | 76 | null | cloud_to_cloud:node:c++_server:large_unary | cloud_to_cloud | node | c++_server | large_unary | TIMEOUT | 2017-11-01 20:54:49 UTC | 183.8 |  
18 | grpc/ubuntu/master/grpc_interop_tocloud | 75 | null | cloud_to_cloud:node:python_server:large_unary | cloud_to_cloud | node | python_server | large_unary | TIMEOUT | 2017-11-01 15:54:38 UTC | 186.2 |  
19 | grpc/ubuntu/master/grpc_interop_tocloud | 75 | null | cloud_to_cloud:node:ruby_server:client_streaming | cloud_to_cloud | node | ruby_server | client_streaming | TIMEOUT | 2017-11-01 15:55:50 UTC | 188.49 |  
20 | grpc/ubuntu/master/grpc_interop_tocloud | 75 | null | cloud_to_cloud:node:csharpcoreclr_server:large_unary | cloud_to_cloud | node | csharpcoreclr_server | large_unary | TIMEOUT | 2017-11-01 15:55:08 UTC | 187.81 |  
21 | grpc/ubuntu/master/grpc_interop_tocloud | 74 | null | cloud_to_cloud:node:c++_server:large_unary | cloud_to_cloud | node | c++_server | large_unary | TIMEOUT | 2017-11-01 13:34:11 UTC | 189.75 |  
22 | grpc/ubuntu/master/grpc_interop_tocloud | 71 | null | cloud_to_cloud:node:c++_server:large_unary | cloud_to_cloud | node | c++_server | large_unary | TIMEOUT | 2017-10-31 22:31:47 UTC | 180.1 |  
23 | grpc/ubuntu/master/grpc_interop_tocloud | 70 | null | cloud_to_cloud:node:csharpcoreclr_server:large_unary | cloud_to_cloud | node | csharpcoreclr_server | large_unary | TIMEOUT | 2017-10-31 16:35:16 UTC | 183.01 |  
24 | grpc/ubuntu/master/grpc_interop_tocloud | 70 | null | cloud_to_cloud:node:csharp_server:client_streaming | cloud_to_cloud | node | csharp_server | client_streaming | TIMEOUT | 2017-10-31 16:35:11 UTC | 188.27 |  
25 | grpc/ubuntu/master/grpc_interop_tocloud | 70 | null | cloud_to_cloud:node:python_server:large_unary | cloud_to_cloud | node | python_server | large_unary | TIMEOUT | 2017-10-31 16:35:00 UTC | 187.33 |  
26 | grpc/ubuntu/master/grpc_interop_tocloud | 70 | null | cloud_to_cloud:node:csharpcoreclr_server:client_streaming | cloud_to_cloud | node | csharpcoreclr_server | client_streaming | TIMEOUT | 2017-10-31 16:35:21 UTC | 182.79 |  
27 | grpc/ubuntu/master/grpc_interop_tocloud | 69 | null | cloud_to_cloud:node:csharp_server:large_unary | cloud_to_cloud | node | csharp_server | large_unary | TIMEOUT | 2017-10-31 13:34:46 UTC | 183.74 |  
28 | grpc/ubuntu/master/grpc_interop_tocloud | 67 | null | cloud_to_cloud:node:ruby_server:client_streaming | cloud_to_cloud | node | ruby_server | client_streaming | TIMEOUT | 2017-10-30 22:35:24 UTC | 186.33 |  

the query

select * from jenkins_test_results.interop_results where test_name like '%' and job_name like '%grpc_interop_tocloud%' and timestamp > '2017-10-25 00:00' and result != 'PASSED'  order by build_id desc limit 10000

This was referenced Nov 6, 2017
Closed
Closed
Closed
Closed
Closed
@ZhouyihaiDing ZhouyihaiDing mentioned this issue Nov 6, 2017
Merged
This was referenced Nov 7, 2017
Merged
Merged
@murgatroid99 murgatroid99 mentioned this issue Nov 7, 2017
Merged
@kpayson64 kpayson64 mentioned this issue Nov 7, 2017
Merged
@sreecha sreecha mentioned this issue Nov 9, 2017
Merged
@yang-g yang-g mentioned this issue Nov 9, 2017
Merged
@sreecha sreecha mentioned this issue Nov 9, 2017
Merged
@dgquintas
Copy link
Contributor Author

Ping. This happens consistently.

@dgquintas dgquintas mentioned this issue Nov 10, 2017
Merged
@yang-g yang-g mentioned this issue Nov 10, 2017
Merged
@sreecha sreecha mentioned this issue Nov 13, 2017
Merged
@dgquintas
Copy link
Contributor Author

Ping. Any update?

@dgquintas dgquintas mentioned this issue Nov 14, 2017
Merged
@jtattermusch
Copy link
Contributor

jtattermusch commented Nov 15, 2017 •
edited
Loading

Another issue that started affecting node (looks like all node interop tests are failing).

Failure
/var/local/git/grpc-node/test/any_grpc.js:9
    throw new Error([
    ^

Error: Invalid value for global._client_implementation: undefined. If running from the command line, please --require a fixture first.
    at getImplementation (/var/local/git/grpc-node/test/any_grpc.js:9:11)
    at Object.<anonymous> (/var/local/git/grpc-node/test/any_grpc.js:23:20)
    at Module._compile (module.js:624:30)
    at Object.Module._extensions..js (module.js:635:10)
    at Module.load (module.js:545:32)
    at tryModuleLoad (module.js:508:12)
    at Function.Module._load (module.js:500:3)
    at Module.require (module.js:568:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/var/local/git/grpc-node/test/interop/interop_client.js:23:12)

https://sponge.corp.google.com/target?id=90b8ad12-58d3-4099-8fc7-970e4a7ffea6&target=github/grpc&searchFor=&show=ALL&sortBy=STATUS

@jtattermusch
Copy link
Contributor

That issue has been already filed as grpc/grpc-node#89

@dgquintas
Copy link
Contributor Author

@nicolasnoble and I talked offline about this yesterday. The plan is to have these be disabled while being investigated. @nicolasnoble can you confirm?

@markdroth markdroth mentioned this issue Nov 15, 2017
Merged
@murgatroid99
Copy link
Member

The fix for that latest issue is #13394.

This was referenced Nov 17, 2017
Merged
Merged
@murgatroid99
Copy link
Member

This should be fixed by grpc/grpc-node#95.

@jtattermusch jtattermusch mentioned this issue Nov 21, 2017
Merged
@lock lock bot locked as resolved and limited conversation to collaborators Sep 30, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@murgatroid99 murgatroid99

Labels
disposition/BUILDNURSE For all buildnurse related build/test failures and flakes infra/New Failure priority/P1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dgquintas @murgatroid99 @jtattermusch @adelez