Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI test windows://python/ray/tests:test_actor_retry is flaky #43845

Open
can-anyscale opened this issue Mar 9, 2024 · 91 comments
Open

CI test windows://python/ray/tests:test_actor_retry is flaky #43845

can-anyscale opened this issue Mar 9, 2024 · 91 comments
Labels
bug Something that is supposed to be working; but isn't ci-test core Issues that should be addressed in Ray Core flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ P2 Important issue, but not time-critical ray-test-bot Issues managed by OSS test policy stability windows

Comments

@can-anyscale
Copy link
Collaborator

CI test windows://python/ray/tests:test_actor_retry is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/3389#018e1ff8-1398-4a05-a8b7-43413ef55a8a
- https://buildkite.com/ray-project/postmerge/builds/3383#018e1f7f-0d95-479c-9005-452d67ed8401
- https://buildkite.com/ray-project/postmerge/builds/3379#018e1ecc-ee34-4d64-a6f4-d1138f0ace37
- https://buildkite.com/ray-project/postmerge/builds/3368#018e1c3d-c90d-4558-8d6f-5325b1259500
- https://buildkite.com/ray-project/postmerge/builds/3308#018e120f-a1ab-44f7-9453-b52795157a13

DataCaseName-windows://python/ray/tests:test_actor_retry-END
Managed by OSS Test Policy
@can-anyscale can-anyscale added bug Something that is supposed to be working; but isn't ci-test core Issues that should be addressed in Ray Core flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ ray-test-bot Issues managed by OSS test policy stability triage Needs triage (eg: priority, bug/not-bug, and owning component) weekly-release-blocker Issues that will be blocking Ray weekly releases and removed weekly-release-blocker Issues that will be blocking Ray weekly releases labels Mar 9, 2024
@jjyao jjyao added P2 Important issue, but not time-critical windows and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Mar 11, 2024
@anyscalesam anyscalesam added P1 Issue that should be fixed within a few weeks and removed P2 Important issue, but not time-critical labels Mar 13, 2024
@anyscalesam anyscalesam added P2 Important issue, but not time-critical and removed P1 Issue that should be fixed within a few weeks labels Mar 14, 2024
@anyscalesam
Copy link
Contributor

p2 as it doesn't block weekly or release for ray210; but @mattip please continue prioritizing

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/3600#018e53cd-4b89-433f-8dba-cfd9ecb177fe

@can-anyscale
Copy link
Collaborator Author

CI test windows://python/ray/tests:test_actor_retry is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/3651#018e5f6b-db09-47af-ba45-de1d203bca2b
- https://buildkite.com/ray-project/postmerge/builds/3517#018e3ebc-79cc-45ef-a6fd-9a694adc4e40
- https://buildkite.com/ray-project/postmerge/builds/3506#018e3c69-8eb5-4d34-b0bf-d9a9bab35987
- https://buildkite.com/ray-project/postmerge/builds/3438#018e321c-f9d7-48f1-bb85-50db9bd7d508
- https://buildkite.com/ray-project/postmerge/builds/3415#018e2beb-f879-45e8-82b7-ba7138f66730

DataCaseName-windows://python/ray/tests:test_actor_retry-END
Managed by OSS Test Policy

@can-anyscale can-anyscale reopened this Mar 21, 2024
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/3747#018e806e-bca9-4556-8869-80c3a431915d

@can-anyscale can-anyscale changed the title CI test windows://python/ray/tests:test_actor_retry is flaky CI test windows://python/ray/tests:test_actor_retry is consistently_failing Jun 14, 2024
@can-anyscale can-anyscale reopened this Jun 14, 2024
@can-anyscale
Copy link
Collaborator Author

CI test windows://python/ray/tests:test_actor_retry is consistently_failing. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4923#0190178e-5173-477b-a1b6-75b831f139c4
- https://buildkite.com/ray-project/postmerge/builds/4923#0190178e-5173-477b-a1b6-75b831f139c4
- https://buildkite.com/ray-project/postmerge/builds/4895#01901395-190c-4ef7-aa0f-b9e1419790de
- https://buildkite.com/ray-project/postmerge/builds/4811#019003de-4fb0-4249-9030-6b7f545369b3

DataCaseName-windows://python/ray/tests:test_actor_retry-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4953#01901a4f-5320-4166-88d9-481caddeed15

@can-anyscale
Copy link
Collaborator Author

CI test windows://python/ray/tests:test_actor_retry is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/5979#01917bc4-e34d-41a5-a162-7534e8c859b8
- https://buildkite.com/ray-project/postmerge/builds/5977#01917bae-bb5c-4412-9383-b4768f0e3da9
- https://buildkite.com/ray-project/postmerge/builds/5967#0191787c-409a-4537-907d-1a241a057aff

DataCaseName-windows://python/ray/tests:test_actor_retry-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/5991#01917ffd-f4fd-4af5-a30d-1bd017b53c32

@can-anyscale can-anyscale reopened this Aug 23, 2024
@can-anyscale
Copy link
Collaborator Author

CI test windows://python/ray/tests:test_actor_retry is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/5990#01917f6d-d651-4cf1-91f4-853bddd9cb5a
- https://buildkite.com/ray-project/postmerge/builds/5986#01917e25-44b2-4925-bb2f-1a9fec1c8e8b
- https://buildkite.com/ray-project/postmerge/builds/5986#01917c91-6fd3-4389-9674-a7e6375c8ebb
- https://buildkite.com/ray-project/postmerge/builds/5982#01917cc1-bd98-4dfb-b7b2-587873a46879
- https://buildkite.com/ray-project/postmerge/builds/5985#01917c65-03b6-43b3-8a25-1a0a859ffcb6

DataCaseName-windows://python/ray/tests:test_actor_retry-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/6001#019181bc-9f1e-4eed-9151-9ab4cf42ba8a

@can-anyscale can-anyscale reopened this Aug 26, 2024
@can-anyscale
Copy link
Collaborator Author

CI test windows://python/ray/tests:test_actor_retry is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/6004#019182c8-b4e8-4b23-8022-581199b8aa3a
- https://buildkite.com/ray-project/postmerge/builds/5990#01917f6d-d651-4cf1-91f4-853bddd9cb5a
- https://buildkite.com/ray-project/postmerge/builds/5986#01917e25-44b2-4925-bb2f-1a9fec1c8e8b

DataCaseName-windows://python/ray/tests:test_actor_retry-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/6014#01919066-6c99-4828-a5d1-7b07e13015ed

@can-anyscale can-anyscale reopened this Aug 27, 2024
@can-anyscale
Copy link
Collaborator Author

CI test windows://python/ray/tests:test_actor_retry is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/6022#01919388-7046-4102-a8d5-88d49afec6aa
- https://buildkite.com/ray-project/postmerge/builds/6022#01919286-bc62-45a6-9d01-7727a99ea365
- https://buildkite.com/ray-project/postmerge/builds/6020#0191923d-4946-4eb3-afb7-ba5a07c893d4

DataCaseName-windows://python/ray/tests:test_actor_retry-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/6036#01919651-67ce-46ce-a13b-e9e02743079a

@can-anyscale can-anyscale reopened this Aug 29, 2024
@can-anyscale
Copy link
Collaborator Author

CI test windows://python/ray/tests:test_actor_retry is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/6053#01919c88-eb1f-49a8-bfa4-84f6f544c90c
- https://buildkite.com/ray-project/postmerge/builds/6048#01919b48-d02f-45a8-b490-a4afd4662ee7
- https://buildkite.com/ray-project/postmerge/builds/6048#01919a9b-f9de-4f9e-8616-51ebd67cefd6

DataCaseName-windows://python/ray/tests:test_actor_retry-END
Managed by OSS Test Policy

aslonnie pushed a commit that referenced this issue Aug 29, 2024
@mattip
Split python/ray/tests/test_actor_retry over two files (#47188)
f0f29bd 
The `test_actor_retry` tests are failing/flaky on windows. They pass locally. I have not been able to access the CI logs to see what is going wrong. In order to shrink the problem (is it a overall timeout? Is one of the tests failing?) we can start by splitting the tests into two files.

Toward solving #43845.
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this issue Oct 12, 2024
@mattip @ujjawal-khare
Split python/ray/tests/test_actor_retry over two files (ray-project#4…
3557f98 
…7188)

The `test_actor_retry` tests are failing/flaky on windows. They pass locally. I have not been able to access the CI logs to see what is going wrong. In order to shrink the problem (is it a overall timeout? Is one of the tests failing?) we can start by splitting the tests into two files.

Toward solving ray-project#43845.

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this issue Oct 15, 2024
@mattip @ujjawal-khare
Split python/ray/tests/test_actor_retry over two files (ray-project#4…
5376479 
…7188)

The `test_actor_retry` tests are failing/flaky on windows. They pass locally. I have not been able to access the CI logs to see what is going wrong. In order to shrink the problem (is it a overall timeout? Is one of the tests failing?) we can start by splitting the tests into two files.

Toward solving ray-project#43845.

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this issue Oct 15, 2024
@mattip @ujjawal-khare
Split python/ray/tests/test_actor_retry over two files (ray-project#4…
90cc33d 
…7188)

The `test_actor_retry` tests are failing/flaky on windows. They pass locally. I have not been able to access the CI logs to see what is going wrong. In order to shrink the problem (is it a overall timeout? Is one of the tests failing?) we can start by splitting the tests into two files.

Toward solving ray-project#43845.

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this issue Oct 15, 2024
@mattip @ujjawal-khare
Split python/ray/tests/test_actor_retry over two files (ray-project#4…
063d004 
…7188)

The `test_actor_retry` tests are failing/flaky on windows. They pass locally. I have not been able to access the CI logs to see what is going wrong. In order to shrink the problem (is it a overall timeout? Is one of the tests failing?) we can start by splitting the tests into two files.

Toward solving ray-project#43845.

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this issue Oct 15, 2024
@mattip @ujjawal-khare
Split python/ray/tests/test_actor_retry over two files (ray-project#4…
9f76655 
…7188)

The `test_actor_retry` tests are failing/flaky on windows. They pass locally. I have not been able to access the CI logs to see what is going wrong. In order to shrink the problem (is it a overall timeout? Is one of the tests failing?) we can start by splitting the tests into two files.

Toward solving ray-project#43845.

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
@jjyao jjyao added P2 Important issue, but not time-critical and removed P1 Issue that should be fixed within a few weeks labels Oct 30, 2024
@mattip mattip removed their assignment Dec 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something that is supposed to be working; but isn't ci-test core Issues that should be addressed in Ray Core flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ P2 Important issue, but not time-critical ray-test-bot Issues managed by OSS test policy stability windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mattip @jjyao @anyscalesam @can-anyscale