Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release test torch_batch_inference_hetero_10tb_parquet failed #41390

Closed
can-anyscale opened this issue Nov 27, 2023 · 7 comments · Fixed by #41712
Closed

Release test torch_batch_inference_hetero_10tb_parquet failed #41390

can-anyscale opened this issue Nov 27, 2023 · 7 comments · Fixed by #41712
Assignees
@c21 @bveeramani
Labels
bug Something that is supposed to be working; but isn't data Ray Data-related issues P0 Issues that should be fixed in short order ray 2.9 Issues targeting Ray 2.9 release (~Q4 CY2023) release-blocker P0 Issue that blocks the release release-test release test

Comments

@can-anyscale
Copy link
Collaborator

Release test torch_batch_inference_hetero_10tb_parquet failed. See https://buildkite.com/ray-project/release/builds/2173#018c0eac-2b4a-4d0d-91dd-909fac1e9275 for more details.

Managed by OSS Test Policy
@can-anyscale can-anyscale added bug Something that is supposed to be working; but isn't data Ray Data-related issues P0 Issues that should be fixed in short order release-test release test labels Nov 27, 2023
@can-anyscale can-anyscale assigned can-anyscale and c21 and unassigned can-anyscale Nov 27, 2023
@anyscalesam anyscalesam added release-blocker P0 Issue that blocks the release ray 2.9 Issues targeting Ray 2.9 release (~Q4 CY2023) labels Nov 28, 2023
@scottjlee scottjlee self-assigned this Nov 29, 2023
@anyscalesam
Copy link
Contributor

@scottjlee did you get a chance to look into this yesterday? Do we have a root cause on why this test is failing?

@scottjlee
Copy link
Contributor

@anyscalesam not yet, have some other work i am trying to get in before branch cut. talked to Cheng and he said we can investigate this after branch cut

@can-anyscale
Copy link
Collaborator Author

Test has been failing for far too long. Jailing.

@can-anyscale can-anyscale added the jailed-test Indicate that this test is jailed and will stop running in CI/CD label Dec 2, 2023
@bveeramani bveeramani assigned bveeramani and unassigned scottjlee Dec 7, 2023
@bveeramani
Copy link
Member

I have a hunch as to why this test is failing. Opening a PR to test a fix now.

@bveeramani bveeramani mentioned this issue Dec 7, 2023
Merged
8 tasks
@bveeramani bveeramani mentioned this issue Dec 8, 2023
Merged
8 tasks
@scottjlee scottjlee removed the jailed-test Indicate that this test is jailed and will stop running in CI/CD label Dec 8, 2023
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/release/builds/3680#018c4ed6-e5b0-4b7c-ab50-11bcc48a95ef

@anyscalesam
Copy link
Contributor

@can-anyscale I think you're already tracking this but another example where a quick search on go/flaky brings up no recent runs of this test. How does one verify that the test is still passing?

@can-anyscale
Copy link
Collaborator Author

They should show up on go/flaky once I fix it in Jan. Release test results are broken on go/flaky.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@c21 c21

@bveeramani bveeramani

Labels
bug Something that is supposed to be working; but isn't data Ray Data-related issues P0 Issues that should be fixed in short order ray 2.9 Issues targeting Ray 2.9 release (~Q4 CY2023) release-blocker P0 Issue that blocks the release release-test release test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Data] Avoid serializing datasource for Parquet read tasks bveeramani/ray
5 participants
@c21 @scottjlee @bveeramani @anyscalesam @can-anyscale