Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RayJob][Status][9/n] RayJob should not pass any changes to RayCluster #1776

Merged
merged 2 commits into from
Dec 27, 2023

Conversation

kevin85421
Copy link
Member

@kevin85421 kevin85421 commented Dec 27, 2023

Why are these changes needed?

Without this PR, RayJob has a complex behavior to pass changes from RayJob's spec to the associated RayCluster.

  • If Ray Autoscaling is enabled, do not pass any changes to the associated RayCluster.
  • If the change concerns only replicas, update the RayCluster.
  • Otherwise, do not update the RayCluster.

This PR decides to remove the behavior because:

  • For batch jobs, the required computation resources should be deterministic. Hence, no need to pass changes to RayCluster.
  • If users still wish to update replicas at runtime, they should use the Ray Autoscaler instead.
  • If users still want to update replicas manually, they still can update RayCluster directly.
  • Allowing users to update replicas at runtime will make gang scheduling much more complex.
  • The best practice for batch jobs should include the ability to tolerate downtime and implement application-level fault tolerance within the jobs themselves, such as writing and reading checkpoints from S3.

Hence, I remove this behavior in this PR to avoid unnecessary complexity of the RayJob codebase.

Related issue number

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

@kevin85421 kevin85421 changed the title WIP [RayJob] RayJob should not pass any changes to RayCluster Dec 27, 2023
@kevin85421 kevin85421 changed the title [RayJob] RayJob should not pass any changes to RayCluster [RayJob][Status][9/n] RayJob should not pass any changes to RayCluster Dec 27, 2023
@kevin85421 kevin85421 marked this pull request as ready for review December 27, 2023 06:56
Copy link
Contributor

@architkulkarni architkulkarni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good, I agree with the reasoning.

Should we print some kind of warning message if we detect the user is trying to make a change to the RayCluster in the RayJob spec? If we don't print a message, the user might interpret it as "silently failing" which isn't a good experience.

@kevin85421
Copy link
Member Author

Should we print some kind of warning message if we detect the user is trying to make a change to the RayCluster in the RayJob spec? If we don't print a message, the user might interpret it as "silently failing" which isn't a good experience.

Good point! I will open an issue, and @Yicheng-Lu-llll will work on this.

@kevin85421 kevin85421 merged commit 6af407d into ray-project:master Dec 27, 2023
25 checks passed
@kevin85421
Copy link
Member Author

#1777

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants