Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: checkpoint loading error with those saving with DDP #768

Closed
Gy-Lu opened this issue Apr 14, 2022 · 0 comments · Fixed by #769
Closed

[BUG]: checkpoint loading error with those saving with DDP #768

Gy-Lu opened this issue Apr 14, 2022 · 0 comments · Fixed by #769
Labels
bug Something isn't working

Comments

@Gy-Lu
Copy link
Contributor

Gy-Lu commented Apr 14, 2022

🐛 Describe the bug

When saving the checkpoint with a DDP model with the hook and loading with a pipeline model, the load_checkpoint can't load the ckpt correctly.

I found that the state_dict of a DDP model contains module. as the prefix. Therefore, the loading needs to be handled specially.

Environment

No response
@Gy-Lu Gy-Lu added the bug Something isn't working label Apr 14, 2022
@Gy-Lu Gy-Lu linked a pull request Apr 14, 2022 that will close this issue
[hotfix] fix the ckpt hook bugs when using DDP #769
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[hotfix] fix the ckpt hook bugs when using DDP
1 participant
@Gy-Lu