I confirm that this change fixes the problem. Could we please merge it asap, since other Deepspeed PRs are afflicted as well.

Thank you!

cc: @tjruwase + @jeffra to review please. and please consider a new release to undo the breakage.

p.s. The problem manifests in distributed Accelerate. Not sure how easy it'd be to write a test since Accelerate isn't easy to set up.

p.s. The problem manifests in distributed Accelerate. Not sure how easy it'd be to write a test since Accelerate isn't easy to set up.

Do you have a minimal example we could add to our tests to ensure we don't break this in the future?

As I'm not on Accelerate team, I'm just its user, let's ping @pacman100 who might be able to help with contributing such test.

But since it could take days it's far more urgent to merge your fix first.

Thank you.

Thank you for the quick fix and merge, Michael!

The only remaining step is a new minor release, since many users will not know to install from master.

Thank you for the quick fix and merge, Michael!

The only remaining step is a new minor release, since many users will not know to install from master.

@stas00 v0.9.1 is now live:
https://pypi.org/project/deepspeed/0.9.1/
https://github.com/microsoft/DeepSpeed/releases/tag/v0.9.1

Amazing! Thank you, Jeff!

As I'm not on Accelerate team, I'm just its user, let's ping @pacman100 who might be able to help with contributing such test.

Hello, as this issue only comes up when using multi-gpu/multi-node setup, all the current slow tests in accelerate are failing (see the screenshots below). I don't know if DeepSpeed runs these slow tests on their front.

Commands to run the slow tests from accelerate main folder on a node with 2 gpus

export RUN_SLOW=true
python -m pytest -s -v ./tests/deepspeed