NVIDIA Megatron Core 0.7.0

ericharper released this 05 Jun 23:12

· 1564 commits to main since this release

a645f89

MoE
- Token drop support
- Several efficiency optimizations
- Improved model parallelism
- Memory optimizations
Distributed checkpointing
- Enabled for Retro
- Asynchronous checkpoint saving
Several minor bug fixes, speed improvements, and memory optimizations

Assets 2