Merge branch 'github-main' into 'main'

Bringing github main and internal main back together.

See merge request ADLR/megatron-lm!275

Merge branch 'main' into github-main

Cherry-pick fix from development repo.

Use torch.cuda.synchronize() right after calling batch_isend_irecv() …

…communication API

Merge branch 'TOC-fix' into 'main'

TOC fix

See merge request ADLR/megatron-lm!208

Merge branch 'readme_update' into 'main'

Update scaling numbers in README and other small tweaks.

See merge request ADLR/megatron-lm!130

Updated with minor fix to l2 norm computation.

Merge branch 'torchddp' into 'master'

Use DDP command line argument instead of source flag in pretrain_bert.py.

See merge request ADLR/megatron-lm!2

initial commit