I'm thinking it might be safer to set it to a different value per epoch, e.g. 42 + epoch? One of my concerns is that it might interact badly with lhotse, which might rely on the seed being different for each epoch. But also on general principles.

Thanks! It is fixed now.

Cool!

This reminds me, we may want to ensure that rng used for data augmentation in dataloading workers is seeded differently per worker, otherwise it might be applying the same augmentations. Dynamic batch sizes probably partially amend that though.

The samplers instantiate their own rng which they seed at the beginning of each epoch so changing global settings won’t affect them.

is dataloading done in main process?
could maybe do " + 100 * rank" or something

no, in a multi-GPU training scheme, there are (at least) three levels of subprocesses:

1. top-level script that spawns one subproc per GPU
2. per-GPU process, spawns X subprocesses for dataloading and runs the sampler and the training loop
3. X dataloading worker subprocesses for each per-GPU process, each runs I/O, augmentation, and collation

probably sth like "+ 100 * rank + worker_idx" would work yeah -- I'll check it out eventually