hm.. are you looking at the ema checkpoint? at 0.9999 ema it can take more than 10000 steps to see a change

this code implements data parallel training. Each gpu should be doing the same thing just with different data. Larger batch sizes allows for larger learning rates, so you should increase the learning rate to see a change. When training using fp16 the multiplier should be about 20 after warmup. If it's lower than that the learning rate is likely too high, if it's higher it's likely too low.

hm.. are you looking at the ema checkpoint? at 0.9999 ema it can take more than 10000 steps to see a change

this code implements data parallel training. Each gpu should be doing the same thing just with different data. Larger batch sizes allows for larger learning rates, so you should increase the learning rate to see a change. When training using fp16 the multiplier should be about 20 after warmup. If it's lower than that the learning rate is likely too high, if it's higher it's likely too low.

Thank you for the reply, this clarifies things, I'll increase my dataset size and give this another go.

Confirmed that everything works as expected and issue was with the size of my dataset.

Confirmed that everything works as expected and issue was with the size of my dataset.

How large was it, just wondering