I have made some minor improvements:

• README tweaks
• add support for multi GPU on pytorch cifar100 and food101. I am still learning ML its the first time I used Pytorch so hopefully this is already acceptable enough.
• test results for
  • M1 Pro (10 GPU + 16 GPU) with 16GB Ram: Very similar results to your 32GB, most slower slightly and can't run bigger batches. Interesting to see on Pytorch NLP at 128 batch size it was swapping a lot, however the additional 2GB ram of M3 Pro 18GB was enough for squeeze that run. Was slightly faster on pytorch cifar100, however I did see run slower at times significantly. Maybe I was running a background task, I rerun tensorflow food101 as my first run was 2,000 secs.. unexpected.
  • Nvidia 1070 4 cards. Old miner i3-7100 and 16GB ram. I revived my friend's old miner, the PSU wasn't enough for most benchmarks. Interesting though on pytorch cifar100 it was fastest on 3072 batch size and also beat single card numbers. However this is real evidence that multi-GPU doesn't scale well.... worse on the other incomplete benchmarks.