@mpj1234 what's your batch size? Reducing that might help.

The bug of OOM is batch size = 120.

Now it is batch size = 100, and eight epoch have been run, so there is no problem for the time being.

👋 Hello! Thanks for asking about CUDA memory issues. YOLOv5/v8 🚀 can be trained on CPU, single-GPU, or multi-GPU. When training on GPU it is important to keep your batch-size small enough that you do not use all of your GPU memory, otherwise you will see a CUDA Out Of Memory (OOM) Error and your training will crash. You can observe your CUDA memory utilization using either the nvidia-smi command or by viewing your console output:

CUDA Out of Memory Solutions

If you encounter a CUDA OOM error, the steps you can take to reduce your memory usage are:

• Reduce --batch-size
• Reduce --img-size
• Reduce model size, i.e. from YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s > YOLOv5n
• Train with multi-GPU at the same --batch-size
• Upgrade your hardware to a larger GPU
• Train on free GPU backends with up to 16GB of CUDA memory:

AutoBatch

You can use YOLOv5 AutoBatch (NEW) to find the best batch size for your training by passing --batch-size -1. AutoBatch will solve for a 90% CUDA memory-utilization batch-size given your training settings. AutoBatch is experimental, and only works for Single-GPU training. It may not work on all systems, and is not recommended for production use.

Good luck 🍀 and let us know if you have any other questions!

Yesterday, after I switched batch size=100, I ran 67 epochs normally, but at 68epoch, I suddenly allocated a lot of memory, and OOM appeared. I think there may be memory leakage in the code.

I'm switching to automatic batch now, and I'm still experimenting. The automatic batch allocation is 151.

I hope if it's not a bug, explain why after training so many epoch, I suddenly have to allocate a lot of video memory.

This is the automatic batch, and it is still OOM.
I'm surprised that epoch in front is fine, but OOM in the back suddenly. Is there an explanation for this?

@mpj1234 hi, looks the number of instances in your dataset is variable. The memory is instances-related, the more instances you got the more memory would be occupied. So the OOM could happen when you got more instances for a sudden in one batch. You have to reduce batch-size or use small model to solve this OOM issue.

ok, Thanks♪(･ω･)ﾉ

for me the following things seem to work:

1. reduce the batch size
2. call torch.cuda.empty_cache() before training job, some frameworks do this automatically.
3. os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

@Petros626 thank you for sharing these suggestions! Reducing batch size and using torch.cuda.empty_cache() are effective, and setting PYTORCH_CUDA_ALLOC_CONF can help manage memory fragmentation. These approaches align well with best practices for resolving CUDA OOM issues.