[utils] add synchronized cuda memory monitor #740
Conversation
| def finish(self): | ||
| torch.cuda.synchronize() | ||
| self.time_stamps.append(time()) | ||
| max_usage = torch.cuda.max_memory_allocated() |
There was a problem hiding this comment.
reserved?
pytorch/pytorch#40989
There was a problem hiding this comment.
Pytorch uses a caching allocator.
max_memory_allocated returns maximum cuda memory used by tensors.
max_memory_reserved returns maximum cuda memory used by the allocator.
If user still has enough memory in GPU, caching allocator may not recycle unused segments.
So the reserved memory can be much bigger than allocated memory. This will bring inaccuracy when we trying
to know maximum memory used by all tensors.
But there is a possiblity that the maximum cuda memory used is bigger than memory allocated.
We are better to set the maximum cuda memory slightly smaller than the actual cuda memory.
So there is some memory to place buffer for Pytorch.
No description provided.