Hi there!
I have a question regarding GPU memory management: When memory is frequently allocated, it can lead to memory fragmentation. As a result, even though there appears to be available memory in total, subsequent calls to cudaMalloc fail to allocate memory successfully.
Is there a method to quickly reclaim or defragment the fragmented GPU memory to resolve this issue?
cuMemMap and cuMemSetAccess should be able to combine multiple small arrays as one contiguous virtual array. Handles are created with cuMemAddressReserve and cuMemCreate.
CUDA Driver API :: CUDA Toolkit Documentation
If you haven’t already investigated it, the Stream Ordered Allocator may be worth a look.
In a scenario where there are only memory allocations, without any deallocation, fragmentation is typically a minor issue only (e.g. due to alignment constraints). It can start to play a major role when allocations are mixed with deallocations, especially when there is a “random mix” of allocation and deallocation.
One way of largely avoiding fragmentation issues is to minimize deallocations and re-use existing allocations as much as feasible. Deallocation in reverse order of allocation is another way of minimizing fragmentation issues.
To my knowledge there are no established defragmenters in use for either CPUs or GPUs, as this typically involves large amount of data movement, with a resulting impact on available memory bandwidth. If you still remember rotational mass storage (hard disks), the amount of copying necessary was obvious from how long it took to (partially) defragment just one disk.
Thank you for your detailed explanation!
The methods you mentioned for reducing memory fragmentation are very insightful, especially minimizing deallocations, reusing existing allocations, and deallocating in reverse order of allocation. These practices can indeed alleviate GPU memory fragmentation issues to a large extent.
This helps me a lot.
Thank you for the suggestion! I haven’t looked into the Stream Ordered Allocator yet, but I’ll definitely check it out. It sounds like a promising approach to explore for better GPU memory management. I appreciate you pointing this out—this could be really helpful for addressing the memory fragmentation issues we’ve been dealing with.
Thank you for sharing this approach using the CUDA Driver API!
I’ll definitely dive into the CUDA Driver API documentation you linked to explore how to implement this. This could be a valuable technique for our use case, and I appreciate you taking the time to point me in this direction!