In my application, I load multiple TensorRT engines and depending on certain conditions I use one of them. I wish to reduce memory consumption.
I see that I can use createExecutionContextWithoutDeviceMemory to allocate memory at a latter time but I’m not seeing an option to deallocate the assigned memory. The only option I’m seeing is to delete the current context and create a new one using the function mentioned above.
Is this the appropriate method and if not what alternative method should I follow?
Thanks for the reply!
Sorry, but as I understand it’s not specifically a Jetson issue but related to TensorRT itself.
The same issue will come on other devices with limited RAM.
The main issue is that I’m not able to find an API or class members that I could free to freeup some memory.
the function: createExecutionContextWithoutDeviceMemory allows to allocate memory when we want but I couldn’t find a function to freeup the used up memory.
I believe you would use cudaMalloc and cudaFree. You can pass the allocated memory using IExecutionContex::setDeviceMemory(). And you can get the size of memory needed for malloc using ICudaEngine::getDeviceMemorySize().
Thanks a lot. I had mistaken that this function would allocate the memory itself but on checking the API reference I can see that we provide it the required memory block.
Thanks.
I have one further question:
The description of the above mentioned function says: The memory must be aligned with cuda memory alignment property. Would a memory block allocated using cudaMalloc (and size specified using getDeviceMemorySize()) satisfy the condition. I’m new to working with cuda so sorry if i’m asking something basic.
I believe cudaMalloc would give you the correct alignment (I think that would be cruel if it didn’t, and they didn’t provide any more information on how to align it properly). But do not, for example, cudaMalloc a large block of memory (p) and pass in an arbitrary sub block of that memory (p+offset), unless you ensure that offset is a multiple of the proper alignment. To be honest, I’m not sure exactly what alignment requirement they are referring to, CUDA Runtime API :: CUDA Toolkit Documentation cudaMalloc documentation just says “The allocated memory is suitably aligned for any kind of variable”