context->setBindingDimensions Would case gpu memory leak.

Just got an answer, they caught this in a similar bug and have fixed it for the next release. Thanks for pointing this out.

Hi, Can you provide the full .cpp script for this small example? Thanks, NVIDIA Enterprise Support

You could just use any *.engine file with dynamic/fixed input size, and use the following code to find that the gpu memory usage is always rising. class MyArray { public: int inp_size{0}; int out_size{0}; int inp_bytes{0}; int out_bytes{0}; int time_step{…

Case 1: for(int i=0; i < 1000000; ++i) { context->setBindingDimensions context->enqueueV2(buffers, stream, nullptr); } Case 2: for(int i=0; i < 1000000; ++i) { context->enqueueV2(buffers, stream, nullptr); } Case 3: for(int i=0; i < 1000000; ++i) { context->setBindingDimensions …

Is the fixed version released yet ?

Hi yfjiaren, Not yet. Sorry, but I can’t release the timeline.

[image] TensorRT6 Dynamic Input Size is slow and has gpu memory leak. context->setBindingDimensions The above interface would cause slow performance and gpu memory leak. Gpu memory leak is fixed on TensorRT7, but, slow performance is Same as before.

Hi yfjiaren, TRT7 has been released and should fix the memory leak issue.

I found tensorRT7 also has this problem. I use python api and every time I call context.set_binding_shape and context.execute_async_v2, the gpu memory grows utill out of memory. But it didn’t happen to one of my other simple network.

[image] TensorRT6 Dynamic Input Size is slow and has gpu memory leak. context->setBindingDimensions The above interface would cause slow performance and gpu memory leak. Gpu memory leak is fixed on TensorRT7, but, slow performance is Same as before. I use…

@NVES_R , @yfjiaren , @nilshinn I found tensorRT7 also has this problem. I use python api and every time I call context.set_binding_shape and context.execute_async_v2, the gpu memory grows utill out of memory. But it didn’t happen to one of my other simple network. Indeed, ResNet-like heavy net…

context->setBindingDimensions casing gpu memory leak

AI & Data Science Deep Learning (Training & Inference) TensorRT

tamirg September 8, 2020, 8:36am 21

@NVES_R any updates? this bug is causing a severe memory leak when using triton server.

Topic		Replies	Views
TensorRT6 Dynamic Input Size is slow and has gpu memory leak. TensorRT	1	1025	October 18, 2019
TensorRt 6 - IExecutionContext->execute cause GPU memory leak TensorRT	8	1713	November 4, 2019
GPU memory leak - With dynamic batch models Triton Inference Server (archived)	0	666	September 9, 2020
Questions about frequently changing the batch size TensorRT tensorrt	1	574	March 31, 2021
Dynamic engine building and optimization profile TensorRT	4	2099	July 18, 2023
TensorRT: Cannot set bindings for dynamic shapes TensorRT	3	5796	June 21, 2021
Direct GPU Inference TensorRT	6	2299	May 26, 2020
check allInputDimensionsSpecified() for second profile fail TensorRT	8	5656	June 24, 2020
Memory leak in IExecutionContext TRT6 TensorRT	1	1333	March 2, 2020
TensorRT 6 memory leak Jetson AGX Xavier tensorrt	11	2255	August 19, 2020

context->setBindingDimensions casing gpu memory leak

Related topics