TensorRT Memory Leak On One System And Not Anouther

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I am dealing with some issues where the same executable, config files, model, and dlls work have a constant memory usage on one system but not another. In particular one stays around 1GB while the other continuously goes up to above the 16GB. This same system did not have issues a month ago running the same thing. But it showed up once I put a fresh OS on it.

Environment

TensorRT Version: 6.0.1.5
GPU Type: NVidia RTX 5000 and NVidia P5200
Nvidia Driver Version: Checked with Various 419, and 431
CUDA Version: 10.0.130
CUDNN Version: 7.6.3.30
Operating System + Version: Windows 10
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): Baremetal

2

Hi @curiousguy,
Request you to share the profiler and performance output for both the machines.
Also, i could see you are using TRT 6, for that we recommend you to try the latest TRT release, as this will have improved performance.

Thanks!

3

@AakankshaS Due to the environment, I do not think I can share this information. Is there something I can look for in particular?

4

Hi @curiousguy,
Can you please try working with latest TRT release.

Thanks!