Valgrind reports memory leaks on CUDA library

Software versions:
JetPack: 5.0.1
CUDA: 11.4
TensorRT: 8.4.0.11
CuDNN: 8.3.2.49
Valgrine: 3.19.0

Problem description:
We run a valgrine memory leak check on trtexec the MINST onnx model installed with TensorRT samples, with the following commands.

valgrind --tool=memcheck --leak-check=full -- trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx

The valgrind reports 468 bytes of definitely lost.

The call stacks is as follows:

==353287== 20 bytes in 1 blocks are definitely lost in loss record 82 of 4,025
==353287==    at 0x484B0C8: malloc (vg_replace_malloc.c:381)
==353287==    by 0x4B8BD267: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvcucompat.so)
==353287==    by 0x470A3A8F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x47019F17: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x4709836B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x471895BF: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x4719247F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x47003443: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x2108F81F: ??? (in /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudart.so.11.4.167)
==353287==    by 0x2109231B: ??? (in /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudart.so.11.4.167)
==353287==    by 0x21137DF3: __pthread_once_slow (pthread_once.c:116)
==353287==    by 0x210D0893: ??? (in /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudart.so.11.4.167)

From the report we know malloc called by libnvcucompat.so causes the memory leak.

Is this a real memory leak problem or a false-positive report? and how to solve it?

Hi,

Thanks for reporting this.

We are going to investigate this.
Will share more information with you later.

Hi,

We test TensorRT in our JetPack 5.0 GA which will be released in the near future.
There is no definite leakage anymore. Please wait for the release.

$ valgrind --tool=memcheck --leak-check=full -- /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx
...
==60328== 
==60328== LEAK SUMMARY:
==60328==    definitely lost: 0 bytes in 0 blocks
==60328==    indirectly lost: 0 bytes in 0 blocks
==60328==      possibly lost: 528 bytes in 9 blocks
==60328==    still reachable: 23,371,539 bytes in 53,376 blocks
==60328==                       of which reachable via heuristic:
==60328==                         stdstring          : 1,272,345 bytes in 11,397 blocks
==60328==         suppressed: 0 bytes in 0 blocks
==60328== Reachable blocks (those to which a pointer was found) are not shown.
==60328== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==60328== 
==60328== For lists of detected and suppressed errors, rerun with: -s
==60328== ERROR SUMMARY: 9 errors from 9 contexts (suppressed: 0 from 0)

Thanks.

When will you release JetPack 5.0 GA?

Hi,

Sorry that we cannot disclose the detailed schedule here.

Based on our roadmap below, it should be available recently.
But it’s expected to be slightly delayed.

Jetson Roadmap | NVIDIA Developer

Thanks.