Valgrind reports memory leaks on CUDA library

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
1

Software versions:
JetPack: 5.0.1
CUDA: 11.4
TensorRT: 8.4.0.11
CuDNN: 8.3.2.49
Valgrine: 3.19.0

Problem description:
We run a valgrine memory leak check on trtexec the MINST onnx model installed with TensorRT samples, with the following commands.

valgrind --tool=memcheck --leak-check=full -- trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx

The valgrind reports 468 bytes of definitely lost.

The call stacks is as follows:

==353287== 20 bytes in 1 blocks are definitely lost in loss record 82 of 4,025
==353287==    at 0x484B0C8: malloc (vg_replace_malloc.c:381)
==353287==    by 0x4B8BD267: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvcucompat.so)
==353287==    by 0x470A3A8F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x47019F17: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x4709836B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x471895BF: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x4719247F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x47003443: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==353287==    by 0x2108F81F: ??? (in /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudart.so.11.4.167)
==353287==    by 0x2109231B: ??? (in /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudart.so.11.4.167)
==353287==    by 0x21137DF3: __pthread_once_slow (pthread_once.c:116)
==353287==    by 0x210D0893: ??? (in /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudart.so.11.4.167)

From the report we know malloc called by libnvcucompat.so causes the memory leak.

Is this a real memory leak problem or a false-positive report? and how to solve it?

3

Hi,

Thanks for reporting this.

We are going to investigate this.
Will share more information with you later.

4

Hi,

We test TensorRT in our JetPack 5.0 GA which will be released in the near future.
There is no definite leakage anymore. Please wait for the release.

$ valgrind --tool=memcheck --leak-check=full -- /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx
...
==60328== 
==60328== LEAK SUMMARY:
==60328==    definitely lost: 0 bytes in 0 blocks
==60328==    indirectly lost: 0 bytes in 0 blocks
==60328==      possibly lost: 528 bytes in 9 blocks
==60328==    still reachable: 23,371,539 bytes in 53,376 blocks
==60328==                       of which reachable via heuristic:
==60328==                         stdstring          : 1,272,345 bytes in 11,397 blocks
==60328==         suppressed: 0 bytes in 0 blocks
==60328== Reachable blocks (those to which a pointer was found) are not shown.
==60328== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==60328== 
==60328== For lists of detected and suppressed errors, rerun with: -s
==60328== ERROR SUMMARY: 9 errors from 9 contexts (suppressed: 0 from 0)

Thanks.

5

When will you release JetPack 5.0 GA?

6

Hi,

Sorry that we cannot disclose the detailed schedule here.

Based on our roadmap below, it should be available recently.
But it’s expected to be slightly delayed.

Jetson Roadmap | NVIDIA Developer

Thanks.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.