Deepstream pipeline memory leak

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)：Jetson Orin Nano 4G
• DeepStream Version：deepstream-6.2

My deepstream pipeline has a serious memory leak. I used AddressSanitizer to detect the memory leak, and the logs are as follows:
AddressSanitizer.txt (35.7 KB)

valgrind --tool=memcheck --leak-check=full --num-callers=100 --show-leak-kinds=definite,indirect --track-origins=yes ./main：
va.txt (9.8 KB)

Using nvmemstat.by to detect memory, VmRSS and RssAnon continue to increase

my pipeline is : nvurisrcbin–streammux–nvinfer–queue1, nvvidconv, queue2, nvvidconv_cap, queue3, sink

3

We fixed a lot of memory leaks issues in the latest version, please update to the latest version DS7.1 first to try it out

7 
==981947== 9 bytes in 1 blocks are definitely lost in loss record 69 of 2,001
==981947==    at 0x4849D8C: malloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so)
==981947==    by 0x4AB071F: g_malloc (in /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0.6400.6)
==981947==    by 0x4AA0257: ??? (in /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0.6400.6)
==981947==    by 0x4AA146B: g_key_file_get_string (in /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0.6400.6)
==981947==    by 0x487058F: ??? (in /opt/nvidia/deepstream/deepstream-6.2/lib/libnvds_amqp_proto.so)
==981947==    by 0x4870AB3: nvds_msgapi_connect (in /opt/nvidia/deepstream/deepstream-6.2/lib/libnvds_amqp_proto.so)
==981947==    by 0x11BD7B: connect_rabbitmq(ReturnInfo*, void**) (in /home/nvidia/code/tgsd_code/test3/main)
==981947==    by 0x115BB7: main (in /home/nvidia/code/tgsd_code/test3/main)

From the valgrind log you provided, there is a memory leak in libnvds_amqp_proto.so. I checked the code and found that the problem has been fixed in the latest code.

Please upgrade DS-7.1. If it is not convenient for you to upgrade, since libnvds_amqp_proto.so is not open source on DS-6.2, you can consider porting the code from DS-7.1 to DS-6.2 to solve this problem.

8

@junshengy

Thank you for your reply. The memory leak caused by libnvds_amqp-pro to.so is not a big problem. I found that the drop frame interval parameter of nvurisrcbin in deepstream-6.2 can cause serious memory leaks when started. I have abandoned this parameter now. If you have time, you can conduct testing and comparison. I use nvcr.io/nvidia/deptstream-l4t:6.2-triton as the basic Docker image.

9

This problem can be verified with gst-launch-1.0, and from the valgrind logs, this does not seem to be a problem.

valgrind --tool=memcheck --leak-check=full --num-callers=100 --show-leak-kinds=definite,indirect --track-origins=yes gst-launch-1.0 nvurisrcbin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 drop-frame-interval=2  ! fakesink
10

@junshengy
I will conduct a test later because opening this parameter in my code caused a serious memory leak. I had to discard this parameter, and after that, the memory leak problem disappeared. I can only determine that this parameter caused the memory leak.