Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.1.0
There seems to be a issue with endlessly growing memory when using nvv4l2decoder. When I run this gstreamer pipeline and check htop, I can see RES memory increasing linearly. When I have webm file and use vp9dec, I see no issues with the memory.
gst-launch-1.0 filesrc location=../samples/kuressaare.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! progressreport update-freq=1 ! nvv4l2h264enc bitrate=1000000 ! h264parse ! mp4mux ! filesink location=out.mp4
I also checked it using valgrind with the following command:
valgrind --tool=memcheck --leak-check=full --num-callers=100 --show-leak-kinds=definite,indirect --track-origins=yes --log-file="valgrind-out.txt" gst-launch-1.0 filesrc location=../samples/kuressaare.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! progressreport update-freq=1 ! nvv4l2h264enc bitrate=1000000 ! h264parse ! mp4mux ! filesink location=out.mp4
This is the result:
==308609== HEAP SUMMARY:
==308609== in use at exit: 9,828,670 bytes in 12,276 blocks
==308609== total heap usage: 122,717 allocs, 110,441 frees, 3,284,469,481 bytes allocated
==308609==
==308609== 8 bytes in 1 blocks are definitely lost in loss record 132 of 4,009
==308609== at 0x483B7F3: malloc (vg_replace_malloc.c:309)
==308609== by 0x6E5032B: ???
==308609== by 0x6E59E3A: ???
==308609== by 0x6E4F95C: ???
==308609== by 0x681343A: ???
==308609== by 0x682108D: v4l2_ioctl (in /opt/nvidia/deepstream/deepstream-6.1/lib/libnvv4l2.so)
==308609== by 0x67C9228: ??? (in /opt/nvidia/deepstream/deepstream-6.1/lib/gst-plugins/libgstnvvideo4linux2.so)
==308609== by 0x67CD41E: gst_v4l2_buffer_pool_process (in /opt/nvidia/deepstream/deepstream-6.1/lib/gst-plugins/libgstnvvideo4linux2.so)
==308609== by 0x67E22AF: ??? (in /opt/nvidia/deepstream/deepstream-6.1/lib/gst-plugins/libgstnvvideo4linux2.so)
==308609== by 0x6598B4F: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1602.0)
==308609== by 0x48E1EFE: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1602.0)
==308609== by 0x48E3F60: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1602.0)
==308609== by 0x48EAD72: gst_pad_push (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1602.0)
==308609== by 0x646943F: ??? (in /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0.1602.0)
==308609== by 0x48E1EFE: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1602.0)
==308609== by 0x48E3F60: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1602.0)
==308609== by 0x48EAD72: gst_pad_push (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1602.0)
==308609== by 0x658A8A4: ??? (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1602.0)
==308609== by 0x6591FFA: gst_video_decoder_finish_frame (in /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0.1602.0)
==308609== by 0x67E0D2F: ??? (in /opt/nvidia/deepstream/deepstream-6.1/lib/gst-plugins/libgstnvvideo4linux2.so)
==308609== by 0x4919106: ??? (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1602.0)
==308609== by 0x4A8A373: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.6)
==308609== by 0x4A89AD0: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.6)
==308609== by 0x4B3F608: start_thread (pthread_create.c:477)
==308609== by 0x4C7B132: clone (clone.S:95)
==308609==
==308609== 16,384 bytes in 1 blocks are definitely lost in loss record 3,951 of 4,009
==308609== at 0x483B7F3: malloc (vg_replace_malloc.c:309)
==308609== by 0x4A65E98: g_malloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.6)
==308609== by 0x4A707D3: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.6)
==308609== by 0x4011B99: call_init.part.0 (dl-init.c:72)
==308609== by 0x4011CA0: call_init (dl-init.c:30)
==308609== by 0x4011CA0: _dl_init (dl-init.c:119)
==308609== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==308609== by 0x15: ???
==308609== by 0x1FFF000152: ???
==308609== by 0x1FFF000161: ???
==308609== by 0x1FFF000169: ???
==308609== by 0x1FFF00018C: ???
==308609== by 0x1FFF00018E: ???
==308609== by 0x1FFF000196: ???
==308609== by 0x1FFF000198: ???
==308609== by 0x1FFF0001A2: ???
==308609== by 0x1FFF0001A4: ???
==308609== by 0x1FFF0001B2: ???
==308609== by 0x1FFF0001B4: ???
==308609== by 0x1FFF0001C3: ???
==308609== by 0x1FFF0001D1: ???
==308609== by 0x1FFF0001D3: ???
==308609== by 0x1FFF0001E1: ???
==308609== by 0x1FFF0001F1: ???
==308609== by 0x1FFF0001F3: ???
==308609== by 0x1FFF0001FD: ???
==308609== by 0x1FFF0001FF: ???
==308609== by 0x1FFF000206: ???
==308609== by 0x1FFF000208: ???
==308609== by 0x1FFF000211: ???
==308609==
==308609== LEAK SUMMARY:
==308609== definitely lost: 16,392 bytes in 2 blocks
==308609== indirectly lost: 0 bytes in 0 blocks
==308609== possibly lost: 34,244 bytes in 238 blocks
==308609== still reachable: 9,762,346 bytes in 11,919 blocks
==308609== of which reachable via heuristic:
==308609== length64 : 1,104 bytes in 21 blocks
==308609== newarray : 1,664 bytes in 24 blocks
==308609== suppressed: 0 bytes in 0 blocks
==308609== Reachable blocks (those to which a pointer was found) are not shown.
==308609== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==308609==
==308609== For lists of detected and suppressed errors, rerun with: -s
==308609== ERROR SUMMARY: 240 errors from 240 contexts (suppressed: 0 from 0)
I guess the issue is with “still reachable” memory.
The “still reachable” category within Valgrind’s leak report refers to “memory was allocated and was not subsequently freed before the program terminated.”
I have a lot of live stream cameras running in deepstream and this is how the container_memory_rss graph looks. :|
Can you check and confirm this issue?
A moderator asked me to make a new topic. This is the old one: