We are seeing some issues with memory leaks when using the bufapi-version=true option in nvvideoconvert/nvv4l2h264enc. We’re testing with L4T 32.3.1.
There are some similar threads here and here, but they don’t provide any solutions.
Overview of the tests that we ran:
// INPUT OUTPUT BUFAPI-VERSION RESULT
nvarguscamerasrc -> nvvideoconvert -> h264 / h265 true LEAK
nvarguscamerasrc -> nvvideoconvert -> h264 / h265 false GOOD
nvarguscamerasrc -> identity -> h264 / h265 true LEAK
nvarguscamerasrc -> identity -> h264 / h265 false GOOD
nvarguscamerasrc -> nvvidconv -> omxenc n/a GOOD
videotestsrc x-raw -> nvvideoconvert -> identity -> h264 true LEAK
videotestsrc x-raw -> nvvideoconvert -> identity -> h264 false LEAK
videotestsrc x-raw -> nvvidconv -> identity -> h264 true SEGV (expected, but we would prefer an error)
videotestsrc x-raw -> nvvidconv -> identity -> h264 false GOOD
We have attached some code for you to compile so you can reproduce the issue yourself. You can compile it with:
g++ -o memleak memleak.cpp
pkg-config --cflags --libs gstreamer-1.0 -lgstapp-1.0
Please check the memory usage with htop while it is running so you can see it increase after every run.
Hope you can provide us with a solution for this. Thanks in advance!
memleak.cpp (7.2 KB)
Are you able to try r32.4.3? This might be specific to r32.3.1.
I just flashed my device with JetPack 4.4 (r32.4.3), seeing the same issue there.
I have now replicated the problem with a much simpler code example without any appsources or appsinks (please see attachment). Still testing on JetPack 4.4 with Deepstream 5.0.
memleak-simple.cpp (4.1 KB)
Essentially the pipeline is this, only written in code:
videotestsrc ! video/x-raw, width=4096, height=3046, framerate=30/1, format=(string)NV12 ! nvvideoconvert ! video/x-raw(memory:NVMM), width=4096, height=3046, framerate=30/1, format=(string)NV12 ! nvv4l2h264enc bufapi-version=false ! mpegtsmux ! filesink location=test.mp4
Compilation instructions still hold:
g++ -o memleak memleak-simple.cpp `pkg-config --cflags --libs gstreamer-1.0` -lgstapp-1.0
Commenting line 26 or 27 will switch between nvvidconv and nvvideoconvert. With the former, we see no leaks, with the latter we see approximately an 180MB leak on every iteration.
If there is anything I can do to make it easier to debug/reproduce on your end, please let me know!
We will try to reproduce it. Thanks for providing more information.
Hey DaneLLL, this is still causing issues for us. Have you been able to reproduce it yet? If not, do you have any idea for a timeline?
Thanks in advance!
This is a known issue and under investigation. Will update once there is new findings.
I saw DeepStream 5.1 was released: https://docs.nvidia.com/metropolis/deepstream/DeepStream_5.1_Release_Notes.pdf.
I do see mentions of 'Fixes for Memory leaks and RTSP for improved stability ’ and ‘Misc. fixes in nvinfer, muxer, osd and video convert plugins’, but I don’t know what the exact fixes are.
I was wondering if this memory leak issue is fixed in the latest release?
On DS5.1, we have the fix to memory leak in nvvideoconvert and nvv4l2decoder plugins.