Memory leak in DeepStream

QoS memory leak has been fixed in DS 6.0 GA version. There will be more memory leak fixes in the future release. Can you please get more leak info with the method in DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums?

Hi @Fiona.Chen , I see it mentioned in the release notes for DS6.0 https://docs.nvidia.com/metropolis/deepstream/DeepStream_6.0_Release_Notes.pdf . What is the difference between version 6.0 and 6.0 GA if any?

There’s more information in the 1st, 2nd and 3rd posts.

I run my pipeline with 1 camera and I have the following output:

==15180== LEAK SUMMARY:
==15180==    definitely lost: 19,058 bytes in 85 blocks
==15180==    indirectly lost: 5,499 bytes in 200 blocks
==15180==      possibly lost: 1,604,596 bytes in 6,843 blocks
==15180==    still reachable: 1,497,043,413 bytes in 940,007 blocks
==15180==                       of which reachable via heuristic:
==15180==                         stdstring          : 1,200,349 bytes in 20,069 blocks
==15180==                         length64           : 2,248 bytes in 49 blocks
==15180==                         newarray           : 2,296 bytes in 40 blocks
==15180==         suppressed: 0 bytes in 0 blocks
==15180== Reachable blocks (those to which a pointer was found) are not shown.
==15180== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==15180== 
==15180== For counts of detected and suppressed errors, rerun with: -v
==15180== ERROR SUMMARY: 38579 errors from 2065 contexts (suppressed: 0 from 0)

How should I stop valgrind? I just run a kill pid. It looks like from data has been lost from the report above. That seems to confirm the memory leak.

@marcoslucianops assuming that a queue with leaky=2 would drop frames when the pipeline can’t keep up (still not 100% sure, what’s your understanding?) if this does not work (and it doesn’t) I think we can assume there is a component after the queue that keeps pulling data from the pipeline. What do you think?
I have been fighting with this leak for weeks. The only solution I found was to decode the video using rtpsrc using drop-on-latency=1 but this lead to corrupted/pixelated images on some video streams: see Gstreamer corrupted image from rtsp video stream - Stack Overflow . However, the fact this solves the memory leak (despite not being a solution) seems to be in contrast with my first statement where I said that the leak should be after the queue. That’s why at this point I am not really sure a leaky queue would drop the frames.

@marcoslucianops this user seems to have a memory leak too Significant Memory leak when streaming from a clients RTSP source - #4 by karan.shetty

I think I found a way to reproduce the memory leak here: Deepstream RTSP memory leak (with code to reproduce the issue)

@mfoglio the queue drops the frames but the rtpjitterbuffer still seems to buffer the frames. I did long tests and, with drop-frame-interval or videorate to set streams to 1 FPS, it doesn’t seem to increase memory after about 3-4 hours. I’m doing more tests but it takes long time.

1 Like

Thanks. I read about that parameter in the documentation. However, how one can find a correct value for drop-frames-interval without knowing a priori the fps for the video streams? You might end up dropping too many or too few frames.

@marcoslucianops how did you exactly set up your pipeline?
I have rtspsrc -> rtph264depay -> nvv4l2decoder -> videorate -> leaky queue -> fakesink but I still see a memory leaky.
I set max-rate=1 for the videorate element and leaky=2 and max-size-buffers=1 for the queue. Did you have to use both videorate and drop-frame-interval at the same time?
Thank you for sharing!

@mfoglio I did the pipeline in DeepStream code.

The first one was:
20 rtsp (rtspsrc -> rtph264depay -> h264parse -> nvv4l2decoder -> videorate) -> nvstreammux -> nvinfer -> nvmultistreamtiler -> nvvideoconvert -> nvdsosd -> nveglglessink

The second one was:
20 rtsp (uridecodebin + drop-frame-interval in nvv4l2decoder) -> nvstreammux -> nvinfer -> nvmultistreamtiler -> nvvideoconvert -> nvdsosd -> nveglglessink

I put the nvmultistreamtiler -> nvvideoconvert -> nvdsosd -> nveglglessink part to see if everything was running well.

I simulated the streams by using a rtsp server in FFmpeg.

@marcoslucianops thank you. Did you set the property max-rate of videorate or did you use other properties? Weird fact: if I set max-rate to 1 and drop-only=1 when connecting 64 cameras I would expect that only 64 streams per second would be produced. Instead my pipeline can process 500 frames per second. I checked by attaching a probe to the nvinfer element. Not sure where all these frames are coming from.

I set only max-rate to 1.

Did you verify if the frames are actually dropped? (e.g. did you check a video or does the gpu load seems to be same when you set max-rate to 1 vs 30). It really does not seem to work for me unfortunately

@marcoslucianops I also noticed this suspicious warning:

0:00:07.145423577  1405 0x7f87540250a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1512:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder_40:pool:src> Driver should never set v4l2_buffer.field to ANY

I have one of this warning for each stream.

I think I found the issue. The T4 probably can’t decode more than 30-40 1080 streams. With more, NVDEC usage goes above 100%, and this leads to the memory leak.

Did you verify if the frames are actually dropped? (e.g. did you check a video or does the gpu load seems to be same when you set max-rate to 1 vs 30 ). It really does not seem to work for me unfortunately

Yes it’s dropping in my case.

I have one of this warning for each stream.

There’s no problem.

I think I found the issue. The T4 probably can’t decode more than 30-40 1080 streams. With more, NVDEC usage goes above 100%, and this leads to the memory leak.

I don’t know, but I’m using my GTX 1050 to test with 20 rtsp streams (H264 1080p 30fps) dropped to 1fps.

Thank you. I still have no idea as to why videorate is not dropping images but I will try to work on this. Inserting a queue between the nvv4l2decoder and nvvideoconverter seems to avoid the memory leak.
For future reference, in case anyone will encounter the same issue: I don’t think it’s possible to decode more than approximately 30 1080p h264 video streams on a T4. So when reaching that point, I think you would either have to choose between corrupted images (dropping rtsp packet) or corrupted images

Sorry for the late response, Setting rtspsrc drop-on-latency can improve, here is the test report, there is only 6M memory leak after 8 hours.
with_patch_VmRss_8h.txt (720 Bytes)

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks