Memory leak in DeepStream

I remember reading in the Release Notes of Deepstream 6.0 about a potential memory leak related to QOS messages being sent upstream. Please double check the document to make sure my memory is not tricking me !

Hey, you’re right. I noticed that too.
Btw, there were some unattended-upgrades on my machine yesterday. After that, I am unable to reproduce the memory leak error. It’s quite weird because even the valgrind logs don’t match anymore. I was able to reproduce it several times earlier, but cannot do it after the upgrade.

I have tested the same code on 2 machines.

  • 1 with RTX 2080 Ti (that has the unattended-updates enabled) I cannot reproduce it on this machine anymore. The driver version of this is 470
  • with RTX 3090 (this has not had any upgrades). I can still reproduce it on this machine. The driver version of this machine is 510

Have you faced anything similar?

@mfoglio Nice to see you too. I’ve tested the pipeline adding a queue element with leaky=2 and max-size-buffers=1 properties between source and nvstreammux (source_src_pad → queue_sink_pad and queue_src_pad → streammux_sink_pad) but I’m still facing memory leaks in the pipeline.

My pipeline is uridecodebin (50 sources) → queue (50 sources) → nvstreammux → nvinfer → nvvideoconvert → nvdsosd → fakesink

NOTE: The sink qos property is set to 0 (FALSE).

@marmikshah In my case, I use nvv4l2decoder for mpeg4/h264/h265 decode.

I am not sure the leaky queue works as expected in that scenario because I experienced the same issue. I opened a thread about that here: Gstreamer leaky queue stops the pipeline

@marcoslucianops do you how we can track the memory usage of each gstreamer component?

QoS memory leak has been fixed in DS 6.0 GA version. There will be more memory leak fixes in the future release. Can you please get more leak info with the method in DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums?

Hi @Fiona.Chen , I see it mentioned in the release notes for DS6.0 . What is the difference between version 6.0 and 6.0 GA if any?

There’s more information in the 1st, 2nd and 3rd posts.

I run my pipeline with 1 camera and I have the following output:

==15180== LEAK SUMMARY:
==15180==    definitely lost: 19,058 bytes in 85 blocks
==15180==    indirectly lost: 5,499 bytes in 200 blocks
==15180==      possibly lost: 1,604,596 bytes in 6,843 blocks
==15180==    still reachable: 1,497,043,413 bytes in 940,007 blocks
==15180==                       of which reachable via heuristic:
==15180==                         stdstring          : 1,200,349 bytes in 20,069 blocks
==15180==                         length64           : 2,248 bytes in 49 blocks
==15180==                         newarray           : 2,296 bytes in 40 blocks
==15180==         suppressed: 0 bytes in 0 blocks
==15180== Reachable blocks (those to which a pointer was found) are not shown.
==15180== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==15180== For counts of detected and suppressed errors, rerun with: -v
==15180== ERROR SUMMARY: 38579 errors from 2065 contexts (suppressed: 0 from 0)

How should I stop valgrind? I just run a kill pid. It looks like from data has been lost from the report above. That seems to confirm the memory leak.

@marcoslucianops assuming that a queue with leaky=2 would drop frames when the pipeline can’t keep up (still not 100% sure, what’s your understanding?) if this does not work (and it doesn’t) I think we can assume there is a component after the queue that keeps pulling data from the pipeline. What do you think?
I have been fighting with this leak for weeks. The only solution I found was to decode the video using rtpsrc using drop-on-latency=1 but this lead to corrupted/pixelated images on some video streams: see Gstreamer corrupted image from rtsp video stream - Stack Overflow . However, the fact this solves the memory leak (despite not being a solution) seems to be in contrast with my first statement where I said that the leak should be after the queue. That’s why at this point I am not really sure a leaky queue would drop the frames.

@marcoslucianops this user seems to have a memory leak too Significant Memory leak when streaming from a clients RTSP source - #4 by karan.shetty

I think I found a way to reproduce the memory leak here: Deepstream RTSP memory leak (with code to reproduce the issue)

@mfoglio the queue drops the frames but the rtpjitterbuffer still seems to buffer the frames. I did long tests and, with drop-frame-interval or videorate to set streams to 1 FPS, it doesn’t seem to increase memory after about 3-4 hours. I’m doing more tests but it takes long time.

1 Like

Thanks. I read about that parameter in the documentation. However, how one can find a correct value for drop-frames-interval without knowing a priori the fps for the video streams? You might end up dropping too many or too few frames.

@marcoslucianops how did you exactly set up your pipeline?
I have rtspsrc -> rtph264depay -> nvv4l2decoder -> videorate -> leaky queue -> fakesink but I still see a memory leaky.
I set max-rate=1 for the videorate element and leaky=2 and max-size-buffers=1 for the queue. Did you have to use both videorate and drop-frame-interval at the same time?
Thank you for sharing!

@mfoglio I did the pipeline in DeepStream code.

The first one was:
20 rtsp (rtspsrc -> rtph264depay -> h264parse -> nvv4l2decoder -> videorate) -> nvstreammux -> nvinfer -> nvmultistreamtiler -> nvvideoconvert -> nvdsosd -> nveglglessink

The second one was:
20 rtsp (uridecodebin + drop-frame-interval in nvv4l2decoder) -> nvstreammux -> nvinfer -> nvmultistreamtiler -> nvvideoconvert -> nvdsosd -> nveglglessink

I put the nvmultistreamtiler -> nvvideoconvert -> nvdsosd -> nveglglessink part to see if everything was running well.

I simulated the streams by using a rtsp server in FFmpeg.

@marcoslucianops thank you. Did you set the property max-rate of videorate or did you use other properties? Weird fact: if I set max-rate to 1 and drop-only=1 when connecting 64 cameras I would expect that only 64 streams per second would be produced. Instead my pipeline can process 500 frames per second. I checked by attaching a probe to the nvinfer element. Not sure where all these frames are coming from.

I set only max-rate to 1.

Did you verify if the frames are actually dropped? (e.g. did you check a video or does the gpu load seems to be same when you set max-rate to 1 vs 30). It really does not seem to work for me unfortunately

@marcoslucianops I also noticed this suspicious warning:

0:00:07.145423577  1405 0x7f87540250a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1512:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder_40:pool:src> Driver should never set v4l2_buffer.field to ANY

I have one of this warning for each stream.