Deepstream RTSP memory leak (with code to reproduce the issue)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

After a while I think I found a way to reproduce the RTSP video decoding memory leak. I read about other users having this problem, so it seems to be a common problem. I beg you to try to investigate this with me :)
The pipeline is as follow: rtsp_decode_bin → leaky queue → fake sink.
For you convenience you can find the pipeline graph here: pipeline_rtsp_decode.pdf - Google Drive

Note: as you can see from the pipeline graph, the fakesink for some reason does not seem to be in a play state. I happy to any suggestion on how to fix this. In any case, I would have expected the leaky queue to drop data in excess.

The following code will allow you to reproduce the issue. It leads to a memory leak in the Deepstream 6.0 devel container on Tesla T4.

How to run the script:

  • You need to add your rtsp video url at line 149
  • On line 150 you can select how many decoding bin you want to create using the stream above. Set the number of streams to 64. With 64 streams, the memory leak is evident in a few seconds using utilities such as top and htop. With just 1 stream you won’t notice the memory leak.
  • If you uncomment line 38-39 you will solve the memory leak, but this will lead to corrupted video frames when connecting this video decoding bin to an actual Deepstream pipeline (with nvinfer, etc)

main.py (7.5 KB)

Possibly related issues:

Let me know if you have any question!

3

How could know it is a memory leak? How did you test it?

4

Hi @Fiona.Chen , the memory keeps increasing in a pretty evident way. However, after performing some tests, I think I found out the issue. This happens only if I decode more than 40 1080p video streams. I suppose NVDEC can’t keep up and therefore some video content is buffered.
Do you know if Deepstream is capable to use both NVDEC chips on the Tesla T4?
Thank you for your help

5

@Fiona.Chen , is if I set drop-on-latency=1 for the rtspsrc element I get pixelated images. Is there a way I can drop data without decoding it without having corrupted images?

Also, why do you think I had to add a queue between the nvv4l2decoder and the nvvideoconverter make the pipeline above running?

7

Hi @mfoglio，if you uncomment line 38-39 , you will drop too much frame to lead to corrupted video frames.

Cause the rtsp source maybe affected by the Internet or other env, could you change the rtspsrc to filesrc and have a test about this issue again?

Could you use the memory detect tool such as valgrind to test this pipeline,
https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/19？

Also Could you provide us some data about how much memory has been increased at a definite time?
Thanks a lot.

9

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.