Memory leak on replacing eos source bin with new one

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Hi,

I’m using DeepStream (7.1) and handling dynamic RTSP sources via uridecodebin wrapped inside Gst Bin. Upon EOS , I remove the source from the pipeline and create new Gst Bin and attempt reconnection.

Now I’m also logging the ram utilization in the deepstream docker and it seems to only increase on reconnection.

Appreciate any help.

Thanks!

inception

3

Could you please elaborate on the following two questions?

  1. Please share your memory statistical data, like what the amount of memory leaked each time is.
  2. How did you do the reconnection?
4

To debug this issue, I created a simple pipeline: two RTSP sources, each wrapped in a GstBin, feeding into a newstreammux and then a fakesink. One RTSP source produces EOS every 5 seconds, the other after ~3 hours.

The reconnection process works as follows:

  1. EOS is detected on a pad probe on the GstBin.

  2. A bus callback triggers the EOS handler.

  3. The handler unlinks the pads between streammux and the EOS bin.

  4. The streammux pad is released.

  5. The EOS bin is set to NULL state.

  6. Once the state change is complete, a flush stop event is sent.

  7. The EOS bin is then removed from the pipeline.

This handler runs via Gst.idle_add and performs the operation only in Gst.PadProbeType.IDLE.

For every EOS event from the source generating EOS every 5 seconds, memory initially increases by 7–10 MB. After a few EOS cycles, the increase stabilizes at 0.25–1 MB per reconnection.

Please note that the memory stats given above are only on reconnection for the 5 sec eos source, single source reconnection only.

5

Could you attach your source code about the simple pipeline so that we can try that on our side?
And how to create a rtsp source produces EOS every 5 seconds?

6

Yes, I have sent it to you privately.
I have hosted the rtsp server on local docker container, which takes video file and streams it on connection.

7

Did you run that on the Jetson Board or dGPU?

8

dGPU
NVIDIA GeForce RTX 3050 6GB

9

Could you please try the following experiment to narrow down the scope?

  1. Replace the “uridecoderbin” in your code with “rtspsrc->rtph264depay ! h264parse ! nvv4l2decoder !”. to check if there is any memory leak.
  2. Replace nvv4l2decoder with avdec_h264 and check if there is any memory leak
10

Hi I tried both:

  1. this did not work, same increase in memory on eos
  2. facing issues with avdec_h264, pipeline not working
11

For the 2nd method, could you try to add nvvideoconvert->caps_vidconvsrc after the avdec_h264. You can refer to the deepstream_test_1_usb.py.

This step is important and can be used to check whether there is a leak from the nvv4l2decoder .

12

It says:

TypeError: cannot unpack non-iterable NoneType object

this was logged for the avdec_h264

13

You can try to run the command below to install the avdec_h264 plugin.

$cd /opt/nvidia/deepstream/deepstream
$apt --reinstall install libmp3lame0 libxvidcore4
$./user_additional_install.sh
$rm ~/.cache/gstreamer-1.0/*
$gst-inspect-1.0 avdec_h264
14

yes the dependencies were installed and on testing the changes made as suggested. The memory increment for every eos went up to 50-100 mb, it is increasing more rapidly. I have also added the nvvideoconvert->caps_vidconvsrc elements right after the decoder.

15

OK. Thanks for your investigation. From the results, it is hard to determine which plugin is causing the leak. I’ll try the reconnection function with our nvurisrcbin plugin. If there are no issues, we recommend that you directly use this plugin to achieve your needs.

16

I have tried with our nvurisrcbin in the pipeline below.

gst-launch-1.0 nvurisrcbin uri=rtsp://<rtsp_source> rtsp-reconnect-attempts=-1 rtsp-reconnect-interval=3  ! m.sink_0 nvstreammux name=m width=1280 height=720 batch-size=1 ! fakesink

The memory is basically stable. Could you please try using nvurisrcbin plugin to fulfill your reconnection requirements? It is more convenient to use and gets relatively stable memory.

17

Sure Ill try and let you know. By the way, was the reconnection stable, as in, was it reconnecting without errors on every eos?

Thanks

18

Yes. You can try that with the pipeline I attached.

19

Hi, I have implemented the pipeline, it’s still increasing steadily but slower. I have shared the script and logs accordingly.

Thanks

20

According to the data you have collected, it took approximately 2 hours and 48 minutes. The reconnection occurred every 5 seconds, and the memory increased from 97.5 to 109.40.

Each reconnection results in approximately 5K increase in memory. As you’ve said, the leak is very slow now. We suggest that you adopt this solution first. We will do a simultaneous analysis.

21

Thank you for the clarification. I’d like to point out that the increase from 97.5 to 109.40 MB actually occurred over 2 minutes, not 2 hours. Also, this observation was made with just a single source reconnection. In cases where multiple sources are involved, we can expect the increments to be significantly larger.