Deepstream-app may cause mem leak with source reconnecting

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (GPU)
• DeepStream Version 6.0.1
• TensorRT Version 8.0.1.6
• NVIDIA GPU Driver Version 510.108.03
• Issue Type (bugs)
• How to reproduce the issue ?

We run deepstream-app and disconnect the network of camera every 20 seconds, found it causes the host mem increase obviously. we use the yolov3 sample, source is configerd like
type=4
rtsp-reconnect-attempts=-1
rtsp-reconnect-interval-sec=10

sink uses fakesink.

3
  1. are you testing in docker? did you modify the deepstream-app code?
  2. 6.0.1 is an old version. can you reproduce this issue on DeepStream 6.3?
4

thank you for your reply.

  1. We used docker for our testing without any modify with deepstream-app.
  2. This porblem is occured in production environment，it is hard to update the SDK there.
  3. We just reproduce the same issue with deepstream 6.3 in docker, getting a similar result.
  4. there is a mistake with my discribe. we do not recover the net of camera immediately after the disconnect. Instead, we will do it just after the deepstream log shows “Resetting source 0” message.
5

please refer to this topic. to narrow down this issue, can you use this method to get a memory leak log? and can you use this method to get a valgrind memory leak analysis log? Thanks!

6

we get the logs in separate tests.vmemstat.log without valgrind.
nvmemstat.log (91.7 KB)
valReport.log (146.4 KB)

7

Is this enough ? what else can I do to help ?

8

thanks for the sharing! it is helpful. we will try to reproduce and investigate.

9

case1:
==2767== by 0x1200ED: reset_source_pipeline (deepstream_source_bin.c:1554)
==2767== by 0x11BCDA: watch_source_status (deepstream_source_bin.c:688)

case2:
==2767== by 0x663D6B7: gst_event_new_flush_stop (in /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0.1405.0)
==2767== by 0x11FF54: reset_source_pipeline (deepstream_source_bin.c:1540)

from the valReport.log, there is memory leak in gst_nvevent_new_stream_reset. it is not opensource. we will fix it in the latter versions. Thank you!

11

Thank you. And it will be very useful if you can provide a version can fix the mem leak problem with deepstream 6.0.1 .

12
  1. this memory leak only happens when reconnecting RTSP, why do you need to reconnect RTSP frequently?
  2. sorry, we are unable to publish versions on the forum. you can contact sales or after-sales.
13
  1. deal to the bad network sometimes and more than a hundrod of cameras reconnecting.
  2. which sales or after-sales can we seek to help? deepstream or nvidia GPU card?
14

nvidia GPU card. deepstream is free to use.

15

There is no update from you for a period, assuming this is not an issue any more. Hence we are closing this topic. If need further support, please open a new one. Thanks.
what is your company name?

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.