Memory continuously increase about 50mb/day, and pipeline crash

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) rtx3050
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only)
• TensorRT Version official deepstream 6.2 docker image
• NVIDIA GPU Driver Version (valid for GPU only) NVIDIA-Linux-x86_64-535.86.05
• Issue Type( questions, new requirements, bugs) bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

Only do some config modification with python demo “deepstream_nvdsanalytics.py”. I run the application with 16 rtsp streams and with yolov8 model(fp16) for almost a day, and the memory increased about 50mb. Is this an obvious memory leak or not?
Another problem is the pipeline will definitely crash after running for about a day with no logs(may be logs in the console output but I can only see the logging file) while the computer memory is enough, this is a serious problem in production use. How to solve it?

Thank you!

this is the memory monitor log:

2023-08-06-23:33:08
VmRSS: 1899216 kB
VmSize: 21025980 kB
…
2023-08-07-20:46:51
VmRSS: 1949432 kB
VmSize: 21025980 kB
…
2023-08-07-23:11:22
VmRSS: 1952336 kB
VmSize: 21025980 kB

53120/1024=51mb

this is the gpu monitor log:

#Date Time gpu fb bar1 ccpm sm mem enc dec jpg ofa
#YYYYMMDD HH:MM:SS Idx MB MB MB % % % % % %
20230806 23:24:56 0 1 1 0 0 0 0 0 0 0
20230806 23:25:01 0 1 1 0 0 0 0 0 0 0
20230806 23:25:06 0 187 4 0 0 0 0 0 0 0
20230806 23:25:11 0 1679 6 0 44 22 5 21 0 0
20230806 23:25:16 0 1679 6 0 75 48 4 24 0 0

…

#Date Time gpu fb bar1 ccpm sm mem enc dec jpg ofa
#YYYYMMDD HH:MM:SS Idx MB MB MB % % % % % %
20230807 23:11:24 0 1679 6 0 76 47 6 33 0 0
20230807 23:11:29 0 1679 6 0 78 47 6 30 0 0
20230807 23:11:34 0 1679 6 0 75 43 5 30 0 0
20230807 23:11:39 0 1679 6 0 78 48 6 31 0 0
20230807 23:11:44 0 1679 6 0 80 48 6 31 0 0
20230807 23:11:49 0 1679 6 0 83 53 6 28 0 0
20230807 23:11:54 0 1 1 0 0 0 0 0 0 0

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

3

please refer to this topic. Setting rtspsrc drop-on-latency can improve.

4

I have already set the drop-on-latency to True with the above test。
how about the crash issue？ cpu，gpu，memory all seems ok

5
  1. about the memroy leak issue, can you use this method to get a memory leak log? and can you use this method to get a valgrind memory leak analysis log?
  2. about the crash log, to narrow down this isue, can you capture a resource monitoring log? can you run with gdb to get a crash stack?
6

Hi fanzh:
I use (GST_DEBUG=3 GST_DEBUG_FILE=/opt/ds/dbg.log ./app) this time with fake sink, the app runs abount 29 hours before crash.

this is the console output when crash:
**(python3:2821): GStreamer-CRITICAL : 11:37:35.176: gst_buffer_get_sizes_range: assertion ‘GST_IS_BUFFER (buffer)’ failed

there are gpu and memory logs
mem.log.tar.gz (28.1 KB)
gpu.log.tar.gz (307.4 KB)

**This is the lastest log from gdb.log when it’'s crash, without any crash stack **
29:29:10.684041781 ^[[332m 2821^[[00m 0x7fe64800e120 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:58.751384888 < 29:28:58.783523749)
29:29:10.684357709 ^[[332m 2821^[[00m 0x7fe67c0038c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:58.752288761 < 29:28:58.784413569)
29:29:10.686610605 ^[[332m 2821^[[00m 0x7fe64001ef60 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:58.752198025 < 29:28:58.784318951)
29:29:10.686669715 ^[[332m 2821^[[00m 0x7fe58401ee40 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:58.751691648 < 29:28:58.783831602)
29:29:10.686832809 ^[[332m 2821^[[00m 0x7fe58401f640 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:58.751345432 < 29:28:58.783483145)
29:29:11.316017992 ^[[332m 2821^[[00m 0x7fe63c02f0c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:59.559720303 < 29:28:59.593041577)
29:29:11.316061834 ^[[332m 2821^[[00m 0x7fe63000baa0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:59.559913710 < 29:28:59.593235556)
29:29:11.670578831 ^[[332m 2821^[[00m 0x7fe64800e120 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:59.783100835 < 29:28:59.815495469)
29:29:11.671574035 ^[[332m 2821^[[00m 0x7fe58401f640 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:59.783091862 < 29:28:59.815485594)
29:29:11.677084949 ^[[332m 2821^[[00m 0x7fe67c0038c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:59.784377804 < 29:28:59.816761395)
29:29:11.677173544 ^[[332m 2821^[[00m 0x7fe6d0009580 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:59.784366001 < 29:28:59.816748356)
29:29:11.677282607 ^[[332m 2821^[[00m 0x7fe58401ee40 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:59.783378567 < 29:28:59.815774061)
29:29:11.677647697 ^[[332m 2821^[[00m 0x7fe64001ef60 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:28:59.784390141 < 29:28:59.816770681)
29:29:12.343844286 ^[[332m 2821^[[00m 0x7fe63000baa0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:00.559771461 < 29:29:00.593109004)
29:29:12.344719276 ^[[332m 2821^[[00m 0x7fe63c02f0c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:00.559634133 < 29:29:00.592970016)
29:29:12.686489201 ^[[332m 2821^[[00m 0x7fe58401f640 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:00.807648836 < 29:29:00.840260395)
29:29:12.686581864 ^[[332m 2821^[[00m 0x7fe67c0038c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:00.809309485 < 29:29:00.841908866)
29:29:12.686749195 ^[[332m 2821^[[00m 0x7fe64001ef60 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:00.809350102 < 29:29:00.841949206)
29:29:12.687075954 ^[[332m 2821^[[00m 0x7fe64800e120 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:00.807633224 < 29:29:00.840245658)
29:29:12.687249327 ^[[332m 2821^[[00m 0x7fe6d0009580 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:00.809342535 < 29:29:00.841940478)
29:29:12.689884425 ^[[332m 2821^[[00m 0x7fe58401ee40 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:00.807870254 < 29:29:00.840483989)
29:29:12.875434767 ^[[332m 2821^[[00m 0x7fe654003300 ^[[31;01mERROR ^[[00m ^[[00;04m default gstrtspconnection.c:1046:gst_rtsp_connection_connect_with_response:^[[00m failed to connect: Could not connect to 192.168.2.252: No route to host
29:29:12.875460395 ^[[332m 2821^[[00m 0x7fe654003300 ^[[31;01mERROR ^[[00m ^[[00m rtspsrc gstrtspsrc.c:5047:gst_rtsp_conninfo_connect:^[[00m Could not connect to server. (Generic error)
29:29:12.875469532 ^[[332m 2821^[[00m 0x7fe654003300 ^[[33;01mWARN ^[[00m ^[[00m rtspsrc gstrtspsrc.c:7893:gst_rtspsrc_retrieve_sdp:^[[00m error: Failed to connect. (Generic error)
29:29:12.875509005 ^[[332m 2821^[[00m 0x7fe654003300 ^[[33;01mWARN ^[[00m ^[[00m rtspsrc gstrtspsrc.c:7973:gst_rtspsrc_open:^[[00m can’t get sdp
29:29:12.875519084 ^[[332m 2821^[[00m 0x7fe654003300 ^[[33;01mWARN ^[[00m ^[[00m rtspsrc gstrtspsrc.c:8512:gst_rtspsrc_play:^[[00m failed to open stream
29:29:12.875529544 ^[[332m 2821^[[00m 0x7fe654003300 ^[[33;01mWARN ^[[00m ^[[00m rtspsrc gstrtspsrc.c:6032:gst_rtspsrc_loop:^[[00m we are not connected
29:29:12.875550553 ^[[332m 2821^[[00m 0x7fe5ac068410 ^[[33;01mWARN ^[[00m ^[[00m discoverer gstdiscoverer.c:1591:handle_message:^[[00m Got an error [debug:gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:Discoverer/GstURIDecodeBin:discoverer-uri/GstRTSPSrc:source:
Failed to connect. (Generic error)], [message:Could not open resource for reading and writing.]
29:29:12.875676678 ^[[332m 2821^[[00m 0x7fe654003300 ^[[33;01mWARN ^[[00m ^[[00m rtspsrc gstrtspsrc.c:6032:gst_rtspsrc_loop:^[[00m we are not connected
29:29:13.300173114 ^[[332m 2821^[[00m 0x7fe63000baa0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:01.559659674 < 29:29:01.592996316)
29:29:13.354479241 ^[[332m 2821^[[00m 0x7fe63c02f0c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:01.559566417 < 29:29:01.592901754)
29:29:13.649957547 ^[[332m 2821^[[00m 0x7fe58401f640 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:01.826813574 < 29:29:01.859579688)
29:29:13.658305054 ^[[332m 2821^[[00m 0x7fe58401ee40 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:01.826977220 < 29:29:01.859745043)
29:29:13.713610062 ^[[332m 2821^[[00m 0x7fe64800e120 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:01.826774742 < 29:29:01.859541543)
29:29:13.713610092 ^[[332m 2821^[[00m 0x7fe64001ef60 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:01.828792601 < 29:29:01.861557572)
29:29:13.713902256 ^[[332m 2821^[[00m 0x7fe67c0038c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:01.828750178 < 29:29:01.861514461)
29:29:13.714208957 ^[[332m 2821^[[00m 0x7fe6d0009580 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:01.828812951 < 29:29:01.861577484)
29:29:14.334669592 ^[[332m 2821^[[00m 0x7fe63000baa0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:02.559625450 < 29:29:02.592959796)
29:29:14.334676014 ^[[332m 2821^[[00m 0x7fe63c02f0c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:02.559565302 < 29:29:02.592898668)
29:29:14.684190214 ^[[332m 2821^[[00m 0x7fe58401ee40 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:02.842110192 < 29:29:02.874993956)
29:29:14.688461119 ^[[332m 2821^[[00m 0x7fe64800e120 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:02.841962816 < 29:29:02.874844549)
29:29:14.688517163 ^[[332m 2821^[[00m 0x7fe64001ef60 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:02.843883877 < 29:29:02.876770552)
29:29:14.688704672 ^[[332m 2821^[[00m 0x7fe67c0038c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:02.843859732 < 29:29:02.876745866)
29:29:14.688781265 ^[[332m 2821^[[00m 0x7fe58401f640 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:02.841993207 < 29:29:02.874875572)
29:29:14.688903352 ^[[332m 2821^[[00m 0x7fe6d0009580 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:02.843915872 < 29:29:02.876802203)
29:29:15.306095340 ^[[332m 2821^[[00m 0x7fe63c02f0c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:03.559788453 < 29:29:03.593114648)
29:29:15.306357378 ^[[332m 2821^[[00m 0x7fe63000baa0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:03.559861069 < 29:29:03.593186804)
29:29:15.665368106 ^[[332m 2821^[[00m 0x7fe58401f640 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:03.854117307 < 29:29:03.887083956)
29:29:15.672252999 ^[[332m 2821^[[00m 0x7fe64800e120 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:03.854090565 < 29:29:03.887057543)
29:29:15.672253009 ^[[332m 2821^[[00m 0x7fe64001ef60 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:03.855908690 < 29:29:03.888878263)
29:29:15.672299636 ^[[332m 2821^[[00m 0x7fe6d0009580 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:03.855946863 < 29:29:03.888916526)
29:29:15.672708568 ^[[332m 2821^[[00m 0x7fe58401ee40 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:03.854215868 < 29:29:03.887182743)
29:29:15.672783808 ^[[332m 2821^[[00m 0x7fe67c0038c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:03.855894711 < 29:29:03.888864955)
29:29:16.332109491 ^[[332m 2821^[[00m 0x7fe63c02f0c0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:04.559821199 < 29:29:04.593154869)
29:29:16.332303663 ^[[332m 2821^[[00m 0x7fe63000baa0 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:04.559910591 < 29:29:04.593243727)
29:29:16.672725037 ^[[332m 2821^[[00m 0x7fe58401f640 ^[[33;01mWARN ^[[00m ^[[00m videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame:^[[00m decreasing timestamp (29:29:04.863812987 < 29:29:04.896878390)

Ignore the connection error because one camera is down during the test.

7

If I use nvurisrcbin with file://xxx , the gdb log shows no warnings. But if use rtsp source. log shows

0:01:10.554519748 7605 0x7f4950044980 WARN v4l2videodec gstv4l2videodec.c:1315:gst_v4l2_video_dec_loop: Decoder is producing too many buffers

Is that the cause of crash?

8

The above topic may face the same issue .

After another two day test.
the crash problems occured when If use GitHub - bluenviron/mediamtx: Ready-to-use SRT / WebRTC / RTSP / RTMP / LL-HLS media server and media proxy that allows to read, publish, proxy and record video and audio streams. for rtsp server.
But if I use real IP camera, the app has running for two days not crash. I am still watching…

9

Thanks for the update.
from the log “mem.log.tar.gz”, there is no obvious memory leak. here is the key log:
2023-08-08-14:03:49
VmRSS: 1935108 kB
VmSize: 21474772 kB

2023-08-09-19:37:06
VmRSS: 2000328 kB
VmSize: 21417588 kB

it is Gstreamer warning logs. please refer to Gstreamer code.

[quote=“shao77622, post:6, topic:262325”]
**This is the lastest log from gdb.log when it’'s crash, without any crash stack **
[/quote
when using gdb to debug, can you try “bt” to get the current call stack? and can you use “thread apply all bt” to get all thread 's call stack?

do you have any new findings? Thanks!

10

The rtsp source cause this problem. Ip Camera rtsp source run without memory leak and crash. Solved

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.