MJPEG rtsp source cause memory leak

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) rtx3050
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only)
• TensorRT Version official deepstream 6.2 docker image
• NVIDIA GPU Driver Version (valid for GPU only) NVIDIA-Linux-x86_64-535.86.05
• Issue Type( questions, new requirements, bugs) bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

When I’m using some IP cameras with MJPEG h264 streams, my python deepstream-nvdsanalytics app appears memory leak, with VMRss continous increase. But with other cameras like hikvision, there is no memory leak . I’ve noticed that there is a jetson L4T patch that fix the mjpeg nvv4l2decoder memory leak issue, but not with dGpu. Please help me , thank you!

2023-08-28-17:26:29
VmRSS: 1439096 kB
VmSize: 17137560 kB

2023-08-28-17:26:59
VmRSS: 1440152 kB
VmSize: 17137560 kB

2023-08-28-17:27:29
VmRSS: 1440680 kB
VmSize: 17137560 kB

2023-08-28-17:27:59
VmRSS: 1442264 kB
VmSize: 17137560 kB

2023-08-28-17:28:29
VmRSS: 1443056 kB
VmSize: 17137560 kB

2023-08-28-17:28:59
VmRSS: 1444112 kB
VmSize: 17137560 kB

2023-08-28-17:29:29
VmRSS: 1446488 kB
VmSize: 17137560 kB

3

could you share the link about “here is a jetson L4T patch that fix the mjpeg nvv4l2decoder memory leak issue”? thanks!

5
6
  1. Is the source RTSP with MJPEG codec or H264 codec?
  2. please refer to this topic , setting drop-on-latency can improve.
7

I’ve already set drop-on-latency to 1

        if "source" in name:
            source_element = child_proxy.get_by_name("source")
            if source_element.find_property('drop-on-latency') != None:
                Object.set_property("drop-on-latency", True)

image
image1250×856 61 KB

I don’t know what the codec really is. It is a chinese LPR camera that widely used.

along with these gdb warnings.

0:00:07.108245152 3134 0x7f4cdc00ee40 WARN default descriptions.c:1233:gst_pb_utils_get_codec_description: No description available for media type: application/x-rtp
0:00:07.108491811 3134 0x7f4cdc00ee40 WARN uridecodebin gsturidecodebin.c:920:unknown_type_cb: warning: No decoder available for type ‘application/x-rtp, media=(string)vzinfo, payload=(int)108, clock-rate=(int)90000, encoding-name=(string)VND.ONVIF.METADATA, a-tool=(string)“LIVE555\ Streaming\ Media\ v2022.07.14”, a-type=(string)broadcast, x-qt-text-nam=(string)“RTSP/RTP\ stream\ from\ VZ\ Smart-IPC”, x-qt-text-inf=(string)h264, clock-base=(uint)1206952092, seqnum-base=(uint)23075, npt-start=(guint64)0, play-speed=(double)1, play-scale=(double)1, ssrc=(uint)3641389273’.
0:00:07.108603859 3134 0x7f4cdc00ee40 WARN default descriptions.c:1233:gst_pb_utils_get_codec_description: No description available for media type: application/x-rtp
0:00:07.108646539 3134 0x7f4cdc00ee40 WARN decodebin gstdecodebin2.c:4678:gst_decode_bin_expose: error: no suitable plugins found:
Missing decoder: application/x-rtp (application/x-rtp, media=(string)vzinfo, payload=(int)108, clock-rate=(int)90000, encoding-name=(string)VND.ONVIF.METADATA, a-tool=(string)“LIVE555\ Streaming\ Media\ v2022.07.14”, a-type=(string)broadcast, x-qt-text-nam=(string)“RTSP/RTP\ stream\ from\ VZ\ Smart-IPC”, x-qt-text-inf=(string)h264, clock-base=(uint)1206952092, seqnum-base=(uint)23075, npt-start=(guint64)0, play-speed=(double)1, play-scale=(double)1, ssrc=(uint)3641389273)

0:00:11.110380766 3134 0x7f4d14019180 WARN basesrc gstbasesrc.c:3072:gst_base_src_loop: error: Internal data stream error.
0:00:11.110415691 3134 0x7f4d14019180 WARN basesrc gstbasesrc.c:3072:gst_base_src_loop: error: streaming stopped, reason not-linked (-1)

8

please refer to this topic. can you use this method to get a memory leak log? and can you use this method to get a valgrind memory leak analysis log?

9

I will try, and do you mean I must use rtspsrc —rtph264depay - nvv4l2decoder, instead of simply use uridecodebin in python ?

10

It seemed that memory increaseed approximately 200mb and stop to glow when test with deepstream-nvdsanalytics.py decoding mjpeg.

11
  1. uridecodebin is an encapsulation of “rtspsrc —rtph264depay - nvv4l2decoder”. if you confirm uridecodebin has memory leak( you can use uridecodebin + fakesink to test), you can use “rtspsrc —rtph264depay - nvv4l2decoder” to simplify.
  2. can you share the source’s media information? you can use “gst-discoverer-1.0 rtsp://xxx” or “ffprobe rtsp://xxx”.
12

Input #0, rtsp, from ‘rtsp://192.168.8.79:554/h264’:
Metadata:
title : RTSP/RTP stream from VZ Smart-IPC
comment : h264
Duration: N/A, start: -17.794500, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 2304x1296, 25 fps, 25 tbr, 90k tbn, 180k tbc
Stream #0:1: Audio: pcm_alaw, 8000 Hz, 1 channels, s16, 64 kb/s

13
  1. how long the memory increaseed approximately 200mb?
    could you provide the monitor log and valgrind log by the method mentioned above? Thanks!
14

1 how long the memory increaseed approximately 200mb?

about 2 hours

2 could you provide the monitor log and valgrind log by the method mentioned above? Thanks!

I use command below, is that right?
valgrind --tool=memcheck --leak-check=full --num-callers=100 --show-leak-kinds=definite,indirect --track-origins=yes --log-file=valReport.log python3 deepstream_nvdsanalytics.py rtsp://192.168.8.79:554/h264 rtsp://192.168.8.79:554/h264 rtsp://192.168.8.79:554/h264

python3 nvmemstat.py -p valgrind > nvmemstat.log

there is some errors in deepstream6.3 docker container using nvmemstat.py
cat: /sys/kernel/debug/nvmap/iovmm/clients: No such file or directory

nvmemstat.log (59.9 KB)
valReport.log (2.0 MB)

15

Thanks for your help! here are some questions:

  1. were these two files the same time? did you modify deepstream-nvdsanalytics code?
    from VmRSS data in the nvmemstat.log, there are about 700MB(4049MB ~ 4718MB) memory leak in 7 minutes, but from the valgrind log, there are little definite memory leak during that time(definitely lost: 18,308 bytes in 32 blocks, indirectly lost: 10,145 bytes in 247 blocks).

  2. from the valgrind log. we find little memory leak in nvdsosd plugin, which is not opensource, we will fix it in the following release.

  3. to narrow down this issue, I also tested “gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test drop-on-latency=1 ! rtph264depay ! h264parse ! queue ! nvv4l2decoder ! fakesink”, there was about 4MB memory leak in 2 hours. the media information is 1080PX30fps. here is the log:
    vlgrind.log (26.9 KB)
    vmrss.txt (367.3 KB)
    we will test python deepstream-nvdsanalytics.

16

I also test python deepstream-nvdsanalytics with 1080PX30fps RTSP source. from py_monitor.log, there was about 8MB memory increase in 2 hours.
besides nvdsosd bug, Currently I did not find other bugs.

py_2098.log (367.5 KB)
py_val_2096.log (2.6 MB)

yes, at the beginning, some memory are reserved for runtime, after some time, the memory usage increase little.

17

Some camera source cause this problem，it is very weird. Hikvision is fine

---- Replied Message ----

From | Fanzh via NVIDIA Developer Forumsnotifications@nvidia.discoursemail.com |

  • | - |
    Date | 09/05/2023 15:52 |
    To | shao77622shao77622@163.com |
    Subject | [NVIDIA Developer Forums] [Intelligent Video Analytics/DeepStream SDK] MJPEG rtsp source cause memory leak |

| fanzh Moderator
September 5 |

  • | - |

I also test python deepstream-nvdsanalytics with 1080PX30fps RTSP source. from py_monitor.log, there was about 8MB memory increase in 2 hours.
besides nvdsosd bug, Currently I did not find other bugs.

py_2098.log (367.5 KB)
py_val_2096.log (2.6 MB)

shao77622:

It seemed that memory increaseed approximately 200mb and stop to glow when test with deepstream-nvdsanalytics.py decoding mjpeg.

yes, at the beginning, some memory are reserved for runtime, after some time, the memory usage increase little.

19

yes, it is weird. to narrow down this issue,

  1. could you share these camera’s brand and device model? Thanks!
  2. could you share these camera’s stream data? 3 minutes is enough. we will test on our side. you can use the following command to make a recording.
    gst-launch-1.0 rtspsrc location=rtsp://xxx ! rtph264depay ! h264parse ! mux. mpegtsmux name=mux ! filesink location=output.ts

nvdsosd plugin is already opensource. please refer to the fix of memory leak in nvdsosd.
fix.diff (927 Bytes)

20

output.ts (71.2 MB)

the camera brand and product information is here(chinese).
https://www.vzenith.com/case/parking-r3-productpage/

I’m using deepstream 6.3-samples docker
Do you fixed the nvdsosd leak issue?

21

Thanks for your help!

nvdsosd is opensource. the path is /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsosd/. you can apply the path above in your docker container, then rebuild and replace /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_osd.so with the new libnvdsgst_osd.so.
we will fix in the latter version not the old versions.

22

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.