Memory Leak - DS 5.1 - nvinfer

[sh2222]

We’re plagued by memory leaks during work on a 4x RTSP camera based person detection setup running on a Xavier NX.
To make sure that the leaks are not caused by us and that it is possible to easily reproduce them we created a simple command line GStreamer pipeline with a single RTSP camera as input. Please find attached the embedded_vision_inference_config.txt that makes use of a resnet18_peoplenet_pruned.etlt_b1_gpu0_int8.engine
embedded_vision_inference_config.txt (1.3 KB)

gst-launch-1.0 rtspsrc location="rtsp://192.168.0.246/mainstream" ! rtph264depay ! nvv4l2decoder disable-dpb=1 ! nvstreammux0.sink_0 nvstreammux name=nvstreammux0 batch-size=1 batched-push-timeout=0 width=1920 height=1080 live-source=TRUE ! nvvideoconvert ! nvinfer config-file-path="/home/nvidia/applications/embeddedvision/configurations/people_net_config/embedded_vision_inference_config.txt" ! nvmultistreamtiler width=1920 height=1080 qos=0 ! nvtracker tracker-width=240 tracker-height=200 ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_iou.so ll-config-file=/opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/iou_config.txt ! nvdsosd process-mode=0 ! nv3dsink sync=0

Checking /proc/PID/smaps reveals that the above pipeline leaks around 35MB per hour:

diff t13 t14=>
t13/t14
> 7f60000000-7f62349000 rw-p 00000000 00:00 0 
> Size:              36132 kB
> Rss:               36132 kB
> Pss:               36132 kB
5148,5151c5288,5291
< Private_Dirty:       248 kB
< Referenced:          248 kB
< Anonymous:           248 kB
< AnonHugePages:         0 kB
---
> Private_Dirty:     36132 kB
> Referenced:        36132 kB
> Anonymous:         36132 kB
> AnonHugePages:     18432 kB

7f60000000-7f62349000 rw-p 00000000 00:00 0 
Size:              36132 kB
Rss:               36132 kB
Pss:               36132 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     36132 kB
Referenced:        36132 kB
Anonymous:         36132 kB
AnonHugePages:     18432 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me nr

Using gdb -p PID to dump the memory segment does not help because there are no strings in the memory that could reveal the leaking process.

So, we decided to check whether the input or output elements cause the memory leak by using the following pipeline, which works flawlessly:
gst-launch-1.0 rtspsrc location="rtsp://192.168.0.246/mainstream" ! rtph264depay ! nvv4l2decoder disable-dpb=1 ! nvvideoconvert ! nv3dsink sync=0

Then we removed the nvtracker as well as the nvmultistreamtiler. This is possible because only one stream is used. But that pipeline still leaks the same amount of memory per hour:
gst-launch-1.0 rtspsrc location="rtsp://192.168.0.246/mainstream" ! rtph264depay ! nvv4l2decoder disable-dpb=1 ! nvstreammux0.sink_0 nvstreammux name=nvstreammux0 batch-size=1 batched-push-timeout=0 width=1920 height=1080 live-source=1 ! nvvideoconvert ! nvinfer config-file-path="/home/nvidia/applications/embeddedvision/configurations/people_net_config/embedded_vision_inference_config.txt" ! nv3dsink sync=0

The next step was to remove the nvinfer plugin from the pipeline to finally nail it down.
That pipeline does not show any memory leaks:
gst-launch-1.0 rtspsrc location="rtsp://192.168.0.246/mainstream" ! rtph264depay ! nvv4l2decoder disable-dpb=1 ! nvstreammux0.sink_0 nvstreammux name=nvstreammux0 batch-size=1 width=1920 height=1080 live-source=1 ! nvvideoconvert ! nv3dsink sync=0

So, after all those steps we would assume that the nvinfer plugin (version 5.1.0) is creating the memory leak we’re observing.
The same issue is present running the pipelines under DeepStream 5.0 on a RTX2080TI.

That is quite a critical issue for us because in our application DeepStream 5.1 needs to run for weeks without being restartet.

[sh2222]

Dear NVIDIA support, we would really appreciate to get feedback on this issue.
A memory leak is quite critical because it finally ends up in a crash.

[kayccc]

Moving into DeepStream SDK forum for resolution.

[Fiona.Chen]

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)