Deepstream osd plugin problem

I am trying to modify deepstream osd plugin i.e. /opt/nvidia/deepstream/deepstream-5.0/sources/gst-plugins/gst-nvdsosd/gstnvdsosd.c. There I am using nvll_osd_draw_segment_masks function to add some masks performed in the screen. It runs well. But since it started, the “host mem” shown in nvtop command is raising. I doubt whether it means a mem leak. When I remove the nvll_osd_draw_segment_masks funtion, the “mem leak” disappears. I am considering this “mem leak” may in the nvll_osd_draw_segment_masks funtion or I am misused this function. How can I solve this?

here is my system info:
Ubuntu 18.04.3
deepstream-app version 5.0
DeepStreamSDK 5.0
CUDA Driver Version: 11.0
CUDA Runtime Version: 10.2
TensorRT Version: 7.0
cuDNN Version: 7.6
libNVWarp360 Version: 2.0.1d3
Display Driver version: 450.80.02
Gstreamer version: 1.14.5
OpenCV version: 3.4.0

Hardware Platform GPU
Issue Type( question or bugs)

Have you observed the same phenomenon without you change?

Before I modify the osd plugin, host mem keeps unchanged for weeks. And also after changing the osd plugin, if I just commont the nvll_osd_draw_segment_masks function, the host mem keeps unchanged.
I checked the segmentation samples in the deepstream path only to find no sample contains the osd plugin. So I can not find a sample using osd plugin to draw masks to see whether the problem result in the the nvll_osd_draw_segment_masks function or the way I used it.

THere is a mrcnn sample you can use

The mrcnn sample using origin gst-nvdsosd plugin shows a similar result.When I set the draw_mask to TRUE, the host_mem increased obviously.

How long did you run your test, does the memory always increase?

The maskrcnn model test runs for 4 hours, if I check the host mem every ten minites, the value keeps increasing. After 4 hours the mem increases about 530MiB.

Thanks, can you share a detailed repro and how to check the memory with us, so we can debug locally.

sorry for late, I will sort and upload one this week.

deepstream-app-with-mrcnn.rar (124.9 KB)
That is the code. It can be built in the same folder with sample code deepstream-app. The README contains some instructions. Wish for you reply soon.

Any update? Does the code show a same result as I discribed?

sorry for the late, can you share what part did you modify, so I can do a quick check.

Also what’s the lib in your rar file you shared?

the origin code is from the sample code deepstream-app with few changes signed by “hhh.”. the lib is from the orgin code deepstream-mrcnn-app with no change and It provides a function called osd_sink_pad_buffer_probe. Below shown is already in the README.

  1. origin code is the sample deepstream-app
  2. code modified only in “Makefile” and “deepstream_app.c” is signed with “hhh.”
  3. use config “mrcnn_test.txt” as input
  4. The libdeepstream_mrcnn_test.so is build in the folder …/deepstream-mrcnn-app/ with origin sample code
  5. host_mem can be got with cmd “nvtop”,“ps -avx” or “top”.

Can I provide more infomation to help?

We had some fixes on latest DS version, could you use DS6.0 to see if the issue persist?

thanks for you reply, I will test it next week.

The program with DS6.0 runs for two hours, and shows a similar result, obvious host mem raising.

I try to repro the issue, but can not. this is the log run when app start for a few seconds and after 4 hour. i used deespstream 6.0EA, did we have any difference still? i got the code from comment 10# link.

root@c7751cad5c3c:/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-app-with-mrcnn# ps avx
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
1 pts/0 Ss 0:00 0 1037 17830 3800 0.0 /bin/bash
2138 pts/0 Tl 0:02 0 204 22297711 1573424 1.1 ./deepstream-app -c mrcnn_test.txt
2161 pts/0 R+ 0:00 0 109 25842 1472 0.0 ps avx
root@c7751cad5c3c:/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-app-with-mrcnn# ps avx
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
1 pts/0 Ss 0:00 0 1037 17830 3800 0.0 /bin/bash
2138 pts/0 Tl 0:02 0 204 22297711 1573424 1.1 ./deepstream-app -c mrcnn_test.txt

Is there an docker env available for 6.0 EA version?