Memory leak in generate_mask_polygon function

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) : Jetson
• DeepStream Version : DS7.0
• JetPack Version (valid for Jetson only) : Jetpack 6.0
• TensorRT Version : 8.6.2.3
• Issue Type( questions, new requirements, bugs) : questions

I am using a modified deepstream-mrcnn-test to run an instance segmentation application with RTSP.
I have discovered that there is a memory leak in the RSS when it runs for a long period.
Upon some investigation, I found out that the memory leak does not occur when I do not use the generate_mask_polygon function.

Since I have not customized the generate_mask_polygon function,
could you please tell me how to fix it so that it no longer leaks memory?

The graph on the left shows the RSS graph when the generate_mask_polygon function was not used, and the graph on the right shows the RSS graph when the generate_mask_polygon function was used.
The one using the generate_mask_polygon function clearly shows an increase in memory usage.

  • Log using generate_mask_polygon function
  • Log without generate_mask_polygon function

image
image1007×292 8.34 KB

In addition, the pipeline configuration was changed.
This is an image of the pipeline after the change.

mrcnn_playing2
mrcnn_playing212702×1177 921 KB

Docker container

How to reproduce

  1. Deliver RTSP.
    I am using sample_1080p_h264.mp4 for RTSP delivery.
sudo docker run --rm --name rtspserver0 -itd -p 554:554 -e MTX_RTSPADDRESS=":554" -e MTX_RTSPTRANSPORTS=tcp bluenviron/mediamtx:latest
ffmpeg -re -stream_loop -1 -i sample_1080p_h264.mp4 -c copy -f rtsp -rtsp_transport tcp rtsp://127.0.0.1:554/test.mpeg4 &
  1. Run
sudo docker run -itd --runtime=nvidia nvcr.io/nvidia/deepstream-l4t:7.0-triton-multiarch
sudo docker ps
sudo docker exec -it <container name> bash
./user_additional_install.sh
# https://forums.developer.nvidia.com/t/can-t-install-update-rtpmanager-sh-in-deepstream-7-0-docker/297573/6?u=y.majima
git clone https://github.com/GNOME/glib.git
cd glib
git checkout 2.76.6
meson build --prefix=/usr
ninja -C build/
cd build/
ninja install
cd ../..
vi update_rtpmanager.sh # Comment out apt-get -y install libglib2.0-dev
./update_rtpmanager.sh
cd sources/apps/sample_apps/deepstream-mrcnn-test/
apt-get install -y libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev \
   libgstrtspserver-1.0-dev libx11-dev libopencv-dev
vi deepstream_mrcnn_test.cpp # Modified to the source code I posted.
export CUDA_VER=12.2
make
git clone -b DS_7.0 https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
cd deepstream_reference_apps/deepstream_app_tao_configs/
apt install -y wget zip
cp -a * /opt/nvidia/deepstream/deepstream/samples/configs/tao_pretrained_models/
cd /opt/nvidia/deepstream/deepstream/samples/configs/tao_pretrained_models/
./download_models.sh
cd /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-mrcnn-test
./deepstream-mrcnn-test -i rtsp://<IP Address>:554/test.mpeg4 --no-display
3

Thanks for your analysis. The rootcause is that the g_array_new created array has not been released. You can try to release that in the meta_free_func function.

static void meta_free_func (gpointer data, gpointer user_data)
{
...
#ifdef WITH_OPENCV
if(obj->mask) {
  for (GList *iter = obj->mask; iter != NULL; iter = iter->next) {
    GArray *array = (GArray *)iter->data;
    g_array_free(array, TRUE);
  }
  g_list_free(obj->mask);
}
#endif
...
4

Thank you.
I added the fix and ran it for about 24 hours and the memory leak is resolved.
psinfo.log (2.7 MB)

image
image488×302 4.99 KB

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.