Memory leak in split pipeline

y.majima · December 17, 2025, 8:58am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): Jetson
• DeepStream Version: DS7.1
• JetPack Version (valid for Jetson only): 6.1
• TensorRT Version: 10.3
• Issue Type( questions, new requirements, bugs): questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

When I ran this pipeline for about 3 hours , I was able to verify by using the ps -aux command that it kept raising the physical memory usage without staying at a constant value.
Please tell me how to prevent memory leaks.

The following is a log showing the physical memory usage of gst-launch-1.0 obtained using the ps -aux command.
psinfo.gst2.log (10.6 MB)

I run this command in the docker container.

nvcr.io/nvidia/deepstream:7.1-samples-multiarch

commnad

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! \
    tee ! \
    nvstreamdemux name=demux \
    demux.src_0 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_1 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_2 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_3 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_4 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_5 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_6 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_7 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_8 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_9 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_10 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_11 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_12 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_13 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_14 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0 \
    demux.src_15 ! queue ! tee ! queue ! nv3dsink sync=0 async=0 enable-last-sample=0

junshengy · December 18, 2025, 8:54am

1.glibc may exhibit some memory usage fluctuations.

2.new nvstreamux is open source, If you suspect a memory leak in new nvstreammux, you can add the following two makefiles to the compilation options and then recompile.

This will allow you to use valgrind to check the number of potentially leaking function lines.

/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmultistream2/Makefile and /opt/nvidia/deepstream/deepstream/sources/libs/nvstreammux/Makefile

CFLAGS+= ....
+	 -O0 \
+	 -g

3.As an alternative, you can use videotestsrc to run continuously for 3 hours and then exit.

valgrind videotestsrc num-buffers=54000 pattern=0 ! video/x-raw,width=1280,height=720,framerate=5/1,format=NV12 ! ...

y.majima · December 23, 2025, 7:15am

Thank you for your reply.

Add options to the two Makefile, I ran Valgrind using videotestsrc.
The following is the Valgrind log.
Is there a memory leak in nvstreammux?
valgrind_nvstreammux.txt (638.5 KB)

junshengy · December 23, 2025, 11:38am

Only very minor leaks usually do not require treatment.

LEAK SUMMARY:
   definitely lost: 394,426 bytes in 355 blocks
   indirectly lost: 51,067 bytes in 682 blocks
     possibly lost: 508,912 bytes in 90 blocks
   still reachable: 16,731,348 bytes in 16,119 blocks

Main Persistent Memory Leaks:

118 bytes in 16 blocks (records 2719-2725) - Occurs multiple times, from gst_object_get_name in gst_nvstreammux_sink_query
528 bytes in 16 blocks (records 3435-3436) - From gst_nvstreamdemux_sink_event and gst_nvstreammux_sink_event
9,248 bytes in 16 blocks (record 3896) - From nvvideosinks library

Large Single-Instance Leaks:
72,704 bytes - From dlopen in nvtracker library

If you are very concerned about these leaks, please try the following patch and recompile nvstreammux/nvtracker. The remaining definitely lost data is not caused by deepstream.

diff --git a/sources/gst-plugins/gst-nvmultistream2/gstnvstreamdemux.cpp b/sources/gst-plugins/gst-nvmultistream2/gstnvstreamdemux.cpp
index 3d8c18f..cd02fff 100644
--- a/sources/gst-plugins/gst-nvmultistream2/gstnvstreamdemux.cpp
+++ b/sources/gst-plugins/gst-nvmultistream2/gstnvstreamdemux.cpp
@@ -796,6 +796,7 @@ gst_nvstreamdemux_sink_event (GstPad * pad, GstObject * parent,
 	  g_mutex_unlock (&nvstreamdemux->ctx_lock);
 	  //GST_OBJECT_UNLOCK (nvstreamdemux);
 
+	  gst_event_unref (event);
 	  return ret;
   }
 
@@ -816,6 +817,7 @@ gst_nvstreamdemux_sink_event (GstPad * pad, GstObject * parent,
     LOGD("sending eos event on pad %d\n", source_id);
     gboolean ret = gst_pad_push_event (src_pad, gst_event_new_eos ());
     LOGD("eos send ret=%d\n", ret);
+    gst_event_unref (event);
     return ret;
   }
 
@@ -835,6 +837,7 @@ gst_nvstreamdemux_sink_event (GstPad * pad, GstObject * parent,
             stream_id + (char *)NULL));
       if (!src_pad)
       {
+        gst_event_unref (event);
         return TRUE;
       }
       ret = gst_pad_push_event (src_pad, event);
@@ -843,6 +846,7 @@ gst_nvstreamdemux_sink_event (GstPad * pad, GstObject * parent,
       return ret;
     }
     gst_message_unref(msg);
+    gst_event_unref (event);
   }
 
   if (GST_EVENT_TYPE (event) == GST_NVEVENT_STREAM_SEGMENT) {
diff --git a/sources/gst-plugins/gst-nvmultistream2/gstnvstreammux.cpp b/sources/gst-plugins/gst-nvmultistream2/gstnvstreammux.cpp
index 5d403da..befe88c 100644
--- a/sources/gst-plugins/gst-nvmultistream2/gstnvstreammux.cpp
+++ b/sources/gst-plugins/gst-nvmultistream2/gstnvstreammux.cpp
@@ -606,7 +606,7 @@ static gboolean
 gst_nvstreammux_sink_query (GstPad * pad, GstObject * parent, GstQuery * query)
 {
   GstNvStreamMux *mux = GST_NVSTREAMMUX (parent);
-  gchar * name = gst_pad_get_name(pad);
+  gchar * name = GST_PAD_NAME(pad);
   guint stream_index = 0;
   GstQuery *peer_query;
   GstCaps* sink_caps;
@@ -615,7 +615,7 @@ gst_nvstreammux_sink_query (GstPad * pad, GstObject * parent, GstQuery * query)
     return FALSE;
   }
   else if (sscanf (name, "sink_%u", &stream_index) < 1){
-    g_free(name);
+    // g_free(name);
     return FALSE;
   }
 
@@ -1120,11 +1120,13 @@ gst_nvstreammux_sink_event (GstPad * pad, GstObject * parent, GstEvent * event)
     if( name == NULL )
     {
         GST_DEBUG_OBJECT(mux, "Streammux sink event name null\n");
+        return FALSE;
     }
     else if (sscanf (name, "sink_%u", &stream_index) < 1)
     {
         GST_DEBUG_OBJECT(mux, "Streammux sink event name invalid\n");
         g_free(name);
+        return FALSE;
     }
     else
     {
diff --git a/sources/gst-plugins/gst-nvtracker/gstnvtracker.cpp b/sources/gst-plugins/gst-nvtracker/gstnvtracker.cpp
index 78c1ec0..a3b4ceb 100644
--- a/sources/gst-plugins/gst-nvtracker/gstnvtracker.cpp
+++ b/sources/gst-plugins/gst-nvtracker/gstnvtracker.cpp
@@ -317,13 +317,7 @@ static gboolean gst_nv_tracker_start (GstBaseTransform * btrans)
 
 static gboolean gst_nv_tracker_stop (GstBaseTransform * btrans)
 {
-  return TRUE;
-}
-
-static void gst_nv_tracker_finalize (GObject * object)
-{
-  GstNvTracker *nvtracker = GST_NVTRACKER (object);
-
+  GstNvTracker *nvtracker = GST_NVTRACKER (btrans);
   /** De-init the low-level threads and plugin */
   if(nvtracker->trackerIface) {
     nvtracker->trackerIface->deInit();
@@ -339,6 +333,12 @@ static void gst_nv_tracker_finalize (GObject * object)
     g_thread_join(nvtracker->output_loop);
   }
   nvtracker->output_loop = NULL;
+  return TRUE;
+}
+
+static void gst_nv_tracker_finalize (GObject * object)
+{
+  GstNvTracker *nvtracker = GST_NVTRACKER (object);
 
   if(nvtracker->trackerIface) {
     delete nvtracker->trackerIface;

r.takemoto · January 9, 2026, 9:32am

Please advise on the cause.

I applied the patch you provided and ran it again.
I performed three runs under identical conditions and monitored the RSS feed.
However, I observed different results between the three runs.
Below is the RSS log during runs.

First run on Jetson Orin Nano.
Nano_full-patch-1.zip (673.0 KB)

Sencond run on Jetson Orin Nano.
Nano_full-patch-2.zip (482.2 KB)

I also ran it on AGX Orin.
Orin_full-patch.zip (209.1 KB)

junshengy · January 12, 2026, 6:33am

Although the command line are same, and the running times are similar, the difference in the number of objects detected/tracked will lead to different memory usages. and the memory tend to plateau at least, I think it is normal.

Topic		Replies	Views
Question on nvtracker memory leak DeepStream SDK	5	753	April 5, 2022
Memory leak using new nvstreammux DeepStream SDK nvbugs	7	351	September 11, 2024
Memory leakage in nvstreammux + nvmultistreamtiler DeepStream SDK	5	672	October 12, 2021
Memory leak with stationary objects when Multi-Object Tracking DeepStream SDK deepstream61	17	1634	August 16, 2022
Memory Leak in Nvv4l2decoder DeepStream SDK nvbugs	3	850	January 10, 2024
Memory Leak in Deepstream parallel branches DeepStream SDK deepstream	11	241	August 18, 2025
Nvstreammux_alpha cost memory very much and fast DeepStream SDK	4	552	October 12, 2021
Is there some unfreed memory in grtreamer or deepstream plugin ? DeepStream SDK	9	1682	April 27, 2020
Memory leak caused by nvtracker in DeepStream5.1 DeepStream SDK	7	509	January 16, 2024
Memory leak in DeepStream DeepStream SDK	33	6220	June 27, 2022

Memory leak in split pipeline

Related topics