Double free or corruption (out) in Deepstream 7.1 in Jetson AGX Orin Jetpack 6.1 for custom secondary gie

[william21111340]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) : Jetson AGX Orin
• DeepStream Version : 7.1
• JetPack Version (valid for Jetson only) : 6.1
• TensorRT Version : 10.3

I had successfully implemented custom face swapping pipeline using inswapper model in Deepstream 7.0 , Jetpack 6.0. However, after updating Jetpack to 6.1 and Deepstream to 7.1, the pipeline throws double free error.

I have tried the migration guide following the link Application Migration to DeepStream 7.1 from DeepStream 7.0 — DeepStream documentation
but was showing couldn’t open custom library.
So, I updated the Makefile to include newer cuda and deepstream version and build the library.

After debuging using valgrind the following log is observed

==726201== Invalid free() / delete / delete / realloc()
==726201== at 0x4867AD0: free (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==726201== by 0x4E63A17: ??? (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x4E63093: nvds_destroy_batch_meta (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x48FDDEB: gst_buffer_foreach_meta (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x4903917: gst_buffer_pool_release_buffer (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x4903A3F: ??? (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x493B7F7: gst_mini_object_unref (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x1B0A646F: ??? (in /opt/nvidia/deepstream/deepstream-7.1/lib/gst-plugins/libnvdsgst_multistream.so)
==726201== by 0x4944A77: ??? (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x4947CB7: ??? (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x49480E7: gst_pad_push (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x36EB6F2F: ??? (in /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstcoreelements.so)
==726201== Address 0x43b03dc0 is 0 bytes inside a block of size 544 free’d
==726201== at 0x48682F0: operator delete(void*, unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==726201== by 0x3822AECB: ??? (in /opt/nvidia/deepstream/deepstream-7.1/lib/gst-plugins/libnvdsgst_infer.so)
==726201== by 0x4E64067: nvds_clear_meta_list (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x4E64243: ??? (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x4E64067: nvds_clear_meta_list (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x4E64107: ??? (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x4E639F7: ??? (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x4E6306B: nvds_destroy_batch_meta (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x48FDDEB: gst_buffer_foreach_meta (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x4903917: gst_buffer_pool_release_buffer (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x4903A3F: ??? (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x493B7F7: gst_mini_object_unref (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== Block was alloc’d at
==726201== at 0x4869F34: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==726201== by 0x4B51797: g_malloc0 (gmem.c:163)
==726201== by 0x4E63867: ??? (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x4E62FC7: nvds_create_batch_meta (in /opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_meta.so)
==726201== by 0x1B0A1E47: ??? (in /opt/nvidia/deepstream/deepstream-7.1/lib/gst-plugins/libnvdsgst_multistream.so)
==726201== by 0x497180B: ??? (in /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0.2003.0)
==726201== by 0x4B78AB7: g_thread_pool_thread_proxy (gthreadpool.c:350)
==726201== by 0x4B78063: g_thread_proxy (gthread.c:831)
==726201== by 0x5C1D5C7: start_thread (pthread_create.c:442)
==726201== by 0x5C85EDB: thread_start (clone.S:79)
==726201==

The issue occurs after the secondary inference and before the custom probe function within deepstream-app which does the post processing using the output tensor metadata from secondary inference.
below is my pipeline flow

[junshengy]

After simplification, does your pipeline look like the following? If you set output-tensor-meta to false first (ignore the results temporarily), will it still crash?

nvinfer(pgie) --> nvdspreprocess --> nvinfer input-tensor-meta=1 output-tensor-meta=1 classifier-async-mode=0  --> ....

[junshengy]

If the answer to the above is YES, please try the following patch

/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp

diff --git a/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp b/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp
index 4a16203..002ac5a 100644
--- a/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp
+++ b/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp
@@ -485,10 +485,10 @@ release_user_meta_at_frame_level (gpointer data, gpointer user_data)
     g_list_free(roi_meta->classifier_meta_list);
     roi_meta->classifier_meta_list = nullptr;
   }
-  if (roi_meta->object_meta) {
-    delete roi_meta->object_meta;
-    roi_meta->object_meta = nullptr;
-  }
+  // if (roi_meta->object_meta) {
+  //   delete roi_meta->object_meta;
+  //   roi_meta->object_meta = nullptr;
+  // }
 }

}

[william21111340]

Thank you so much. The double free error is resolved now.

[william21111340]

Double free error is resolved. But when I try to access the metadata from the custom probe function within deepstream-app, obj_meta->obj_user_meta_list is empty.

Am I missing something? Here is swapping config file.

[property]
gpu-id=0
operate-on-gie-id=1
gie-unique-id=4
input-tensor-from-meta=1
#onnx-file=inswapper_128.onnx
model-engine-file=inswapper.onnx_b1_gpu0_fp32.engine
batch-size=1
model-color-format=0 # RGB format
network-mode=0
process-mode=2 # Instance mask mode
network-type=3
interval=0
infer-dims=3;128;128
cluster-mode=4
custom-lib-path=parser/libnvdsinfer_custom_impl.so
parse-bbox-instance-mask-func-name=NvDsInferParseSwap
maintain-aspect-ratio=1
symmetric-padding=1
output-tensor-meta=1
output-instance-mask=0
debug-level=2
network-input-order=0
tensor-name=target

[class-attrs-all]
pre-cluster-threshold=0.5

[junshengy]

Please open a new topic to discuss this issue.

In addition, please describe in detail how you modified deepstream-app and the corresponding configuration files, and to which element’s pad (src/sink) the custom probe function was added.

The above information cannot determine any issues.