In Deepstream 7.1 GST pipeline RE-ID model fails

Details of Setup

  • Hardware Platform (Jetson / GPU): GPU
  • DeepStream Version: 7.1.0
  • TensorRT Version: 10.7.0.23-1+cuda12.6 amd64
  • NVIDIA GPU Driver Version (valid for GPU only): 560.35.03 (CUDA Version: 12.6)

Issue Type: Bug

Description of Issue:

When using either resnet50_market1501.etlt or the open source ghost_reid.onnx Re-ID Models for tracking I’m getting the following backtraces:

resnet:

[New Thread 0x756de9e00640 (LWP 286)]
[New Thread 0x756deea00640 (LWP 287)]
[New Thread 0x756def400640 (LWP 288)]
0:01:00.835279545   141 0x62631d2f37c0 INFO                 nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<lp_detector> [UID 2]: Load new model:/home/ds_tracker/component_configs/config_infer_secondary_yoloV7_lp.txt sucessfully
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 552]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:922 INT8 calibration file not specified. Trying FP16 mode.
[NvMultiObjectTracker] Serialized plan file cached at location: /home/ds_tracker/models/ReID/resnet50_market1501.etlt_b100_gpu0_fp16.engine
[NvMultiObjectTracker] Initialized
[New Thread 0x756defe00640 (LWP 329)]
[New Thread 0x756e01800640 (LWP 330)]

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x0000756ed081e550 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) backtrace
#0  0x0000756ed081e550 in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x0000756ed081e8b1 in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000756ed081e9c7 in __gxx_personality_v0 () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x0000756eceb96fe9 in __libunwind_Unwind_Resume () at /usr/lib/x86_64-linux-gnu/libunwind.so.8
#4  0x0000756e5b7e1d1e in  () at /usr/lib/x86_64-linux-gnu/libnvinfer.so.10
#5  0x0000756e5c6ebacc in  () at /usr/lib/x86_64-linux-gnu/libnvinfer.so.10
#6  0x0000756e5c6ec260 in  () at /usr/lib/x86_64-linux-gnu/libnvinfer.so.10
#7  0x0000756e5c7269c8 in  () at /usr/lib/x86_64-linux-gnu/libnvinfer.so.10
#8  0x0000756e5c727426 in  () at /usr/lib/x86_64-linux-gnu/libnvinfer.so.10
#9  0x0000756e5c7274be in  () at /usr/lib/x86_64-linux-gnu/libnvinfer.so.10
#10 0x0000756eac79c258 in  () at /opt/nvidia/deepstream/deepstream/lib/libnvds_infer.so
#11 0x0000756eac773a56 in nvdsinfer::NvDsInferContextImpl::deserializeEngineAndBackend(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::shared_ptr<nvdsinfer::TrtEngine>&, std::unique_ptr<nvdsinfer::BackendContext, std::default_delete<nvdsinfer::BackendContext> >&) () at /opt/nvidia/deepstream/deepstream/lib/libnvds_infer.so
#12 0x0000756eac774873 in nvdsinfer::NvDsInferContextImpl::generateBackendContext(_NvDsInferContextInitParams&) ()
    at /opt/nvidia/deepstream/deepstream/lib/libnvds_infer.so
#13 0x0000756eac77b774 in nvdsinfer::NvDsInferContextImpl::initialize(_NvDsInferContextInitParams&, void*, void (*)(INvDsInferContext*, unsigned int, NvDsInferLogLevel, char const*, void*)) () at /opt/nvidia/deepstream/deepstream/lib/libnvds_infer.so
#14 0x0000756eac77c0ee in createNvDsInferContext(INvDsInferContext**, _NvDsInferContextInitParams&, void*, void (*)(INvDsInferContext*, unsigned int, NvDsInferLogLevel, char const*, void*)) () at /opt/nvidia/deepstream/deepstream/lib/libnvds_infer.so
#15 0x0000756eaf0cff48 in  () at /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_infer.so
#16 0x0000756ecea07441 in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#17 0x0000756ecea07675 in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#18 0x0000756ecec677df in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#19 0x0000756ecec6808c in gst_pad_set_active () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#20 0x0000756ecec4f1f5 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#21 0x0000756ecec5840b in gst_iterator_fold () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#22 0x0000756ececc634a in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#23 0x0000756ecec4caf6 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#24 0x0000756ecec4f4b0 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#25 0x0000756ecec4e7c9 in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#26 0x0000756ecec4eed5 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#27 0x0000756ecec2788c in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#28 0x0000756ecec7a826 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#29 0x0000756ecec4e7c9 in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#30 0x0000756ecec4e80f in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#31 0x0000756ecec4eed5 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#32 0x0000756ed135fe2e in  () at /usr/lib/x86_64-linux-gnu/libffi.so.8
#33 0x0000756ed135c493 in  () at /usr/lib/x86_64-linux-gnu/libffi.so.8
#34 0x0000756ed0ec5722 in  () at /usr/lib/python3/dist-packages/gi/_gi.cpython-310-x86_64-linux-gnu.so
#35 0x0000756ed0ec3826 in  () at /usr/lib/python3/dist-packages/gi/_gi.cpython-310-x86_64-linux-gnu.so
#36 0x0000756ed0ec3a7d in  () at /usr/lib/python3/dist-packages/gi/_gi.cpython-310-x86_64-linux-gnu.so
#37 0x00006262e5e2639b in _PyObject_MakeTpCall ()
#38 0x00006262e5e20009 in _PyEval_EvalFrameDefault ()
#39 0x00006262e5e3038c in _PyFunction_Vectorcall ()
#40 0x00006262e5e1a97f in _PyEval_EvalFrameDefault ()
#41 0x00006262e5e3038c in _PyFunction_Vectorcall ()
#42 0x00006262e5e1a790 in _PyEval_EvalFrameDefault ()
#43 0x00006262e5effa16 in  ()
#44 0x00006262e5eff8e6 in PyEval_EvalCode ()
#45 0x00006262e5f26108 in  ()
#46 0x00006262e5f2090f in  ()
#47 0x00006262e5f25ea5 in  ()
#48 0x00006262e5f25488 in _PyRun_SimpleFileObject ()
#49 0x00006262e5f25167 in _PyRun_AnyFileObject ()
#50 0x00006262e5f198ae in Py_RunMain ()
#51 0x00006262e5ef393d in Py_BytesMain ()
#52 0x0000756ed165cd90 in  () at /usr/lib/x86_64-linux-gnu/libc.so.6
#53 0x0000756ed165ce40 in __libc_start_main () at /usr/lib/x86_64-linux-gnu/libc.so.6
#54 0x00006262e5ef3835 in _start ()

onnx:

0:00:53.335015207  2598 0x60082607f3e0 INFO                 nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<lp_detector> [UID 2]: Load new model:/home/ds_tracker/component_configs/config_infer_secondary_yoloV7_lp.txt sucessfully
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Initialized
[New Thread 0x7e0a01400640 (LWP 2718)]
[New Thread 0x7e0a20e00640 (LWP 2719)]

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x00007e0acbc99550 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

What may cause this issue?

Seems setup issue. Can you setup env based on: Installation — DeepStream documentation

You must install the following components:

  • Ubuntu 22.04
  • GStreamer 1.20.3
  • NVIDIA driver 535.183.06 (for Data Center GPUs) and 560.35.03 (for RTX GPUs)
  • CUDA 12.6
  • TensorRT 10.3.0.26

Thank you! The root of the problem was most likely the mismatching versions in ONNX and engine files. Regenerating them in the same environment solved the issue for us.