I have been trying to implement a custom trained yolo model in deepstream-occupancy-analytics app, i have trained the model, converted the model, and made few changes to the configuration file. My aim is to implement a yolo model which detects both two-wheelers and four-wheelers. Now I’m stuck with this error:
root@180880ced8ff:/opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics# ./deepstream-test5-analytics -c config/dstest_occupancy_analytics.txt
(deepstream-test5-analytics:160): GLib-CRITICAL **: 06:04:30.449: g_strchug: assertion 'string != NULL' failed
(deepstream-test5-analytics:160): GLib-CRITICAL **: 06:04:30.449: g_strchomp: assertion 'string != NULL' failed
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvTrackerParams::getConfigRoot()] !!![WARNING] File doesn't exist. Will go ahead with default values
[NvMultiObjectTracker] Initialized
0:00:00.947912459 160 0x55fea632b9d0 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 4]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2109> [UID = 4]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
Building the TensorRT Engine
Building complete
0:01:59.015181030 160 0x55fea632b9d0 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 4]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2141> [UID = 4]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/model_b1_gpu0_fp32.engine successfully
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:612 [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT images 3x640x640
1 OUTPUT kFLOAT output0 25200x10
0:01:59.049514810 160 0x55fea632b9d0 INFO nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<primary_gie> [UID 4]: Load new model:/opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/config/pgie_peoplenet_tao_config.txt sucessfully
Runtime commands:
h: Print this help
q: Quit
p: Pause
r: Resume
NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.
**PERF: FPS 0 (Avg)
Thu Oct 17 06:06:29 2024
**PERF: 0.00 (0.00)
** INFO: <bus_callback:291>: Pipeline ready
** INFO: <bus_callback:277>: Pipeline running
mimetype is video/x-raw
Segmentation fault (core dumped)
root@1b778279e623:/opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics# gdb --args ./deepstream-test5-analytics -c config/dstest_occupancy_analytics.txt
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./deepstream-test5-analytics...
(No debugging symbols found in ./deepstream-test5-analytics)
(gdb) run
Starting program: /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/deepstream-test5-analytics -c config/dstest_occupancy_analytics.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffd79ff000 (LWP 179)]
[New Thread 0x7fffd0f5d000 (LWP 180)]
(deepstream-test5-analytics:170): GLib-CRITICAL **: 13:30:20.156: g_strchug: assertion 'string != NULL' failed
(deepstream-test5-analytics:170): GLib-CRITICAL **: 13:30:20.156: g_strchomp: assertion 'string != NULL' failed
[New Thread 0x7fffbcb42000 (LWP 181)]
[New Thread 0x7fff79dff000 (LWP 182)]
[New Thread 0x7fff795fe000 (LWP 183)]
[New Thread 0x7fff78dfd000 (LWP 184)]
[New Thread 0x7fff6bfff000 (LWP 185)]
[New Thread 0x7fff5d5ff000 (LWP 186)]
[New Thread 0x7fff5cdfe000 (LWP 187)]
[New Thread 0x7fff51fff000 (LWP 188)]
[New Thread 0x7fff517fe000 (LWP 189)]
[New Thread 0x7fff50ffd000 (LWP 190)]
[New Thread 0x7fff43fff000 (LWP 191)]
[New Thread 0x7fff437fe000 (LWP 192)]
[New Thread 0x7fff42ffd000 (LWP 193)]
[New Thread 0x7fff427fc000 (LWP 194)]
[New Thread 0x7fff41ffb000 (LWP 195)]
[New Thread 0x7fff417fa000 (LWP 196)]
[New Thread 0x7fff40ff9000 (LWP 197)]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvTrackerParams::getConfigRoot()] !!![WARNING] File doesn't exist. Will go ahead with default values
[NvMultiObjectTracker] Initialized
[New Thread 0x7fff0c47f000 (LWP 198)]
[New Thread 0x7fff03c7e000 (LWP 199)]
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:1494 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/config/model_b1_gpu0_fp32.engine open error
0:00:12.254612371 170 0x55556b4d0640 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2083> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/config/model_b1_gpu0_fp32.engine failed
0:00:12.284850550 170 0x55556b4d0640 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2188> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/config/model_b1_gpu0_fp32.engine failed, try rebuild
0:00:12.285059950 170 0x55556b4d0640 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2109> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
Building the TensorRT Engine
Building complete
0:02:14.371469427 170 0x55556b4d0640 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2141> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/model_b1_gpu0_fp32.engine successfully
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:612 [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT images 3x640x640
1 OUTPUT kFLOAT output0 25200x10
[New Thread 0x7fff0bc7e000 (LWP 200)]
[New Thread 0x7fff0b47d000 (LWP 201)]
[New Thread 0x7fff0ac7c000 (LWP 202)]
0:02:14.560847243 170 0x55556b4d0640 INFO nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/config/pgie_peoplenet_tao_config.txt sucessfully
[New Thread 0x7fff0a47b000 (LWP 203)]
[New Thread 0x7fff09c7a000 (LWP 204)]
[New Thread 0x7fff09479000 (LWP 205)]
[New Thread 0x7fff08c78000 (LWP 206)]
[New Thread 0x7fff0347d000 (LWP 207)]
Runtime commands:
h: Print this help
q: Quit
p: Pause
r: Resume
NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.
**PERF: FPS 0 (Avg)
Fri Oct 18 13:32:34 2024
**PERF: 0.00 (0.00)
** INFO: <bus_callback:291>: Pipeline ready
[New Thread 0x7fff02c7c000 (LWP 208)]
[New Thread 0x7fff0247b000 (LWP 209)]
[New Thread 0x7fff00fff000 (LWP 210)]
[New Thread 0x7ffecffff000 (LWP 211)]
** INFO: <bus_callback:277>: Pipeline running
[New Thread 0x7ffecf7fe000 (LWP 212)]
mimetype is video/x-raw
[New Thread 0x7ffeceffd000 (LWP 213)]
[New Thread 0x7ffecd2ce000 (LWP 214)]
[New Thread 0x7ffeccacd000 (LWP 215)]
[New Thread 0x7ffebcad1000 (LWP 216)]
Thread 23 "deepstream-test" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff0bc7e000 (LWP 200)]
0x00007fffbc089346 in decodeTensorYolo(float const*, float const*, float const*, unsigned int const&, unsigned int const&, unsigned int const&, std::vector<float, std::allocator<float> > const&) () from /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/DeepStream-Yolo/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
(gdb) bt
#0 0x00007fffbc089346 in decodeTensorYolo(float const*, float const*, float const*, unsigned int const&, unsigned int const&, unsigned int const&, std::vector<float, std::allocator<float> > const&) ()
at /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/DeepStream-Yolo/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
#1 0x00007fffbc0897c9 in NvDsInferParseCustomYolo(std::vector<NvDsInferLayerInfo, std::allocator<NvDsInferLayerInfo> > const&, NvDsInferNetworkInfo const&, NvDsInferParseDetectionParams const&, std::vector<NvDsInferObjectDetectionInfo, std::allocator<NvDsInferObjectDetectionInfo> >&) ()
at /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/DeepStream-Yolo/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
#2 0x00007fffbc089aef in NvDsInferParseYolo ()
at /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-trafficcam-occupancy-analytics/DeepStream-Yolo/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
#3 0x00007fffbe853d9c in nvdsinfer::DetectPostprocessor::fillDetectionOutput(std::vector<NvDsInferLayerInfo, std::allocator<NvDsInferLayerInfo> > const&, NvDsInferDetectionOutput&) () at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer.so
#4 0x00007fffbe8305d7 in nvdsinfer::DetectPostprocessor::parseEachBatch(std::vector<NvDsInferLayerInfo, std::allocator<NvDsInferLayerInfo> > const&, NvDsInferFrameOutput&) () at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer.so
#5 0x00007fffbe837b0a in nvdsinfer::InferPostprocessor::postProcessHost(nvdsinfer::NvDsInferBatch&, NvDsInferContextBatchOutput&) ()
at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer.so
#6 0x00007fffbe8327e8 in nvdsinfer::NvDsInferContextImpl::dequeueOutputBatch(NvDsInferContextBatchOutput&) ()
at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer.so
#7 0x00007fffd00a4d0d in () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
#8 0x00007ffff778aac1 in g_thread_proxy (data=0x55556fb2b4f0) at ../glib/gthread.c:831
#9 0x00007ffff7259ac3 in () at /usr/lib/x86_64-linux-gnu/libc.so.6
#10 0x00007ffff72eb850 in () at /usr/lib/x86_64-linux-gnu/libc.so.6
(gdb)
From the stack address of the crash, it appears that there is something wrong with your postprocess function decodeTensorYolo. You can add some log info and debug that.
I couldn’t find the cause of the issue, so I decided to try a different approach. I trained my own model using the TAO Toolkit and used that model instead, which is now working perfectly.