Error during run inference with custom yolov5 model in deepstream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

I am using jetson agx orin with deepstream 7.0.
I converted my model first to ONNX with the command python export.py --weights best.pt --img 640 --dynamic --simplify --device cpu --include onnx text and then to .engine with the command /usr/src/tensorrt/bin/trtexec --verbose --onnx=best.onnx --saveEngine=best.engine
This is my onnx model
I’m using the example → deepstream_python_apps/apps/deepstream-rtsp-in-rtsp-out at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

and I replaced the config file with the following

gpu-id=0
model-engine-file=best.engine
labelfile-path=labels.txt
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
num-detected-classes=3
gie-unique-id=1
uff-input-blob-name=input1
output-blob-names=output0
infer-dims=3;640;640

When i run the example i get the following error

INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT images          3x640x640
1   OUTPUT kFLOAT output0         25200x8

0:00:05.265346887 12037 0xaaaad5226290 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2198> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-7.0/sources/deepstream_python_apps/apps/deepstream-rtsp-in-rtsp-out/best.engine
0:00:05.275756026 12037 0xaaaad5226290 INFO                 nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:dstest1_pgie_config.txt sucessfully
Resetting source -1, attempts: 1
Warning: gst-stream-error-quark: No data from source since last 5 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1412): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin
Decodebin child added: depay

Decodebin child added: parser

Decodebin child added: tee_rtsp_pre_decode

Decodebin child added: dec_que

Decodebin child added: tee_rtsp_post_decode

Decodebin child added: decodebin

Decodebin child added: queue

Decodebin child added: nvvidconv

Decodebin child added: src_cap_filter_nvvidconv

In cb_newpad

gstname= video/x-raw
features= <Gst.CapsFeatures object at 0xffff90efc220 (GstCapsFeatures at 0xfffed8078500)>
Decodebin child added: h264parse0

Decodebin child added: capsfilter0

sys:1: Warning: g_object_set_is_valid_property: object class 'nvv4l2decoder' has no property named 'low-latency-mode'
sys:1: Warning: g_object_set_is_valid_property: object class 'nvv4l2decoder' has no property named 'extract-sei-type5-data'
sys:1: Warning: g_object_set_is_valid_property: object class 'nvv4l2decoder' has no property named 'sei-uuid'
Decodebin child added: nvv4l2decoder0

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
0:00:07.124151203 12037 0xaaaad529dcc0 ERROR                nvinfer gstnvinfer.cpp:676:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::parseBoundingBox() <nvdsinfer_context_impl_output_parsing.cpp:60> [UID = 1]: Could not find output coverage layer for parsing objects
0:00:07.124238769 12037 0xaaaad529dcc0 ERROR                nvinfer gstnvinfer.cpp:676:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::fillDetectionOutput() <nvdsinfer_context_impl_output_parsing.cpp:736> [UID = 1]: Failed to parse bboxes
Segmentation fault (core dumped)
3

You can refer to this repository.

Maybe there are some errors in onnx or engine file export .

4

I solved the following problem by creating a library and updating the config file as follows:

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
model-engine-file=/home/user/Downloads/Kira_test/models/yolo/yolov5s.engine
labelfile-path=labels.txt
batch-size=4
process-mode=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
num-detected-classes=80
gie-unique-id=1
network-type=0
output-blob-names=prob
parse-bbox-func-name=NvDsInferParseCustomYoloV5
custom-lib-path=/opt/nvidia/deepstream/deepstream/sources/objectDetector_Yolo/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so

[class-attrs-all]
nms-iou-threshold=0.5
pre-cluster-threshold=0.2

But the model does not perform inference correctly, it returns very different results from the original model. During the onnx → tensorrt conversion phase I receive the following warning Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32
Could this be due to this?

5

Try changing network-mode to 0 (fp32). Since you lack the int8 carlibration file, so only fp32 mode can be used.

network-mode=0
6

I tested the .engine model through the official yolov5 repo and it runs the inference correctly. Could the issue with the inference results be due to the post inference library?
The script I used is as follows
nvdsparsebbox_Yolo.txt (5.3 KB)

7

This post processing code looks incorrect. You need to parse the bbox correctly in post-processing

In addition to the above repositories, you can also refer to deepstream tao apps, which is officially supported.

8

I tested this repository → deepstream_tao_apps/post_processor/nvdsinfer_custombboxparser_tao.cpp at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub → with NvDsInferParseCustomBatchedYoloV5NMSTLT function.
But during inference I get this error:

Mismatch in number of output buffers. Expected 4 output buffers, detected in network :1
0:00:05.629949024 5703 0xaaaab1545f60 ERROR nvinfer gstnvinfer.cpp:676:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::fillDetectionOutput() <nvdsinfer_context_impl_output_parsing.cpp:727> [UID = 1]: Unable to parse bbox using custom parse function
Segmentation fault (core dumped)

I don’t understand where I’m going wrong.

  • During the model conversion phase from .pt to .engine?
  • In the post processing function?

Thanks for the support

9
  1. If you use this yolov5 repository, Please use export_yoloV5.py to export the onnx model and the corresponding post-processing library.

2.If you use the post-processing library provided by deepstream-tao-app, please follow the steps in README to export the model.

Both repositories have made modifications to yolov5. If you want to use your own yolov5 model, you also need to modify the corresponding post-processing code.

These post-processing libraries are probably incompatible with your model