Yolov9 Model Engine Build Failure with nvinfer

ronpai · November 4, 2024, 9:55am

Device Information:

Hardware Platform (Jetson / GPU) GPU
DeepStream Version 7.0
TensorRT Version 8.6
NVIDIA GPU Driver Version (valid for GPU only) 550.120
Issue Type( questions, new requirements, bugs) questions, bugs

We encountered challenges where our model engine fails to build using nvinfer.
We run deepstream-test5-app with the configs below and our model.plan and custombbox parser

deepstream pipeline pgie setup

[primary-gie]
enable=1
gpu-id=0
#(0): nvinfer - Default inference plugin based on Tensorrt
#(1): nvinferserver - inference plugin based on Tensorrt-Inference-Server
plugin-type=0
model-engine-file=/opt/nvidia/deepstream/deepstream-7.0/model.plan
batch-size=32
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=/opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-test5/configs/config_infer_yolo.txt

pgie config

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
#onnx-file=models/yolov9-c-converted-trt.onnx
model-engine-file=/opt/nvidia/deepstream/deepstream-7.0/model.plan
labelfile-path=/opt/nvidia/deepstream/deepstream-7.0/labels.txt
batch-size=32
infer-dims=640;640;3
force-implicit-batch-dim=0
# 0: FP32 1: INT8 2: FP16
network-mode=2
num-detected-classes=123
interval=0
gie-unique-id=1
#process-mode=1
# 0: Detector 1: Classifier 2: Segmentation 3: Instance Segmentation
network-type=0
cluster-mode=2
maintain-aspect-ratio=0
parse-bbox-func-name=CreateInferServerCustomProcess
custom-lib-path=/opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infercustomparser.so

[class-attrs-all]
pre-cluster-threshold=0.25

When running deepstream-test5-app, it fails to build our yolov9 model engine

We’ve tested the same config but with nvinferserver plugin (host model with triton on our own) and it worked properly.

Please help with the error in nvinfer engine build, thanks!

fanzh · November 5, 2024, 2:51am

from the error, the app failed to load the engine because unknown data type. nvinfer plugin is opensource. you can find this error in TrtEngine::getLayerInfo of \opt\nvidia\deepstream\deepstream\sources\libs\nvdsinfer\nvdsinfer_model_builder.cpp. you can add log to print the data type. if int64, this issue has been fixed in DS7.1. can you use DS7.1 or port the logics?

ronpai · November 6, 2024, 8:10am

Thanks for the quick response.
We will test with DeepStream 7.0 for now since our model and production setup are under TensorRT 8.6.

I’ve added logs to print the data type, and it shows type=5. Is there any further steps we can take for debugging?

fanzh · November 6, 2024, 9:31am

“type: 5” means “kUINT8 = 5,”. DS7.1 added a new type INT64, you can refer to this code. To be specific,

use the following steps to modify the code and build nvinfer low-level lib. especially please backup the original so first.
1.1 add a new UINT8 type in NvDsInferDataType of opt\nvidia\deepstream\deepstream\sources\includes\nvdsinfer.h
1.2 add a new UINT8 type in getElementSize of \opt\nvidia\deepstream\deepstream\sources\libs\nvdsinfer\nvdsinfer_func_utils.h.
1.3 add a new UINT8 type in dataType2Str of \opt\nvidia\deepstream\deepstream\sources\libs\nvdsinfer\nvdsinfer_func_utils.cpp
1.4 add a new UINT8 type in getLayerInfo of \opt\nvidia\deepstream\deepstream\sources\libs\nvdsinfer\nvdsinfer_model_builder.cpp
1.5 build nvifner, then replace /opt/nvidia/deepstream/deepstream/lib/libnvds_infer.so with the new so.
use the following steps to modify the code and build nvinfer plugin.
2.1 add a new UINT8 type in get_element_size of \opt\nvidia\deepstream\deepstream\sources\gst-plugins\gst-nvinfer\gstnvinfer.cpp
2.2 build nvinfer plugin, then replace libnvdsgst_infer.so with the new so.

carlosgs · November 6, 2024, 10:44pm

Let us know if this answers the question.

fanzh · November 7, 2024, 2:35am

I generated a yolov9 onnx model from this link by the following cmd, all the layer types are float32. from your screenshot, the input layer type is uint8. could you share the detailed steps to generate yolov9 model and TRT engine with uint8 type? Thanks! or could you share the onnx model by forum private email?

python3 export.py --weights yolov9-c-converted.pt --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640 --include onnx

ronpai · November 8, 2024, 4:23am

I followed the steps and got the input format error

This is the same issue I found in 2021, and the response was that DeepStream pipeline only supports NCHW format.

I checked our model, and the format is NHWC

I would like to know whether NHWC is supported with DeepStream 7.0 and how should I modify the configs, thanks!

fanzh · November 8, 2024, 5:28am

DeepStream7.0 nvinfer can support NHWC. please add the following configuration for nvinfer.

# 0=NCHW, 1=NHWC, 2=CUSTOM
network-input-order=1

Since the collor_space is BRG please add the following nvinfer configuration.

    # 0=RGB, 1=BGR, 2=GRAY
model-color-format=1

Noticing “keep_ratio” is True. please set maintain-aspect-ratio=1.
if still can’t work, could you share the whole log? wondering if the engine was loaded. Since TRT engine is device dependent, it can’t be used in other devices. could you share the ONNX model if possible? Thanks!

ronpai · November 8, 2024, 8:32am

Thanks for the modification. I think the network format is in sync with DeepStream (NHWC)
But after testing, there is dimension issue in the next step.

Is there any other params that are improperly configured?

fanzh · November 8, 2024, 8:53am

For the log, the app loaded the engine. but the engine is inconsistent with configurations. then the app tried to generate a new engine.
Please set infer-dims=3;640;640. nvinfer plugin parses this value by the format of “CHW”. please refer to gst_nvinfer_parse_props in \opt\nvidia\deepstream\deepstream-7.0\sources\gst-plugins\gst-nvinfer\gstnvinfer_property_parser.cpp.

ronpai · November 8, 2024, 9:12am

I am confused about the dimension in the config.
Since we aligne the network to NHWC, it doesn’t mean that the infer dimension would be 640;640;3 ?
After setting infer-dims=3;640;640, the pipeline works, but it failed in the bbox parsing part (our parsing function).

The same parsing function works in nvinferserver where model was hosted with triton. inferserver configs are provided below

infer_config {
  unique_id: 1
  gpu_ids: [0]
  max_batch_size: 16
  backend {
    inputs: [
      {
        name: "INPUT"
        dims: [ 640, 640, 3 ]
      }
    ]
    outputs: [
      {
        name: "OUTPUT"
      }
    ]
    triton {
      model_name: "recognizer:general"
      version: -1

      grpc {
        url: "deepstream-model-1:8001"
        enable_cuda_buffer_sharing: false
      }
    }
    output_mem_type: MEMORY_TYPE_DEFAULT
  }

  preprocess {
    network_format: IMAGE_FORMAT_BGR  # IMAGE_FORMAT_RGB
    tensor_order: TENSOR_ORDER_NHWC
    tensor_name: "INPUT"
    frame_scaling_hw: FRAME_SCALING_HW_DEFAULT
    frame_scaling_filter: 1
    maintain_aspect_ratio: 0
    symmetric_padding: 0
  }

  postprocess {
    other { }
  }

  extra {
    custom_process_funcion: "CreateInferServerCustomProcess"
    copy_input_to_host_buffers: false
    output_buffer_pool_size: 2
  }

  custom_lib {
    path: "/opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infercustomparser.so"
  }
}

input_control {
  process_mode: PROCESS_MODE_FULL_FRAME
  interval: 0
}

output_control {
  output_tensor_meta: false
}

fanzh · November 8, 2024, 9:43am

the nvinfer plugin is opensource. As the code shown, the three values are set to C, H, W respectively. so, you need to set 3;640;640.

        init_params->inferInputDims = NvDsInferDimsCHW {(unsigned int) int_list[0],
                                      (unsigned int) int_list[1], (unsigned int) int_list[2]};

the app crashed in the postprocessing function. nvinfer parse-bbox-func is different with nvinferserver custom_process_funcion. please refer to set1 and code2 for how to write nvinfer parse-bbox-func.
If you want to parse inference results directly, please set network-type=100 and output-tensor-meta=1, then parse the inference results in probe function. please refer to sample deepstream-infer-tensor-meta-test.

fanzh · November 11, 2024, 7:35am

Sorry for the late reply, Is this still an DeepStream issue to support? Thanks!

ronpai · November 11, 2024, 9:06am

After modifying config and parser function, we are able to build engine and run deepstream with our model.
Really thanks for the help!

fanzh · November 13, 2024, 1:46am

why do you need to use uint8 type of model input? why no use fp32 or other types?

ronpai · November 15, 2024, 8:56am

Our data scientist fixed the format and data type (uint8 input, channel last output, …) for easy development and alignment.
Is there recommendations on the data type/format with respect to performance?

system · November 29, 2024, 8:57am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

fanzh · February 14, 2025, 5:48am

@ronpai we fixed this issue of “Unknown data type for bound layer i(INPUT). type:5”. Using the model you provided in the email, we failed to generate TRT engine by trtexec or DeepStream. could you help to verify the fix in DS7.0 docker? Thank you! Here are the steps.

start docker. here is a sample cmd.

xhost + && docker run --name fan-70-int8  --privileged --gpus  "device=0"  -itd --net=host --ipc=host --rm -v /tmp/.X11-unix:/tmp/.X11-unix  -v /home/fanzh:/home        -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream  nvcr.io/nvidia/deepstream:7.0-triton-multiarch

run fix.sh in the fix.zip (6.3 MB), which will copy so to libs. the fix code is in the “code” directory.
run “deepstream-test5-app -c configs/ds_yolov9_config.txt >1.log 2>1.log” to test your model with “type 5” input. then please share 1.log for further analysis. Thanks!

Topic		Replies	Views
No detections on deepstream (nvinfer) with nvdspreprocess plugin add DeepStream SDK deepstream	17	133	October 28, 2024
ONNX as inference model (YoloV8) DeepStream SDK jetpack , onnx , deepstream	12	3424	July 10, 2023
Tutorial: How to run YOLOv7 on Deepstream DeepStream SDK demos-and-tutorials	19	6872	March 26, 2024
Yolov7 onnx model convert to trt model error in deepstream 6.2, but use trtexec ok DeepStream SDK	9	330	June 11, 2024
Generate engine using onnx2trt, The engine was used to call deepstream, but an error was reported DeepStream SDK onnx	5	217	June 11, 2024
How to run deepstream-app without building engine? DeepStream SDK	6	489	March 11, 2024
Getting an error while trying to use deepstream_launchpad.ipynb code DeepStream SDK deepstream	28	1319	January 10, 2024
Error during run inference with custom yolov5 model in deepstream DeepStream SDK deepstream	8	86	September 17, 2024
TRT Fails to parse ONNX Model (yolov8 Segmentation) DeepStream SDK tensorrt , yolo , onnx	4	292	June 11, 2024
Unable to run Deepstream on a YoloV3 ONNX model DeepStream SDK tensorrt , cuda	4	1440	August 28, 2021

Yolov9 Model Engine Build Failure with nvinfer

Related topics