Darknet YoloV4-tiny model in TensorRT 8 inference

god_ra · October 12, 2021, 9:06am

Description

I have darknet yolov4-tiny model trained on 5 objects. I was using tensorrt 6 and tkdnn repo to run inference. Everything was perfect.

Now, I want to use tensorrt 8 and run the inference.
I am trying to convert darknet yolov4-tiny model to onnx and to TensorRT 8.
But unfortunately, I am not able to do it properly.

Could you please let me know How I can perform inference on it ??

Environment

TensorRT Version: 8.0.1

NVES · October 12, 2021, 10:38am

Hi,
Can you try running your model with trtexec command, and share the “”–verbose"" log in case if the issue persist
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

You can refer below link for all the supported operators list, in case any operator is not supported you need to create a custom plugin to support that operation

github.com

onnx/onnx-tensorrt/blob/main/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 8.4 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT, clamping values to `+-INT_MAX` or `+-FLT_MAX` if necessary.

See below for the support matrix of ONNX operators in ONNX-TensorRT.

## Operator Support Matrix

| Operator                  | Supported  | Supported Types | Restrictions                                                                                                           |
|---------------------------|------------|-----------------|------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          | FP32, FP16, INT32 |
| Acos                      | Y          | FP32, FP16 |
| Acosh                     | Y          | FP32, FP16 |
| Add                       | Y          | FP32, FP16, INT32 |

This file has been truncated. show original

Also, request you to share your model and script if not shared already so that we can help you better.

Meanwhile, for some common errors and queries please refer to below link:

Thanks!

spolisetty · October 12, 2021, 1:31pm

Hi,

Please make sure your inference script is correct. You can refer samples here.

If you still face this issue, could you please share issue repro script/input data and complete error logs for better debugging.

Thank you.

spolisetty · October 14, 2021, 4:56am

Could you please confirm if you’re using Deepstream ?

spolisetty · October 20, 2021, 4:50pm

Hi,

Sorry for the delayed response, we have gone through the code, looks normal.
Could you please share actual error logs you’re facing and if possible minimal issue repro for better debugging.

Thank you.

god_ra · October 21, 2021, 6:54am

@spolisetty @NVES I tried this repo to convert to onnx model.
It takes by default 64 batchsize:
So, it take 1.5 - 2 hours to built the engine and inference time is almost 0.7 fps (way too slow than expected.)
Normally it should take batch size 1. but it is not working properly.

I tried to change the batch size to 1 in the config file and generated onnx model.
Then used trtexec to create an engine.

Then I used my script given above to do inference.

But I get smaller boxes and too many boxes (labels and probability is correct but only the box dimensions are not correct). Boxes are not around the object exactly instead they are small and tiny somewhere on the input image.

Is there something wrong the with calculation part ?? or engine creation part ?? or onnx conversion with batch size 1.

Could you provide me the command to be used to create engine with trtexec and required onnx conversion script with batch size 1.

spolisetty · October 21, 2021, 8:14pm

Hi,

As batch size is static input, we can run trtexec command normally to build the engine.
trtexec --onnx=/path/to/.onnx --saveEngine=engine.trt

Also could you please confirm are you getting correct bounding boxes result with onnx-runtime, to make sure onnx file is generated correctly.

Also please make sure your post processing is working fine (as we could get labels and probability is correct).

You can refer the same tensorrt sample shared in previous replies.

Thank you.

god_ra · October 22, 2021, 8:07am

solved it. There was a miss interpretation of the network.

Topic		Replies	Views
Yolov4 TensorRT slower than Yolov4 darknet TensorRT	6	3635	September 1, 2020
Converting yolov4 onnx model to TensorRT for multi batch input TensorRT cudnn	3	765	January 31, 2024
YOlov4-tiny with batch size 64 works , but batch size 1 gives wrong bounding boxes TensorRT tensorrt , yolo	1	1283	October 22, 2021
Yolov4-tiny onnx convert to trt cuDNN tensorrt	2	1563	November 23, 2020
TensorRT Batch Inference: different results TensorRT	4	4478	December 1, 2021
My Yolov4 tf2checkpoint->save_model->onnx->trt TensorRT tensorrt , jetson-inference	3	552	November 1, 2021
YOLOv4 TensorRT inference results wayy off, but onnxruntime is not TensorRT tensorrt	6	1105	June 7, 2022
Help needed to convert yolov4-tiny model to tensorRT engine (DS 5) DeepStream SDK	1	2257	February 25, 2021
Converting Yolo model to TensorRT format without ONNX conversion Jetson Projects tensorrt , yolo , pytorch , cudnn , inference-server-triton , deepstream	2	603	February 10, 2025
Deepstream testing: Yolov8 models for multiple streams DeepStream SDK deepstream	5	283	July 1, 2025

Darknet YoloV4-tiny model in TensorRT 8 inference

Description

Environment

Related topics