Ultralytics yolo Inference Low confidence accuracy by TensorRT 8.5.2.2

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Description

AWD.onnx(Rename it).txt (11.7 MB)

When performing inference using awd.onnx (a Yolov8n model with two classes: 0 for ‘closed’ and 1 for ‘open’), the OpenCV DNN inference on sample images yields confidence scores above 90%. However, when using TensorRT for inference (with consistent results on both Jetson and Windows platforms), the obtained confidence scores are around 4%. How can I eliminate this type of bug? Could it be that I did not configure things correctly when building the engine?

sample:

out
out1920×1080 271 KB

TensorRT Output:

out.jpg_0
out.jpg_01920×1080 271 KB

OpenCV DNN Output:

out_dnn
out_dnn1920×1080 272 KB

Environment

TensorRT Version: 8.5.2.2
GPU Type: Jetson Orin / NX , RTX3070Ti , RTX 4090
Nvidia Driver Version: Jetson Orin/NX Jetpack 5.1.1 Jetpack 5.1.3 , RTX3070Ti 572.83 , RTX 4090 520.61.05
CUDA Version: Jetson Orin/NX 11.4.315 , RTX3070Ti 12.8 , RTX 4090 11.8
CUDNN Version: Jetson Orin/NX 8.6.0.166
Operating System + Version: Ubuntu 20.04 , Windows 11
Python Version (if applicable): 3.8
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro

  • Exact steps/commands to run your repro

  • Full traceback of errors encountered

3

Is this issue a bug in TensorRT?

4

Did you try running TensorRT model with Ultralytics?

5

I used the Ultralytics YOLOv8n model for training and performed inference on the Jetson Orin platform. While the official Ultralytics inference worked without issues on x86 platforms, Ultralytics could not be installed on the Jetson Orin. Therefore, I did not attempt to use Ultralytics’ TensorRT for inference.

6

Ultralytics should work fine on Jetson Orin

7

That’s correct. Running inference with the Ultralytics YOLO series on Jetson using TensorRT is possible, but the accuracy may not align with that on the x86 platform in certain cases, sometimes even differing by more than tenfold in confidence.

8

Are you referring to INT8 accuracy? Why would it be any different from using TensorRT yourself? Ultralytics is just using the TensorRT API underneath.

The TensorRT mAP on Jetson is shown in the docs and they are pretty much the same as ONNX:

10

I used TensorRT with FP32 precision and also tested FP16 precision, but did not test INT8 precision. However, during practical application testing, I observed that their results indeed differ from those of ONNX.