Yolov8 and Yolov11 are giving wrong infer results on deepstream 8.0 on Ubuntu24

[sagar.g]

• OS: Ubuntu 24.04

• Hardware Platform (Jetson / GPU): GPU
• DeepStream Version: 8.0
• TensorRT Version: 10.9.0.34-1+cuda12.8 (both libnvinfer and tensorrt, as mentioned in Deepstream documentation here)
• NVIDIA GPU Driver Version (valid for GPU only): 570

If I used the onnx files given by nvidia here for yolov8 and yolov11 along with config file from that repo, I am getting correct detections (example image at very bottom), however, if I downloaded yolov8 or yolo11 from ultralytics and export the .pt model to .onnx myself and created engine file, I am getting wrong infer results (example below) and also, all the bboxes are at the top left of the image (please see image below).

Here is how I got the onnx file and then the engine file:

from ultralytics import YOLO
yolo = YOLO("yolov11s.pt")  # I have downloaded yolov8s.pt from here https://docs.ultralytics.com/models/yolov8/#performance-metrics
yolo.export(format="onnx")  # This created yolov11s.onnx in the same folder

Then, I created engine file using the command:

/usr/src/tensorrt/bin/trtexec --onnx=yolov8s.onnx --fp16 --saveEngine=yolov8s.onnx_b1_gpu0_fp16.engine

Then, I used this engine file in my deepstream python app which runs deepstream using pyds. I used this config_infer_primary_yoloV8.txt which I got from this nvidia-ai-iot deepstream_tools repo. I built libnvdsinfer_custom_impl_Yolo.so on the same machine using the instructions in that repo, and used it. Only changes I made in the config file before using in my deepstream app are the following:

• onnx-file: absolute path to the onnx file
• int8-calib-file: Commented this out as I was trying fp16 engine
• batch-size: 1
• model-engine-file: absolute path to the engine file saved by the trtexec command above
• labelfile-path: absolute path to the labels file (downloaded and used this from the same repo)
• network-mode: 2
• custom-lib-path: absolute path to the built file

Notes: All other values in the config file are not changed. I have copied all these 4 files (onnx, engine, labels, libnvdsinfer_custom_impl_Yolo) to the same folder as the config file, and gave absolute paths in the config file.

The following are the outputs that I see. Please notice the label names, and also that they are all at the top left.

Counts: Counter({'scissors': 2, 'vase': 1, 'train': 1})

---

I came across this old forum post: https://forums.developer.nvidia.com/t/get-wrong-infer-results-while-testing-yolov4-on-deepstream-5-0/125526

From the solution of the above post, I have checked for compatibility of torch and tensorrt versions using this tensorrt documentation, I have found that they are compatible.

Required versions	My Versions in conda venv which is used for export from pt to onnx
PyTorch >= 2.0	torch==2.9.1
ONNX=1.16.0	onnx==1.16.0 (I also tested with 1.19.0 before downgrading to 1.16.0)

---

The following shows an image of correct detections with same config file with nvidia’s onnx files taken from this nvidia-ai-iot deepstream_tools repo (as mentioned in the beginning of this post):

---

Can you please help with why my pt → onnx → engine is not giving correct results, but, nvidia’s pre-trained and exported onnx → engine is giving correct results with same config file and same lib libnvdsinfer_custom_impl_Yolo.so?

[fanzh]

@sagar.g

I used this engine file in my deepstream python app which runs deepstream using pyds.

Taking yolov11 for example, to narrow down the issue, In deepstream_yolo code, if you only add the following cfgs in nvinfer’s cfg file, will the issue persist?
onnx-file=new model’s path.
#model-engine-file=yolov11s_qat_int8_672_dynamic.onnx_b16_gpu0_best.engine

[fanzh]

The Yolo DeepStream samples provide QAT calibration tools for some open source Yolo models to improve the Yolo models accuracy and performance with DeepStream.
the official yolov11 model’s output is incompatible with the postprocessing function, as shown in this doc, Please add a transpose node at the end of the network with append_transpose_yolov8_v9.py.

[sagar.g]

Thank you very much fanzh. Adding transpose node to the onnx files using the above script worked for both yolo11s.pt and yolov8s.pt ultralytics default models.