DeepStream YOLOv8 Custom Model – No Bounding Boxes Detected Despite Correct Parser, ONNX, and TRT Engine (2-Class Model)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, , , ,
1

Hello NVIDIA Team,
I am trying to deploy a custom YOLOv8s model on DeepStream (Jetson Orin nano), and I am facing an issue where no bounding boxes appear during inference inside the DeepStream pipeline, even though the model works correctly in Python.Model Details

YOLOv8s model, fine-tuned for 2 classes only:

person

roll_stopper

Trained using Ultralytics.

Exported to ONNX, verified that ONNX produces correct outputs for both classes.

Built FP16 TensorRT engine (.engine) from the ONNX file using trtexec, also validated engine loads correctly in TensorRT runtime.

DeepStream Setup
I am using the official NVIDIA GitHub sample:

https://github.com/NVIDIA-AI-IOT/deepstream_tools/tree/main/yolo_deepstream/deepstream_yolo
I have done the following:

. Custom YOLO parser modified for 2 classes
I edited both:

nvdsparsebbox_Yolo.cpp

nvdsparsebbox_Yolo_cuda.cu

Specifically, I set:
static const int NUM_CLASSES_YOLO = 2;
I also verified using grep that no other hard-coded 80 class constants exist.

Then I performed:
make clean
make
and verified the .so is updated and loaded by DeepStream.

  1. ONNX Model Verified

The ONNX model outputs correct detections (both classes) in Python using onnxruntime.

I checked ONNX initializers to ensure no tensor dimension = 80 exist

ONNX clearly reflects 2 classes, not COCO-80.

  1. TensorRT Engine Verified

TRT engine was built on the same Jetson device as DeepStream.

Built with:
--fp16 --explicitBatch --minShapes=1x3x640x640
--optShapes=1x3x640x640 --maxShapes=1x3x640x640

Engine deserializes correctly using a TensorRT test script.

No mismatch warnings appear regarding engine version or shape.

  1. DeepStream Config Verified

Key settings in config_infer.txt:
model-engine-file=.../best_fp16_b1.engine
# onnx-file commented out
num-detected-classes=2
labelfile-path=/abs/path/labels.txt
parse-bbox-func-name=NvDsInferParseCustomYoloV8_cuda
custom-lib-path=/abs/path/libnvdsinfer_custom_impl_Yolo.so
infer-dims=3;640;640
batch-size=1
network-mode=2 # FP16

DeepStream successfully loads:

  • the engine and runs via RTSP stream.

  • Problem
    DeepStream runs without errors, but no bounding boxes appear on the video output at all.

  • the custom parser library

🚀 Platform Details

Jetson Platform: JetPack 5.1.5

nvdsparsebbox_Yolo.cpp.txt (12.0 KB)

nvdsparsebbox_Yolo_cuda.cu.txt (7.1 KB)

DeepStream SDK: 6.3

CUDA: 11.4

TensorRT: (version included with JetPack 5.1.5

Running on a Jetson device (Orin)

4

to narrow down this issue, Please debug the cpp version parsing code. please uncomment “parse-bbox-func-name=NvDsInferParseCustomYoloV8” and commnet out “parse-bbox-func-name=NvDsInferParseCustomYoloV8_cuda”.
If adding log in NvDsInferParseYoloV8, will the the log be printed? could you share some printing of bbi.width, bbi.height, bbi.detectionConfidence? are all bbi.detectionConfidence smaller than 0? is the parsing code logics consistent with test in onnxruntime?

5

I checked the parser code in the repository, and I don’t see a transpose function. YOLOv8 outputs (B, 4 + num_classes, num_objects). It needs to be transposed to (B, num_objects, 4 + num_classes) before processing. Or otherwise, you will have to use stride of num_objects when reading the coordinates and class scores because they’re not adjacent.

6

Sorry for the late reply, Is this still an DeepStream issue to support? Thanks!

7

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks.