Bounding Boxes stuck in corner

[martinminell]

Hardware: Jetson Orin Nano

JetPack: 6.x (R36.4.7)

CUDA: 12.6 TensorRT: 10.7

DeepStream: 7.1.0

cuDNN: 9.1

Models: YOLOv8n

Hi!

I see that im not alone with this problem. I struggle with bounding boxes grouping in the corner.

Does someone have any clue?

[junshengy]

@martinminell

. I struggle with bounding boxes grouping in the corner.

Could you elaborate on what you mean? Or screenshots to illustrate.

If you mean the bounding box exceeds the frame’s width/height, that’s normal. However, you can crop the bounding box by setting the crop-objects-to-roi-boundary property of nvinfer to true.

[Y-T-G]

Did you use Deepstream-Yolo to convert?

[martinminell]

Hi @junshengy and @Y‑T‑G — thanks for jumping in!

I’m seeing all detected boxes snapping into the top‑left corner (0,0). It’s not about boxes exceeding frame boundaries; it looks more like coordinates are not being scaled/mapped correctly. Confidence scores look reasonable, but the drawn bboxes cluster in the corner.

Environment

• Hardware: Jetson Orin Nano

• JetPack: 6.x (R36.4.7)

• CUDA: 12.6, TensorRT: 10.7

• DeepStream: 7.1.0, cuDNN: 9.1

• Model: YOLOv8n (ONNX)

Pipeline & configs
I’m using DeepStream‑Yolo for YOLOv8 with the custom bbox parser. Engine is built fresh from ONNX (no legacy engine). Key bits from my config:

INI

# primary-gie (nvinfer)

onnx-file=/home/aiadmin/DeepStream-Yolo/onnx/yolov8n.onnx

labelfile-path=/home/aiadmin/DeepStream-Yolo/labels/coco_labels.txt

model-engine-file= # left empty to force rebuild

network-type=0 # detector

num-detected-classes=80

infer-dims=3;640;640

maintain-aspect-ratio=1 # also tried 0

batch-size=1

network-mode=2 # FP16

custom-lib-path=/home/aiadmin/DeepStream-Yolo/ds-yolo/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so

parse-bbox-func-name=NvDsInferParseCustomYOLO (Deepstream wont start with V8 at the end,

cluster-mode=4

pre-cluster-threshold=0.25

iou-threshold=0.45

output-tensor-meta=1 # enabled for debugging

crop-objects-to-roi-boundary=1 # tried per suggestion

Visa fler rader

What I already tried

• Toggled maintain-aspect-ratio (0/1) — no change.

• Set crop-objects-to-roi-boundary=1 — issue persists (since boxes are at 0,0 rather than overflowing).

• Rebuilt the custom parser (libnvdsinfer_custom_impl_Yolo.so) and removed any stale .engine.

• Verified input shape matches ONNX (3×640×640).

• Switched FP16 ⇄ FP32 — no behavioral change.

Questions / hypotheses

1. Parser/TRT/DS compatibility: Could this be a mismatch between the YOLOv8 custom parser and DeepStream 7.1 + TensorRT 10.7? (e.g., output tensor indices/order changed and the parser reads zeros ⇒ x,y≈0)

2. Letterboxing mapping: If maintain-aspect-ratio=1 is on, does the parser need a specific compile‑time flag/branch to correctly un‑letterbox YOLOv8 coords in DS 7.1?

3. Required keys: For YOLOv8 with this repo, are cluster-mode=4 + NvDsInferParseCustomYOLOv8 still the canonical pair for DS 7.1, or should I drop cluster-mode and rely entirely on the parser’s internal NMS?

4. Blob names: Do I need to set output-blob-names explicitly for the YOLOv8 head in DS 7.1/TRT 10.7, or should it be auto‑detected?

Next steps I’m planning (happy to adjust if you suggest otherwise):

• Clean rebuild of the custom parser against DS 7.1 headers/TRT 10.7 (remove build cache, rebuild nvdsinfer_custom_impl_Yolo.so).

• Force engine rebuild by deleting any .engine and running once with output-tensor-meta=1 to inspect raw tensor shapes/values.

• Test with maintain-aspect-ratio=0 + a 640×640 source to rule out letterbox mapping entirely.

• If needed, try the latest DeepStream‑Yolo branch/commit known to support DS 7.1 (pointer appreciated).

@Y‑T‑G — yes, I used DeepStream‑Yolo to convert/build and but I cant use its custom parser (NvDsInferParseCustomYOLOv8) with the libnvdsinfer_custom_impl_Yolo.so Deepstream wont start. I use, NvDsInferParseCustomYOLO. Could this be the problem?

Thanks in advance for any pointers!

[junshengy]

@martinminell

Parser/TRT/DS compatibility: Could this be a mismatch between the YOLOv8 custom parser and DeepStream 7.1 + TensorRT 10.7? (e.g., output tensor indices/order changed and the parser reads zeros ⇒ x,y≈0)

It’s possible, so please ensure Jetpack and DeepStream versions are compatible.
For Orin nano and Jetpack 6.x, it should be TRT 10.3.0.31.Refer to this link.

Platform and OS Compatibility - Installation — DeepStream documentation

@martinminell

Letterboxing mapping: If maintain-aspect-ratio=1 is on, does the parser need a specific compile‑time flag/branch to correctly un‑letterbox YOLOv8 coords in DS 7.1?

This value is only used to maintain the aspect ratio when converting an image to a tensor.

@martinminell

Required keys: For YOLOv8 with this repo, are cluster-mode=4 + NvDsInferParseCustomYOLOv8 still the canonical pair for DS 7.1, or should I drop cluster-mode and rely entirely on the parser’s internal NMS?

This depends on your model. In the official demo provided below, the model does not include NMS, so cluster-mode=2 in nvinfer configuration file.

@martinminell

Blob names: Do I need to set output-blob-names explicitly for the YOLOv8 head in DS 7.1/TRT 10.7, or should it be auto‑detected?

For onnx model, no additional settings are required usually.

1. How did you obtain yolov8n.onnx? Did the model work correctly before exporting onnx?
2. What does the post-processing code look like?

You can refer to this official repository.

yolo_deepstream/deepstream_yolo/README.md