YOLOv8 ONNX → TensorRT on Jetson TX2 gives wrong predictions (evaluation stuck / always 1)

Robotics & Edge Computing Jetson Systems Jetson TX2
1

Hello,

I am working on a NVIDIA Jetson TX2 Developer Kit.

I trained a YOLOv8 model for object detection. The workflow I used is:

  • Train YOLOv8 model on PC

  • Export to ONNX

  • Convert ONNX → TensorRT engine for Jetson TX2

  • Run inference on Jetson TX2

Problem:

The model runs but the results are completely wrong / unstable.

  • Predictions are incorrect

  • Evaluation seems stuck (always = 1 or constant value)

  • GPU is working correctly

  • I verified CUDA / TensorRT are installed and functional

  • I also checked image preprocessing (RGB / normalization)

Environment:

  • Jetson TX2

  • Python 3.6

  • TensorRT installed

  • ONNX model exported from YOLOv8

  • Conversion done on PC then deployed to Jetson

What I already tested:

  • GPU is OK

  • Colors (RGB/BGR) checked

  • Input size checked

  • Model runs without crash but outputs are wrong

Question:

What could cause incorrect predictions after ONNX → TensorRT conversion on Jetson TX2?
Is there a known issue with YOLOv8 + TensorRT + Python 3.6 or preprocessing mismatch?

Any help is appreciated.

2

Most likely a postprocessing issue

3

Hello,

Thank you for your feedback.

I tested several configurations:

  • YOLOv8 trained on PC

  • Export to ONNX (different opset versions)

  • Conversion ONNX → TensorRT engine on Jetson TX2

  • Multiple preprocessing versions (RGB/BGR, resize vs letterbox)

  • Different postprocessing attempts (with and without NMS logic, confidence variations)

  • TensorRT FP16 engine on TX2

The GPU and TensorRT are working correctly, but the output remains unstable or incorrect.

At this stage, I suspect a mismatch between YOLOv8 output format and my TensorRT postprocessing implementation.

this is my code import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
import numpy as np
import cv2

ENGINE_PATH = “best2.engine”
IMAGE_PATH = “test.jpg”

IMG_SIZE = 640
CONF_THRESH = 0.3

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

======================

LOAD ENGINE

======================

with open(ENGINE_PATH, “rb”) as f:
runtime = trt.Runtime(TRT_LOGGER)
engine = runtime.deserialize_cuda_engine(f.read())

context = engine.create_execution_context()

print(“✅ Engine loaded”)

======================

LETTERBOX (IMPORTANT FIX)

======================

def letterbox(img, new_shape=640):
h, w = img.shape[:2]
scale = min(new_shape / h, new_shape / w)

nh, nw = int(h * scale), int(w * scale)
resized = cv2.resize(img, (nw, nh))

canvas = np.full((new_shape, new_shape, 3), 114, dtype=np.uint8)

top = (new_shape - nh) // 2
left = (new_shape - nw) // 2

canvas[top:top+nh, left:left+nw] = resized
return canvas, scale, top, left

======================

IMAGE

======================

img = cv2.imread(IMAGE_PATH)
if img is None:
print(“❌ Image not found”)
exit()

orig = img.copy()
h0, w0 = orig.shape[:2]

img, scale, pad_y, pad_x = letterbox(img, IMG_SIZE)

BGR → RGB

img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

Normalize

img = img.astype(np.float32) / 255.0

CHW format

img = np.transpose(img, (2, 0, 1))
img = np.expand_dims(img, axis=0)
img = np.ascontiguousarray(img)

======================

GPU

======================

d_input = cuda.mem_alloc(img.nbytes)

output = np.empty((1, 10000, 6), dtype=np.float32) # adjust if needed
d_output = cuda.mem_alloc(output.nbytes)

bindings = [int(d_input), int(d_output)]

cuda.memcpy_htod(d_input, img)

context.execute_v2(bindings)

cuda.memcpy_dtoh(output, d_output)

print(“✅ Inference done”)

======================

POSTPROCESS (FIXED)

======================

preds = output[0]

detected = False
best_conf = 0.0

for det in preds:

# YOLOv8 format most common:
x, y, w, h, conf, cls_id = det

# ignore invalid
if conf < CONF_THRESH:
    continue

detected = True
best_conf = max(best_conf, conf)

# Undo letterbox
x1 = (x - w / 2)
y1 = (y - h / 2)
x2 = (x + w / 2)
y2 = (y + h / 2)

# remove padding + scale back
x1 = int((x1 - pad_x) / scale)
y1 = int((y1 - pad_y) / scale)
x2 = int((x2 - pad_x) / scale)
y2 = int((y2 - pad_y) / scale)

# clamp
x1 = max(0, min(w0, x1))
y1 = max(0, min(h0, y1))
x2 = max(0, min(w0, x2))
y2 = max(0, min(h0, y2))

cv2.rectangle(orig, (x1, y1), (x2, y2), (0, 255, 0), 2)
cv2.putText(orig,
            f"OBJ {conf:.2f}",
            (x1, y1 - 10),
            cv2.FONT_HERSHEY_SIMPLEX,
            0.7,
            (0, 255, 0),
            2)

======================

RESULT

======================

if detected:
print(“🦀 OBJECT DETECTED | CONF:”, best_conf)
else:
print(“❌ NO DETECTION”)

cv2.imwrite(“result.jpg”, orig)
print(“✅ Saved result.jpg”)

4

The postprocessing used is wrong. Refer to the guide here: