Title
TensorRT 10.14 silently produces wrong detection scores for D-FINE (DETR-style) models on RTX 5090 / sm_120 — default build drops 12 detections to 0, FP16 likewise; TensorRT 10.8 is correct at every precision
—
Description
An engine built from a D-FINE detector ONNX gives drastically wrong outputs on TensorRT 10.14.1.48,
while TensorRT 10.8.0.43 is correct from the *same ONNX, the same input tensor and the same GPU*.
There is no error, no warning and no build failure. `trtexec` prints `PASSED`, the engine
deserializes, inference runs at full speed. Only the numbers are wrong: confidence scores are crushed,
so detections fall below any sane threshold and the model looks like it simply sees nothing.
Minimal case — one 640x640 frame, batch 1, counting output rows with score >= 0.45:
| build flags | TensorRT 10.8.0.43 | TensorRT 10.14.1.48 |
|—|—|—|
| onnxruntime 1.20.1 CPU FP32 (reference) | 12 dets, max 0.9189 | 12 dets, max 0.9189 |
| `–noTF32` (true FP32) | 12 dets, max 0.9189 | 11 dets, max 0.9199 |
| *default* (TF32 enabled) | 12 dets, max 0.9189 | dets, max 0.3117 |
| `–fp16` | 12 dets, max 0.9185 | 0 dets, max 0.3220 |
The default build path — plain `trtexec --onnx=… --saveEngine=…`, i.e. what most users and what
DeepStream’s `nvinfer` do — goes from 12 detections to zero, and the best score in the whole
tensor falls from 0.92 to 0.31.
The same run over 16 frames, for scale (reference = 215 detections >= 0.45):
| build flags | TensorRT 10.8.0.43 | TensorRT 10.14.1.48 |
|—|—|—|
| `–noTF32` | 215 (max 0.9370) | 175 (max 0.9266) |
| *default* (TF32) | 215 (max 0.9370) | 4 (max 0.5154) |
| `–fp16` | 214 (max 0.9360) | 7 (max 0.6792) |
Reproduced with the public D-FINE-N COCO model, opset 17.
Additional observations, all on 10.14.1.48
1. A larger D-FINE-M variant of the same architecture returns all-NaN at `–fp16` — 21,000 NaNs in
a 16x300x6 output tensor, class-id column intact, every score and coordinate NaN. `trtexec` still
reports `PASSED`. NaN output is an unambiguous runtime defect.
2. Accuracy depends on unrelated builder knobs, which points at tactic selection. On the D-FINE-M
variant, all with `–noTF32`: adding
`config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 8 << 30)` took a plate-detection clip
from 23 detections to 0; pinning the optimization profile to `min=opt=max=16` instead of
`min=1,opt=16,max=16` took it from 1 detection to 19. Same weights, same precision flag, only
tactics differ.
3. Graph packaging is irrelevant. The collapse occurs both with the stock D-FINE export (inputs
`images` + `orig_target_sizes`; outputs `labels`/`boxes`/`scores`) and with a fused export (single
`images` input, single `[batch,300,6]` output). Both are correct on 10.8.
4. Not model-specific. Reproduced on two independently trained checkpoints — D-FINE-N/COCO-80 and a
custom D-FINE-M with 8 classes.
5. `–noTF32` was the previously known workaround for this model family. On 10.14 it is no longer
sufficient: it still loses ~19% of detections at batch 16 (175 vs 215).
—
Environment
TensorRT Version: 10.14.1.48 (fails) — 10.8.0.43 (works)
GPU Type: NVIDIA GeForce RTX 5090 (Blackwell, compute capability 12.0), 32 GB
Nvidia Driver Version: 590.48.01
CUDA Version: 13.1 (failing) — 12.8 (working)
CUDNN Version: 9.17.1.4 (failing) — 9.3.0 (working)
Operating System + Version: Ubuntu 24.04.3 LTS (failing) — Ubuntu 22.04.4 LTS (working)
Python Version (if applicable): 3.12.3 (failing) — 3.10.12 (working)
TensorFlow Version (if applicable): n/a
PyTorch Version (if applicable): 2.13.0+cu129 (used only to export the ONNX)
Baremetal or Container (if container which image + tag): Container, both locally built.
Failing: DeepStream 9.0.0 / TensorRT 10.14.1.48 / CUDA 13.1 / cuDNN 9.17.1.4.
Working: DeepStream 7.1.0 / TensorRT 10.8.0.43 / CUDA 12.8 / cuDNN 9.3.0.
Both containers run on the same physical host, the same RTX 5090 and the same driver 590.48.01, so
GPU and driver are held constant; only the CUDA / cuDNN / TensorRT stack differs.
—
The ONNX can also be regenerated from the public repository
(GitHub - Peterande/D-FINE: D-FINE: Redefine Regression Task of DETRs as Fine-grained Distribution Refinement [ICLR 2025 Spotlight] · GitHub) with the checkpoint `dfine_n_coco.pth` and the export wrapper in
step 0 below, so no proprietary weights are needed to reproduce this.
—
Steps To Reproduce
0. (Optional) Regenerate the ONNX from the public checkpoint
import torch, torch.nn as nn
from src.core import YAMLConfig
CFG, CKPT = "configs/dfine/dfine_hgnetv2_n_coco.yml", "dfine_n_coco.pth"
cfg = YAMLConfig(CFG, resume=CKPT)
cfg.yaml_cfg["HGNetv2"]["pretrained"] = False
ck = torch.load(CKPT, map_location="cpu")
cfg.model.load_state_dict(ck["ema"]["module"] if "ema" in ck else ck["model"])
class DS(nn.Module):
def __init__(self):
super().__init__()
self.model, self.post = cfg.model.deploy(), cfg.postprocessor.deploy()
self.register_buffer("sz", torch.tensor([[1, 1]], dtype=torch.float32))
def forward(self, images):
labels, boxes, scores = self.post(self.model(images), self.sz.repeat(images.shape[0], 1))
return torch.cat([labels.unsqueeze(-1).float(), scores.unsqueeze(-1), boxes], dim=-1)
torch.onnx.export(DS().eval(), torch.randn(2, 3, 640, 640), "dfine_n_coco_ds.onnx",
input_names=["images"], output_names=["output"],
dynamic_axes={"images": {0: "batch_size"}, "output": {0: "batch_size"}},
opset_version=17, do_constant_folding=True, dynamo=False)
1. Build the input tensor (letterbox to 640x640, RGB, scale 1/255, NCHW float32)
# make_input.py
import numpy as np
from PIL import Image
im = Image.open("repro_frame.jpg").convert("RGB")
w, h = im.size
s = min(640 / w, 640 / h)
canvas = Image.new("RGB", (640, 640), (0, 0, 0))
canvas.paste(im.resize((int(round(w * s)), int(round(h * s))), Image.BILINEAR), (0, 0))
arr = np.transpose(np.asarray(canvas, np.float32) / 255.0, (2, 0, 1))[None]
arr.astype(np.float32).tofile("input_1x3x640x640.bin") # 4,915,200 bytes
2. Build and run, three precisions — identical commands in both containers
for V in "notf32 --noTF32" "default " "fp16 --fp16"; do
set -- $V; TAG=$1; shift
/usr/src/tensorrt/bin/trtexec \
--onnx=dfine_n_coco_ds.onnx \
--saveEngine=/tmp/n_$TAG.engine \
--minShapes=images:1x3x640x640 \
--optShapes=images:16x3x640x640 \
--maxShapes=images:16x3x640x640 \
$@ --skipInference
/usr/src/tensorrt/bin/trtexec \
--loadEngine=/tmp/n_$TAG.engine \
--shapes=images:1x3x640x640 \
--loadInputs=images:input_1x3x640x640.bin \
--exportOutput=/tmp/n_out_$TAG.json \
--iterations=1 --warmUp=0 --duration=0
done
3. Score the outputs
# score_dump.py
import json
for tag in ("notf32", "default", "fp16"):
o = [x for x in json.load(open(f"/tmp/n_out_{tag}.json")) if x["name"] == "output"][0]
dims = [int(x) for x in o["dimensions"].split("x")]
Q, W, v = dims[-2], dims[-1], o["values"]
n = sum(1 for q in range(Q) if v[q * W + 1] >= 0.45)
mx = max(v[q * W + 1] for q in range(Q))
print(f"{tag:8} detections>=0.45={n:<4} max_score={mx:.4f}")
4. Observed output
TensorRT 10.8.0.43 (CUDA 12.8, cuDNN 9.3.0) — correct at every precision:
```
notf32 detections>=0.45=12 max_score=0.9189
default detections>=0.45=12 max_score=0.9189
fp16 detections>=0.45=12 max_score=0.9185
```
TensorRT 10.14.1.48 (CUDA 13.1, cuDNN 9.17.1.4) — same ONNX, same input file, same GPU:
```
notf32 detections>=0.45=11 max_score=0.9199
default detections>=0.45=0 max_score=0.3117
fp16 detections>=0.45=0 max_score=0.3220
```
onnxruntime 1.20.1, CPU, FP32, same input: `detections>=0.45=12 max_score=0.9189`.
5. Full traceback of errors encountered
There is none, and that is the substance of this report. Every build ends with
`&&&& PASSED TensorRT.trtexec [TensorRT v101401]`, every engine deserializes, and every inference
completes. No `[E]` or `[W]` line in either the build or the run log refers to precision, accuracy,
fallback or unsupported kernels. The defect is visible only in the output values.
For the FP16 all-NaN case on the larger D-FINE-M variant the failure is equally silent — `PASSED`, and
the exported tensor contains:
```
{ “name” : “output”
, “dimensions” : “16x300x6”
, “values” : [ 0, nan, nan, nan, nan, nan, 1, nan, nan, nan, nan, nan, 2, nan, …
```
—
Notes for triage
* Two variables move together between our containers (TensorRT 10.8 → 10.14 and CUDA 12.8 → 13.1 /
cuDNN 9.3 → 9.17), so this report does not isolate TensorRT alone. We are happy to test intermediate
combinations if that helps narrow it.
* The architecture is DETR-style with multi-scale deformable attention. If a single kernel is suspect,
the deformable-attention `grid_sample` path and the LayerNorm feeding the score head look like the
first places to check. We have not run a layer-by-layer `polygraphy` comparison but can do so on request.
* Practical impact: on TensorRT 10.14 we found no build configuration that reproduces the reference
accuracy. `–noTF32` with a single-point optimization profile comes closest, and that combination
forces a static batch size — which in a DeepStream pipeline means the engine silently refuses any
batch smaller than the pinned one, so the whole detector goes quiet with no diagnostic.