TensorRT model export/inference

Hello,
I trained RT-DETR thanks to modifying the rtdetr_detection_distillation.ipynb from tao tutorials. I trained the teacher model and succesfully distilled the student model. The student as well as the teacher model performed with over 80 mAP50 on the evaluation dataset.

But when i convert the student model to onnx and then to tensorrt engine and run inference as defined in the notebook, i get the result with blank labels even though something should be detected.

these are my onnx generation specs:
specs[“dataset”][“batch_size”] = 8 #If OOM error, decrease batch size

specs[“dataset”][“workers”] = 4

specs[“dataset”][“remap_mscoco_category”] = False

specs[“dataset”][“num_classes”] = num_classes + 1

specs[“dataset”][“augmentation”][“eval_spatial_size”] = [960, 640]

specs[“dataset”][“augmentation”][“train_spatial_size”] = [960, 640]

specs[“model”][“backbone”] = “convnext_tiny”

specs[“model”][“train_backbone”] = True

specs[“model”][“return_interm_indices”] = [1,2,3]

specs[“model”][“dec_layers”] = 6

specs[“model”][“enc_layers”] = 1

specs[“model”][“num_queries”] = 300

specs[“export”][“input_width”] = 960

specs[“export”][“input_height”] = 640

print(json.dumps(specs, sort_keys=True, indent=4))

this my spec for onnx to tensorrt
specs[“dataset”][“batch_size”] = 8 #If OOM error, decrease batch size

specs[“dataset”][“workers”] = 4

specs[“dataset”][“remap_mscoco_category”] = False

specs[“dataset”][“num_classes”] = num_classes + 1

specs[“dataset”][“augmentation”][“eval_spatial_size”] = [960, 640]

specs[“dataset”][“augmentation”][“train_spatial_size”] = [960, 640]

specs[“model”][“backbone”] = “convnext_tiny”

specs[“model”][“train_backbone”] = True

specs[“model”][“return_interm_indices”] = [1,2,3]

specs[“model”][“dec_layers”] = 6

specs[“model”][“enc_layers”] = 1

specs[“model”][“num_queries”] = 300

specs[“gen_trt_engine”][“tensorrt”][“data_type”] = “FP16”

inference spec:
specs[“dataset”][“batch_size”] = 1 #If OOM error, decrease batch size

specs[“dataset”][“workers”] = 4

specs[“dataset”][“remap_mscoco_category”] = False

specs[“dataset”][“num_classes”] = num_classes + 1

specs[“dataset”][“augmentation”][“eval_spatial_size”] = [960, 640]

specs[“dataset”][“augmentation”][“train_spatial_size”] = [960, 640]

specs[“model”][“backbone”] = “convnext_tiny”

specs[“model”][“train_backbone”] = True

specs[“model”][“return_interm_indices”] = [1,2,3]

specs[“model”][“dec_layers”] = 6

specs[“model”][“enc_layers”] = 1

specs[“model”][“num_queries”] = 300

specs[“inference”][“input_width”] = 960

specs[“inference”][“input_height”] = 640

specs[“inference”][“outline_width”] = 5

specs[“inference”][“color_map”] = {}

specs[“inference”][“color_map”][“crack”] = “red”

specs[“inference”][“color_map”][“pothole”] = “blue”

what could cause this? i do not think it is the problem of the model as the pth model performed well on evaluation dataset.
Thank you for any insigth

while exporting to onnx i get TracerWarnings: /usr/local/lib/python3.12/dist-packages/timm/models/layers/_init_.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {_name_} is deprecated, please import via timm.layers", FutureWarning)
sys:1: UserWarning:
‘spec.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See Automatic schema-matching | Hydra for migration instructions.
/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/hydra/hydra_runner.py:110: UserWarning:
‘spec.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See Automatic schema-matching | Hydra for migration instructions.
_run_hydra(
/usr/local/lib/python3.12/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See Changes to job's runtime working directory | Hydra for more information.
ret = run_job(
Export results will be saved at: /results/5d554949-36a6-4cbb-b8e1-942438fb457b/export
e[38;20m2025-11-06 07:21:57,516 - [TAO Toolkit] - INFO - Creating RT-DETR model from config for task: export, model_path: /results/5d554949-36a6-4cbb-b8e1-942438fb457b//results/2282d1ee-b1e3-4c3e-a161-929ae383dba2/distill/model_epoch_239.pth (model.py:58)e[0m
Creating RT-DETR model from config for task: export, model_path: /results/5d554949-36a6-4cbb-b8e1-942438fb457b//results/2282d1ee-b1e3-4c3e-a161-929ae383dba2/distill/model_epoch_239.pth
e[38;20m2025-11-06 07:21:57,516 - [TAO Toolkit] - INFO - Loading RT-DETR model from checkpoint: /results/5d554949-36a6-4cbb-b8e1-942438fb457b//results/2282d1ee-b1e3-4c3e-a161-929ae383dba2/distill/model_epoch_239.pth (model.py:102)e[0m
Loading RT-DETR model from checkpoint: /results/5d554949-36a6-4cbb-b8e1-942438fb457b//results/2282d1ee-b1e3-4c3e-a161-929ae383dba2/distill/model_epoch_239.pth
e[38;20m2025-11-06 07:21:58,479 - [TAO Toolkit] - INFO - RT-DETR model loaded from checkpoint successfully. (model.py:110)e[0m
RT-DETR model loaded from checkpoint successfully.
/usr/local/lib/python3.12/dist-packages/torch/nn/functional.py:6178: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
embed_dim == embed_dim_to_check
/usr/local/lib/python3.12/dist-packages/torch/nn/functional.py:6338: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
q_scaled = q * math.sqrt(1.0 / float(E))
/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/rtdetr/model/rtdetr_decoder.py:379: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
level_start_index = torch.tensor(level_start_index, dtype=torch.int64, device=proj_feats[0].device)
/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/deformable_detr/model/ops/modules.py:134: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
e[38;20m2025-11-06 07:22:05,047 - [TAO Toolkit] - INFO - ONNX file stored at /results/5d554949-36a6-4cbb-b8e1-942438fb457b/model.onnx (export.py:193)e[0m
ONNX file stored at /results/5d554949-36a6-4cbb-b8e1-942438fb457b/model.onnx
/usr/lib/python3.12/tempfile.py:1075: ResourceWarning: Implicitly cleaning up <TemporaryDirectory ‘/tmp/tmpoats267v’>
_warnings.warn(warn_message, ResourceWarning)

I managed to run evaluation directly on the exported engine with opset 18 and different precisions, evalluation gives results such as

Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.682AP: 0.296830177
AP50: 0.516228199
AP75: 0.307368338
APl: 0.455235362
APm: 0.214824483
APs: 0.005592352
ARl: 0.682352960
ARm: 0.369892478
ARmax1: 0.307692319
ARmax10: 0.446153849
ARmax100: 0.482248515
ARs: 0.087499999
Finished evaluation.

so obviously the model works, but when running inference i get only blank label files, no detections