TensorRT 8.5.2 conversion of RTDETR (object detection model) is failing


A clear and concise description of the bug or issue.


TensorRT Version:
GPU Type: RTX 3090
Nvidia Driver Version: 537.42
CUDA Version: 12.1
CUDNN Version: 8.9.6
Operating System + Version: ubuntu 20.04.5 LTS
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 2.0.1
Baremetal or Container (if container which image + tag): Container. From baseImage: nvcr.io/nvidia/pytorch:23.04-py3, upgrade pytorch

Relevant Files

The following command is used:
/usr/src/tensorrt/bin/trtexec --onnx=temp/rtdetr.onnx --workspace=4096 --shapes=images:1x3x640x640 --saveEngine=output/model.trt --avgRuns=10

Please fine the onnx files (opset 16 and 17), ErrorFile (that contains the errors), here: tensorrt_challenge - Google Drive

Hi @rajesh21 ,
Can you please help us with onnx model


Thank you for your response. I am not sure what you are asking. Are you asking where the location of the onnx file is?
It is located in two places that is already shared:

As indicated in the above post, the folder (tensorrt_challenge - Google Drive) also contains the error file as well.

Just want to follow up? Do you need any other additional information from us?

After resetting few flags (E.g., specifying batch size), the tensortRT file gets created but is unusable.

with open(engine_path, "rb") as f:
    engine_data = f.read()
engine = trt_runtime.deserialize_cuda_engine(engine_data)
engine is None

The command we used to convert is as follows:

     command = (
            f"/usr/src/tensorrt/bin/trtexec "
            f"--onnx={conversion_config.temp_dir/'rtdetr.onnx'}  "
            f"--workspace=16096 "
            f"--minShapes=images:1x3x640x640,orig_target_sizes:1x2 "
            f"--optShapes=images:4x3x640x640,orig_target_sizes:4x2 "
            f"--maxShapes=images:4x3x640x640,orig_target_sizes:4x2 "