Error when running retinanet model in onnxruntime with tensorrt execution accelerator

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc): A5000 GPU
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc): RetinaNet
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here): TAO 5.2
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

For our application, I want to run the TAO Retinanet model in Onnxruntime with TensorRT execution acceleration in Triton server. I am using Trition tritonserver:23.06-py3. This is the triton model config I am using:
config.txt (580 Bytes)

However, Onnxruntime can’t load the model due to custom TRT NMS:

UNAVAILABLE: Internal: onnx runtime error 10: Load model from /models/retinanet_onnxruntime/1/model.onnx failed:This is an invalid model. In Node, (“NMS”, NMSDynamic_TRT, “”, -1) : (“anchor_data”: tensor(float),“loc_data”: tensor(float),“conf_data”: tensor(float),) → (“NMS”: tensor(float),“NMS_1”: tensor(float),) , Error No Op registered for NMSDynamic_TRT with domain_version of 15

Running the model in tensorrt works fine, however, for our application we prefer to use onnxruntime with tensorRT acceleration.

Is there a way to load the model in onnxruntime and execute it with tensorrt acceleration?

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Please refer to Do we have to use BatchedNMSDynamic_TRT in the model output by ONNX? - #3 by Morganh and Errors while reading ONNX file produced by TAO 5 - #10 by Morganh.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.