Tao retinanet Triton server deployment

I’m trying to deploy a retinanet model trained on my custom dataset to nvidia triton inference server tritonserver:21.08-py3. Trained with TAO v3.21.11-tf1.15.4-py3, on a NVIDIA GeForce RTX 3060, But the server keeps failing with error below

E0520 15:04:50.516897 1 logging.cc:43] 3: getPluginCreator could not find plugin: BatchTilePlugin_TRT version: 1
E0520 15:04:50.519766 1 logging.cc:43] 1: [pluginV2Runner.cpp::load::291] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)

How to resolve

Firstly, if you check “tao info --verbose” , training retinanet model should be using TAO v3.21.11-tf1.15.5-py3 .

After training, you can copy your .etlt model to triton server.
There is an easy way: Please run GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton without any change, then replace your .etlt model with the original one.

Solved by converting the .etlt file from “tao export” with a later version of tensorrt and deploy with a later triton container nvcr.io/nvidia/tritonserver:21.11-py3

tao-converter /tao_models/retinanet_model/retinanet_resnet18_epoch_080_its_trt8.etlt
-k nvidia_tlt
-d 3,544,960
-o NMS
-t fp16
-e /model_repository/retinanet_tao/1/model.plan

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.