Tao retinanet Triton server deployment

[user162297]

I’m trying to deploy a retinanet model trained on my custom dataset to nvidia triton inference server tritonserver:21.08-py3. Trained with TAO v3.21.11-tf1.15.4-py3, on a NVIDIA GeForce RTX 3060, But the server keeps failing with error below

E0520 15:04:50.516897 1 logging.cc:43] 3: getPluginCreator could not find plugin: BatchTilePlugin_TRT version: 1
E0520 15:04:50.519766 1 logging.cc:43] 1: [pluginV2Runner.cpp::load::291] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)

How to resolve

[Morganh]

Firstly, if you check “tao info --verbose” , training retinanet model should be using TAO v3.21.11-tf1.15.5-py3 .

After training, you can copy your .etlt model to triton server.
There is an easy way: Please run GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton without any change, then replace your .etlt model with the original one.

[user162297]

Solved by converting the .etlt file from “tao export” with a later version of tensorrt and deploy with a later triton container nvcr.io/nvidia/tritonserver:21.11-py3

tao-converter /tao_models/retinanet_model/retinanet_resnet18_epoch_080_its_trt8.etlt
-k nvidia_tlt
-d 3,544,960
-o NMS
-t fp16
-e /model_repository/retinanet_tao/1/model.plan