Inference on .etlt model

[p.vahidinia]

Hi.
Is it possible to inference on exported .etlt for segmentation and object detection models??

[Morganh]

Yes, you can deploy the .etlt model for object detection in deepstream. See user guide for reference. For example, YOLOv4 — TAO Toolkit 3.22.05 documentation.
Or you can refer to the github deepstream_tao_apps/configs at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub directly.

But for Unet, please note that current DS cannot parse onnx etlt model, so you need to convert the etlt model to TensoRT engine first use tao-converter. Then run inference against the trt engine.
https://docs.nvidia.com/tao/tao-toolkit/text/semantic_segmentation/unet.html#deploying-to-deepstream

[p.vahidinia]

Except for deepstream, is it possible to inference on the .etlt model inside the TAO or TLT container?

[Morganh]

No, TLT/TAO does not allow to run inference against the .etlt models. Because it already provides ways for end user to run inference against .tlt model.
More, it provides tao-converter to generate trt engine based on .etlt model. Thus, end user can run inference against trt engine or .tlt model.

[p.vahidinia]

What about Triton? Is it possible to inference .tlt, .etlt and trt.engine with Triton?

[Morganh]

Triton is running inference against model.plan .Actually it is also a tensorrt engine. It is generated by tao-converter. See tao-toolkit-triton-apps/download_and_convert.sh at main · NVIDIA-AI-IOT/tao-toolkit-triton-apps · GitHub

[p.vahidinia]

Thank you