Inference on .etlt model

Hi.
Is it possible to inference on exported .etlt for segmentation and object detection models??

Yes, you can deploy the .etlt model for object detection in deepstream. See user guide for reference. For example, YOLOv4 — TAO Toolkit 3.0 documentation.
Or you can refer to the github deepstream_tao_apps/configs at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub directly.

But for Unet, please note that current DS cannot parse onnx etlt model, so you need to convert the etlt model to TensoRT engine first use tao-converter. Then run inference against the trt engine.
https://docs.nvidia.com/tao/tao-toolkit/text/semantic_segmentation/unet.html#deploying-to-deepstream

1 Like

Except for deepstream, is it possible to inference on the .etlt model inside the TAO or TLT container?

No, TLT/TAO does not allow to run inference against the .etlt models. Because it already provides ways for end user to run inference against .tlt model.
More, it provides tao-converter to generate trt engine based on .etlt model. Thus, end user can run inference against trt engine or .tlt model.

1 Like

What about Triton? Is it possible to inference .tlt, .etlt and trt.engine with Triton?

Triton is running inference against model.plan .Actually it is also a tensorrt engine. It is generated by tao-converter. See tao-toolkit-triton-apps/download_and_convert.sh at main · NVIDIA-AI-IOT/tao-toolkit-triton-apps · GitHub

1 Like

Thank you

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.