Custom Models for Deepstream (TLT)

[lambda1]

Hi,

We are having some trouble using custom models in Deepstream. We tried multiple models (onnx, caffe, uff) however the models seem to be TRT (TensorRT) incompatible. We would like to know can we make a model TRT compatible, is there some documentation for that? Also are there readily availabe models that are TRT compatible which would help in easy prototyping?

Some other things that we tried:

1. We tried using the TensorFlow -TRT converter, but when we supply the engine file in Deep Stream (via the config file), it doesn’t build
2. We tried using TLT as well, but that is throwing an error while creating TF-Records.

Specifically:

1. Could you tell us how do we use custom models in Deepstream?
2. How do we use TensorFlow -TRT converter with Deepstream
3. A way to use TLT effectively (some blog post, some documentation) because we followed the current documentation and it still threw an error.

Specs:
• Hardware Platform (Jetson / GPU): Jetson TX2
• DeepStream Version: 5.1
• JetPack Version (valid for Jetson only): 4.5.1

[Morganh]

In TLT,

1. see Overview - NVIDIA Docs,
   there are many purpose-built models for use. There are also some examples for how to run it in deepstream. See https://docs.nvidia.com/tlt/tlt-user-guide/text/deepstream_tlt_integration.html
   and https://docs.nvidia.com/tlt/tlt-user-guide/text/deepstream_tlt_integration.html#peoplenet

2. If you train a TLT model, you will get the tlt format model. Then, please export it to etlt format model. And use the tool “tlt-converter” to generate trt engine. Both etlt model and trt engine can be deployed in deepstream. For example, https://docs.nvidia.com/tlt/tlt-user-guide/text/object_detection/yolo_v4.html#deploying-to-deepstream
   https://docs.nvidia.com/tlt/tlt-user-guide/text/tensorrt.html

3. See Overview - NVIDIA Docs

To learn more about how to use TLT, read the technical blogs which provide step-by-step guide to training with TLT:

Below blogs are compatible with TLT2.0

• Learn to Train with PeopleNet and other pre-trained model using TLT.
• Learn how to train Instance segmentation model using MaskRCNN with TLT.
• Learn how to improve INT8 accuracy using Quantization aware training(QAT) with TLT.

Below blogs are compatible with TLT3.0-dp-py3 or 3.0-py3 verison.

• Learn how to Create a real time license plate detection and recognition app
• Learn how to Prepare state of the art models for classification and object detection with TLT
• Learn more on Building and Deploying Conversational AI models Using the NVIDIA Transfer Learning Toolkit
• Pose estimation dev blog : Training and Optimizing a 2D Pose-Estimation Model with the NVIDIA Transfer Learning Toolkit. Part 1 | Part 2