List of all methods of getting accelerated computing on Jetson Xavier

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Description

I have been trying to get faster (real-time) inference for my Mask-RCNN custom model on the Jetson Xavier. Right now, the setup takes around 15s and then each inference takes 10s per frame, which is very slow. I am informed that Deepstream can be helpful to increase the data pipeline speed and TensorRT can speed up the inference through model optimisation.

1). What is the entire procedure for getting a faster inference ?
2) What all options do I have ?
3) Regarding TensorRT, I am aware that it provides TRT-inference-engine, ONNX-parsers, UFF-parsers, etc. Is there a way to select what works for me ?

3

Hi,

1.
You can find an example below.
Although it is based on the Deepstream 4.0, the procedure is very similar:

2.
The above example is for TensorFlow pretrained model.
If retraining with TLT is acceptable, you can find another example below:
https://docs.nvidia.com/metropolis/TLT/tlt-user-guide/text/instance_segmentation/mask_rcnn.html

3.
It depends on the frameworks you use for training the MaskRCNN model.

  • TensorFlow v1.x → uff parser.
  • TensorFlow v2.x → onnx parser
  • PyTorch → onnx parser.

Thanks.

4

My understanding

This is what I have understood till now.

The docs say:

TLT has been designed to integrate with DeepStream SDK, so models trained with TLT will work out of the box with it.

To deploy a model trained by TLT to DeepStream, you have two options:

  • Option 1 : Integrate the .etlt model directly in the DeepStream app. The model file is generated by export.
  • Option 2 : Generate a device-specific optimized TensorRT engine using tlt-converter . The generated TensorRT engine file can also be ingested by DeepStream.

For this, I will need to train my Mask-RCNN model from scratch and then export the model to .tlt and .etlt. Integration with Deepstream is explained well here. 🤠

I am planning to complete these steps locally on my dGPU system using Deepstream 5.0 docker container and then migrate to Jetson. Is this recommended ?

5

Hi,

YES. You can get an optimized .etlt model from TLT library.
Thanks.

7

What are its advantages over TensorRT ? Should I go for it ?