Help for converting .onnx to TensorRT

[viraldoshi321]

Hi I am a beginner in this field so I require some help. I have a NVIDIA AGX Orin 64 GP Developer Kit and want to implement Realtime Facial Recognition on it. I read the documentation (Developer Guide) but have some queries regarding model conversion

1. Can I convert any model from .onnx to tensorRT engine (eg. arcFace, Resnet100, etc.) using trtexec ??
2. If my TensorRT is utilizing the GPU properly is the above conversion the most efficient way to run the model ?
3. Is the usage of –useDLACore=0 while making the tensorRT engine enough to utilize DLA or do I have to menton anything when running these models on DeepStream ?

[AastaLLL]

Hi,

1. In general yes, but if your model contains a special layer, TensorRT might not have a corresponding implementation for it.
2. Yes, efficient in both performance and memory.
3. You can put the model on DLA with --useDLACore=[ID] when inferring with trtexec.
For Deepstream, please find below the topic to modify the config to run on DLA.

How to use DLA in deepstream-yolov5

Hi, Please add the below configure to enable DLA inference: enable-dla=1 use-dla-core=0 But please noted that you would need to remove the serialized engine file first. Or Deepstream will reuse it instead of recreating. Thanks.

Thanks.

[viraldoshi321]

Thanks a lot @AastaLLL How do I verify whether my .engine file of the model I generated is using DLA.
The execution was successful and I passed --useDLACore=0 --allowGPUFallback as arguements when I converted my onnx file to engine. Just for your reference I am using arcface_r100_v1.onnx and converting that to engine. I later plan to use it in Deepstream.

[AastaLLL]

Hi,

You should see the corresponding info with trtexec:

For example:

$ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/resnet50/ResNet50.onnx --saveEngine=ResNet50.trt --useDLACore=0 --allowGPUFallback
...
[04/10/2024-05:31:18] [I] === System Options ===
[04/10/2024-05:31:18] [I] Device: 0
[04/10/2024-05:31:18] [I] DLACore: 0(With GPU fallback)
[04/10/2024-05:31:18] [I] Plugins:
...
[04/10/2024-05:31:25] [I] [TRT] ---------- Layers Running on DLA ----------
[04/10/2024-05:31:25] [I] [TRT] [DlaLayer] {ForeignNode[node_of_gpu_0/conv1_1 + node_of_gpu_0/res_conv1_bn_1...node_of_gpu_0/pred_1]}
[04/10/2024-05:31:25] [I] [TRT] ---------- Layers Running on GPU ----------
[04/10/2024-05:31:25] [I] [TRT] [GpuLayer] SHUFFLE: reshape_after_node_of_gpu_0/pred_1
[04/10/2024-05:31:25] [I] [TRT] [GpuLayer] SOFTMAX: (Unnamed Layer* 180) [Softmax]
...
&&&& PASSED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/resnet50/ResNet50.onnx --saveEngine=ResNet50.trt --useDLACore=0 --allowGPUFallback

Thanks.