(SHUFFLE): Unsupported on DLA switching to GPU

noir201 · March 24, 2024, 1:27pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson AGX Orin
• DeepStream Version6.2
**• JetPack Version (valid for Jetson only)**5.1
• TensorRT Version8.5.2
I have converted a facenet model (Face Recognition) from PyTorch to .ONNX to be used in deepstream configuration file. However, running the config file to generate engine file that is running on DLA will cause shuffle layers which were are not part of the model itself to fall back to GPU.
As per TRT documentation DLA do support shuffle layers but why in my case they fall back to the GPU. Additionally, how these shuffle layers were imported to the model since the original model does not have shuffle layers.
FaceNet_DLAEngine_unsupportedLayers.txt (9.8 KB)

I have attached the log file generated during engine file creation.

AastaLLL · March 25, 2024, 3:37am

Hi,

The layers are added automatically to meet DLA requirements.
You can find more info in the below repo:

github.com

NVIDIA/Deep-Learning-Accelerator-SW/blob/main/operators/README.md#onnx-operatorfunction-support-matrix

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators & Functions on Orin DLA

DLA operator functionality is exposed through the TensorRT builder, which internally links to DLA SW libraries (see [DLA Workflow](https://developer.nvidia.com/deep-learning-accelerator)). While some ONNX operators or functions may already be available in DLA SW, TensorRT may not expose them yet.
See below for the support matrix of ONNX operators & functions on Orin DLA. If you are interested in a specific DLA operator that is not supported through TensorRT yet, feel free to raise a [GitHub Issue](https://github.com/NVIDIA/Deep-Learning-Accelerator-SW/issues) and/or inform your NVIDIA representative (in particular for NVIDIA DRIVE customers).

See [General Restrictions](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#dla-lay-supp-rest) that apply to all operations below. Many of those ops are supported on Xavier DLA as well, see [Layer Support and Restrictions](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#dla-lay-supp-rest).

TensorRT 8.6 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md).

Note that the scripts in `op_reconstruction/` are intended as a recipe for how ops currently not supported on DLA can be decomposed into supported ops. Depending on your setup, you may choose to perform such op reconstructions in the ONNX domain post-training (as done here) or during the training process (for example in TensorFlow or PyTorch). The case of "Native" in the DLA SW support column and "Reconstruction" in the TensorRT support column indicates that an op can be supported through TensorRT by decomposing it into other DLA ops already supported by TensorRT.

Below Operator Support Matrix requires the following minimum system config (the OS by default gets shipped with the DLA SW and TensorRT versions to its right):

| **Hardware platform** | **OS**               | **DLA SW version** | **TensorRT version** |
| ----------------- | ---------------- | -------------- | ---------------- |
| DRIVE Orin (Automotive)        | DRIVE OS 6.0.6.0 | DLA 3.12.0     | TensorRT 8.5.10  |
| Jetson Orin (Embedded)      | JetPack 5.1.1    | DLA 3.12.1     | TensorRT 8.5.2   |
| DRIVE Orin (Automotive)        | DRIVE OS 6.0.7.0 | DLA 3.13.0     | TensorRT 8.6.10  |

This file has been truncated. show original

Constant

Thanks.

noir201 · March 25, 2024, 10:45am

Thank you for your reply.
I see that Shuffle layers are added 28 time and constant layers are added 28 time. The model it self has 22 layers as below.

what’s the exact requirement by the DLA that requires these shuffle/constant layers to be added. Is it due to reconstruction process? becuase this much of shuffle layers have caused delay in processing as they fall back to the gpu which will cause back and forth data transmission between Gpu/dla

noir201 · April 1, 2024, 8:20pm

I see contact is presented by (See REF) in tensorrt column and Native in DLA. what does this mean?
In my case it fell back to the GPU why did it fall back if it’s native to DLA

noir201 · April 4, 2024, 7:37am

@AastaLLL please support

AastaLLL · April 8, 2024, 9:20am

Hi,

Sorry for the late update.

It looks like a new topic has been opened recently.
We can follow up the discussion there and close this one.

Thanks.

system · April 22, 2024, 9:20am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.