Deploy SigLip Model on Jetson Orin 32GB on DLA

sihouzi21c · July 23, 2025, 7:11am

Subject: Deployment Inquiry: SigLip Vision Model on Jetson Orin 32GB with DLA

Hi all,

Apologies for the interruption. I’m working on deploying the vision component of the SigLip model to the DLA on a Jetson Orin 32GB device.

Target

Deploy the vision segment of SigLip on Jetson Orin 32GB using DLA.

Issue

Most layers default to the GPU, with the log reporting:

"Unsupported on DLA. Switching this layer's device type to GPU."

Layers affected:

SHUFFLE, CONSTANT, CAST, GATHER, MATRIX_MULTIPLY, CONVOLUTION, ELEMENTWISE, REDUCE, UNARY, SLICE.

(Note: I’ve converted Linearlayers to 1x1 convolutions to avoid GPU fallback for those.)

Key question: How can I maximize compute on DLA and resolve these fallbacks?

Request for Guidance

I’d appreciate suggestions on deploying SigLip’s vision model to DLA, specifically:

Should I rewrite the forward function using DLA-supported operators? Are there alternative approaches?
Are there documents for deploying SigLip on DLA? Existing materials only cover ResNet, MobileNet, etc.

Supporting Materials

Log: Details deployment strategy and layer fallbacks.
Code: For reproducibility.

Reproduction Command:
python comprehensive_dla_accuracy_validation.py
Results will be siglip_1layer_linear_only_dla_build.log

Thank you for your support!

Best regards
Material.zip (94.6 KB)

carolyuu · July 23, 2025, 7:16am

*** Please note that this reply is generated by LLM automatically ***

To address the inquiry about deploying the SigLip vision model on a Jetson Orin 32GB device with DLA (Deep Learning Accelerator), we need to consider several aspects, including maximizing compute on DLA, resolving GPU fallbacks, and understanding the deployment strategy for the model. The goal is to efficiently utilize the DLA for the vision segment of the SigLip model, minimizing reliance on the GPU for computations.

Understanding DLA and GPU Fallbacks

DLA is a hardware accelerator designed for deep learning workloads, offering significant performance improvements for supported operations. However, not all layers or operations in a neural network model are compatible with DLA. When a layer cannot be executed on the DLA, it falls back to the GPU, which can lead to performance degradation. The conversion of layers to 1x1 convolutions to avoid GPU fallback is a step in the right direction but may not be sufficient for all layers or may introduce additional computational overhead.

Maximizing Compute on DLA

To maximize the use of DLA for the SigLip vision model:

Layer Compatibility: Identify which layers in the SigLip model are compatible with DLA. NVIDIA provides documentation and tools to help determine layer compatibility.
Model Optimization: Optimize the model to use DLA-supported operators. This might involve rewriting parts of the model or using model optimization tools provided by NVIDIA.
TensorRT: Utilize TensorRT, NVIDIA’s inference optimizer, which can help in optimizing the model for DLA execution. TensorRT can automatically optimize the model for better performance on NVIDIA hardware, including DLAs.
DLA-Supported Operators: Focus on using operators that are natively supported by DLA. This might require modifying the forward function of the model to better align with DLA capabilities.

Alternative Approaches

Model Pruning: Prune the model to reduce complexity and potentially make more layers compatible with DLA.
Knowledge Distillation: Use knowledge distillation to create a smaller, DLA-friendly model that mimics the behavior of the original model.
Quantization: Apply quantization techniques to reduce the precision of model weights and activations, which can make the model more DLA-friendly and improve inference speed.

Documentation and Deployment Strategy

While there might not be specific documentation for deploying SigLip on DLA, NVIDIA provides extensive resources for deploying various models (like ResNet, MobileNet) on Jetson devices with DLA. These resources can serve as a starting point:

NVIDIA Jetson Documentation: Offers guides on optimizing models for DLA and deploying them on Jetson devices.
TensorRT Documentation: Provides detailed information on optimizing and deploying models with TensorRT.
NVIDIA Deep Learning SDK: Includes tools and libraries for developing and deploying AI applications on NVIDIA hardware, including DLAs.

Reproducibility and Results

For reproducibility, it’s essential to document the exact steps taken to optimize and deploy the SigLip model on the Jetson Orin 32GB with DLA. This includes:

Model Modifications: Detail any changes made to the model to support DLA.
Optimization Techniques: List the optimization techniques used (e.g., TensorRT, model pruning, quantization).
Performance Metrics: Report on the performance metrics (e.g., inference speed, accuracy) before and after optimization to quantify the improvements.

By following these steps and leveraging NVIDIA’s resources and tools, you can effectively deploy the SigLip vision model on a Jetson Orin 32GB device with DLA, maximizing the use of the DLA for improved performance and efficiency.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

AastaLLL · July 23, 2025, 9:12am

Hi,

Which JetPack version do you use?
It’s recommended to upgrade to the latest since we have added some support in JetPack 6.

Unfortunately, we don’t have a tutorial for SigLip on DLA.
But in general, you can follow the below repo to implement your use case:

For a non-supported layer, you can either:

Modify the layer:

Or
2. Reformat the layer: (ex. ReduceMean → AveragePool)

github.com/NVIDIA/Deep-Learning-Accelerator-SW

scripts/prepare_models/resnet50.py

main


      
          #
          """ONNX preparation for ResNet-50."""
          import os
          import onnx
          import numpy as np
          import onnx_graphsurgeon as gs
          from onnx import shape_inference
          import common
          
          
          def simplify_classification_head(graph):
              node_dict = dict()
              for node in graph.nodes:
                  node_dict[node.name] = node
          
              # ReduceMean -> AveragePool:
              node_dict['resnet_model/Mean'].op = 'AveragePool'
              node_dict['resnet_model/Mean'].attrs = {'kernel_shape': [7, 7]}
          
              # {Squeeze, MatMul, Add} -> 1x1 Conv:
              node_dict['resnet_model/Squeeze'].inputs.clear()

Thanks.

sihouzi21c · July 23, 2025, 11:03am

Thank you for your reply! My JetPack version is 6.2. I followed the material you provided previously, but encountered compatibility issues with several layers.

So, based on this, it seems there is currently no scheduled release for deploying SigLip on the Jetson Orin DLA. If I wish to proceed with deployment, would I need to manually adapt those unsupported layers?

Additionally, my TensorRT version is 10.3. Would the guidance in the following link still be applicable/correct for this version? https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/work-with-dla.html#dla-supported-layers-and-restrictions

Best wishes,

AastaLLL · July 24, 2025, 6:11am

Hi,

DLA deploys models in a layer-based manner.
Do you know which layer is not working on the DLA?

You can find the detailed support matrix for the DLA operator in the link below:

github.com/NVIDIA/Deep-Learning-Accelerator-SW

operators/README.md

main

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators & Functions on Orin DLA

DLA operator functionality is exposed through the TensorRT builder, which internally links to DLA SW libraries (see [DLA Workflow](https://developer.nvidia.com/deep-learning-accelerator)). While some ONNX operators or functions may already be available in DLA SW, TensorRT may not expose them yet.
See below for the support matrix of ONNX operators & functions on Orin DLA. If you are interested in a specific DLA operator that is not supported through TensorRT yet, feel free to raise a [GitHub Issue](https://github.com/NVIDIA/Deep-Learning-Accelerator-SW/issues) and/or inform your NVIDIA representative (in particular for NVIDIA DRIVE customers).

See [General Restrictions](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#dla-lay-supp-rest) that apply to all operations below. Many of those ops are supported on Xavier DLA as well, see [Layer Support and Restrictions](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#dla-lay-supp-rest).

TensorRT 8.6 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md).

Note that the scripts in `op_reconstruction/` are intended as a recipe for how ops currently not supported on DLA can be decomposed into supported ops. Depending on your setup, you may choose to perform such op reconstructions in the ONNX domain post-training (as done here) or during the training process (for example in TensorFlow or PyTorch). The case of "Native" in the DLA SW support column and "Reconstruction" in the TensorRT support column indicates that an op can be supported through TensorRT by decomposing it into other DLA ops already supported by TensorRT.

Below Operator Support Matrix requires the following minimum system config (the OS by default gets shipped with the DLA SW and TensorRT versions to its right):

| **Hardware platform** | **OS**               | **DLA SW version** | **TensorRT version** |
| ----------------- | ---------------- | -------------- | ---------------- |
| DRIVE Orin (Automotive)        | DRIVE OS 6.0.6.0 | DLA 3.12.0     | TensorRT 8.5.10  |
| Jetson Orin (Embedded)      | JetPack 5.1.1    | DLA 3.12.1     | TensorRT 8.5.2   |
| DRIVE Orin (Automotive)        | DRIVE OS 6.0.7.0 | DLA 3.13.0     | TensorRT 8.6.10  |

This file has been truncated. show original

The document is overall similar but you can find one for TensorRT 10.3 in the link below:

Thanks.

sihouzi21c · July 24, 2025, 6:56am

Thank you. I will follow these documents.

system · August 13, 2025, 12:55am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Run AI models completely on Jetson AGX Orin DLAs Jetson Nano dla	4	476	April 20, 2024
DLA specific classification/detection ml model Jetson AGX Orin tensorrt , jetson-inference , dla	4	436	August 25, 2023
Jetson Orin: All layers pushed to GPU, zero layers on DLA Jetson AGX Orin tensorrt , dla	7	1069	April 26, 2023
Jetson Orin Nano having difficulties to run models on DLA(s) Jetson Nano dla	2	1295	July 12, 2023
[TensorRT] Running a simple onnx model on Jetson Xavier DLA Jetson Xavier NX tensorrt , onnx	12	3141	August 10, 2022
Compute time in DLA slower than expected Jetson AGX Orin dla	5	977	July 28, 2023
TRT not showing layers on DLA Jetson AGX Orin Jetson AGX Orin dla	4	50	December 18, 2024
How to use DLA correctly? Jetson Orin NX dla , gpu	6	274	March 10, 2025
DLA1 is offline on my Jetson Orin Jetson Orin NX dla	4	982	June 13, 2023
Layer inspection on trt model Jetson AGX Orin jetson-inference	4	735	October 30, 2022

Deploy SigLip Model on Jetson Orin 32GB on DLA

​​Target​​

​​Issue​​

​​Request for Guidance​​

​​Supporting Materials​​