API usage error of torch2trt on Jetson Orin nano

user83910 · July 20, 2023, 3:16am

Hi.
I’m facing trouble to use torch2trt. When I run this code, it will stop.

code
model=torchvision.models.efficientnet_v2_s(weights=weights)
model.eval()
model=model.to(device)
model_trt=torch2trt(model,,fp16_mode=True,int8_mode=True,max_batch_size=1)

result
[TRT] [E] 3: [builderConfig.cpp::canRunOnDLA::493] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builderConfig.cpp::canRunOnDLA::493, condition: dlaEngineCount > 0)

I guess torch2trt or tensorRT try to use DLA which Orin nano doesn’t have.
How can I set “No use DLA”?

Thank you for your time.

Environment

TensorRT Version: 8.5.2.2
GPU Type: Jetson Orin Nano (Developer kit)
Nvidia Driver Version: JetPack 5.1.1
CUDA Version: 11.4.315
CUDNN Version: 8.6.0

Operating System + Version: JetPack 5.1.1
Python Version (if applicable): 3.8.10
PyTorch Version (if applicable): 2.0.0+nv23.5
**torch2trt Version **: 0.4.0

AastaLLL · July 20, 2023, 5:02am

Hi,

Please use dla=False when calling torch_trt:
Below is a related topic for your reference:

Thanks.

user83910 · July 20, 2023, 6:05am

Thank you for your support.

I added dla=False but same error appears.
model_trt=torch2trt(model,,fp16_mode=True,int8_mode=True,max_batch_size=1,dla=False)

Did I add at the wrong place?

AastaLLL · July 21, 2023, 7:04am

Hi,

What is the device_types value in your use case?

github.com

NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/torch2trt.py#L670


      
                    max_workspace_size=1<<25,
                    strict_type_constraints=False,
                    keep_network=True,
                    int8_mode=False,
                    int8_calib_dataset=None,
                    int8_calib_algorithm=DEFAULT_CALIBRATION_ALGORITHM,
                    use_onnx=False,
                    default_device_type=trt.DeviceType.GPU,
                    dla_core=0,
                    gpu_fallback=True,
                    device_types={},
                    min_shapes=None,
                    max_shapes=None,
                    opt_shapes=None,
                    onnx_opset=None,
                    max_batch_size=None,
                    avg_timing_iterations=None,
                    **kwargs):
          
          # capture arguments to provide to context
          kwargs.update(locals())

Thanks.

user83910 · July 25, 2023, 7:59am

Hi,

I’m sorry. I didn’t get what you mean because of lacking knowledge.
How can I check the device_types value?

Thank you.

AastaLLL · August 3, 2023, 3:23am

Hi,

You can find some info in the below link:

github.com/NVIDIA-AI-IOT/torch2trt

Dla support

NVIDIA-AI-IOT:master ← jaybdub:dla_support

opened 02:28AM - 25 Feb 21 UTC

jaybdub

+9144 -816

This PR enables DLA support through torch2trt. Please note this is subject to c…hange. ### Todo - [x] concurrent model execution profiling (DLA should free up GPU, also there are two DLAs) - [x] 2 model GPU 0 model DLA - [x] 1 model GPU 1 model DLA - [x] 0 model GPU 2 model DLA - [x] increment version to enable backtracking to legacy conversion - [x] update documentation with DLA instructions / benchmarks - [ ] sanity check models on imagenet data ### Basic usage This will enable DLA usage globally by setting the default device type. If a particular layer is not supported by DLA, it will fall back to GPU unless ``gpu_fallback=False`` is set (it is True by default). Please note, either ``fp16_mode=True`` or ``int8_mode=True`` are required for DLA. ```python import torch from torch2trt import trt, torch2trt from torchvision.models import resnet18 model = resnet18(pretrained=True).cuda().eval() data = torch.randn(1, 3, 224, 224).cuda() model_trt = torch2trt(model, [data], fp16_mode=True, default_device_type=trt.DeviceType.DLA, dla_core=0) ``` ### Set device for particular modules In some cases you may want only a particular part of the model to run on DLA. This may be useful if your model has a large block that can run on DLA, but other parts with mixed supported / unsupported components which may occur overhead cost. This is done at a module-level granularity. Each lower-level set will override a higher level module, but only apply to layers added within the module. For example, here we convert the resnet18 blocks ``layer1`` and ``layer2`` and the first block in ``layer3`` to run on DLA. ```python model_trt = torch2trt(model, [data], default_device_type=trt.DeviceType.GPU, fp16_mode=True, dla_core=0, device_types={ model.layer1: trt.DeviceType.DLA, model.layer2: trt.DeviceType.DLA, model.layer3[0]: trt.DeviceType.DLA }) ``` You could do the inverse of this if you wanted. ```python model_trt = torch2trt(model, [data], default_device_type=trt.DeviceType.DLA, fp16_mode=True, device_types={ model.layer1: trt.DeviceType.GPU, model.layer2: trt.DeviceType.GPU, model.layer3[0]: trt.DeviceType.GPU }) ``` In any instance, a DLA layer will run on GPU if ``gpu_fallback=True``. If you would prefer to disable this behavior, you can set ``gpu_fallback=False``, but TensorRT optimization may fail internally. TODO - validate existing test cases. may be cool to log DLA supported layers https://nvidia-ai-iot.github.io/torch2trt/master/converters.html - perform sanity checks on real world model / data

Could you try if the following command can work?

model_trt = torch2trt(model, [data], default_device_type=trt.DeviceType.GPU, fp16_mode=True, ...)

Thanks.

user83910 · August 8, 2023, 5:52am

Hi,

import torch
from torch2trt import torch2trt,trt
import torchvision

device='cuda'
weights=torchvision.models.segmentation.FCN_ResNet50_Weights.DEFAULT
model=torchvision.models.segmentation.fcn_resnet50(weights=weights).to(device).eval()

input_size=[1,3,256,256]
x=torch.zeros(input_size).to(device)
model_trt=torch2trt(model,[x],default_device_type=trt.DeviceType.GPU, fp16_mode=True,int8_mode=True,max_batch_size=1,dla=False)

It doesn’t work. Same error occurred.

Thank you.

AastaLLL · August 18, 2023, 7:24am

Hi,

Thanks for the feedback.

We are checking this issue internally.
Will share more information with you later.

AastaLLL · August 31, 2023, 9:07am

Hi,

We are trying to reproduce this issue in our environment.

It seems that you have installed torch2trt on JetPack 5.x.
Could you share how do you install it?

We got another topic that reporting the tool is not working on JetPack 5.

Thanks.

user83910 · September 12, 2023, 4:44am

Hi,
Sorry for late reply. I installed tensorRT and torch2trt by following way.

sudo dpkg -i nv-tensorrt-local-repo-l4t-8.5.2-cuda-11.4_1.0-1_arm64.deb
sudo cp /var/nv-tensorrt-local-repo-l4t-8.5.2-cuda-11.4/* -keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get install tensorrt
sudo apt-get install python3-libnvinfer-dev
git clone https://github.com/NVIDIA-AI-IOT/torch2trt
cd torch2trt
sudo python3 setup.py install

Thank you for your support.

system · October 9, 2023, 8:17am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.