The problem of deploying yolov8 on jetson orin with jetpack 5.1.1

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
,
1

Hi, I am trying to develop something with a jetson orin nano module. I trained and exported the model follow: Export - Ultralytics YOLOv8 Docs on another computer, and tried to deploy it on jetson, then I got this:

2023-12-27 20:20:08.470905524 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:20:08 WARNING] onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
2023-12-27 20:21:14.354644247 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:14 WARNING] Tactic Device request: 538MB Available: 443MB. Device memory is insufficient to use tactic.
2023-12-27 20:21:14.424114857 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:14 WARNING] Skipping tactic 3 due to insufficient memory on requested size of 538 detected for tactic 0x0000000000000004.
2023-12-27 20:21:14.452337772 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:14 WARNING] Tactic Device request: 538MB Available: 532MB. Device memory is insufficient to use tactic.
2023-12-27 20:21:14.473431019 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:14 WARNING] Skipping tactic 8 due to insufficient memory on requested size of 538 detected for tactic 0x000000000000003c.
2023-12-27 20:21:14.498303943 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:14 WARNING] Tactic Device request: 538MB Available: 531MB. Device memory is insufficient to use tactic.
2023-12-27 20:21:14.518971600 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:14 WARNING] Skipping tactic 13 due to insufficient memory on requested size of 538 detected for tactic 0x0000000000000074.
2023-12-27 20:21:56.396434906 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:56 WARNING] Tactic Device request: 806MB Available: 409MB. Device memory is insufficient to use tactic.
2023-12-27 20:21:56.417556252 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:56 WARNING] Skipping tactic 3 due to insufficient memory on requested size of 806 detected for tactic 0x0000000000000004.
2023-12-27 20:21:56.457616974 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:56 WARNING] Tactic Device request: 806MB Available: 409MB. Device memory is insufficient to use tactic.
2023-12-27 20:21:56.478414558 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 log] [2023-12-27 12:21:56 WARNING] Skipping tactic 8 due to insufficient memory on requested size of 806 detected for tactic 0x000000000000003c.

The setup of the jetson module is as below:
jetpack: 5.1.1
tensorRT : 8.5.2.2
onnx-runtime: 1.15.1 ( Jetson Zoo - eLinux.org
python: 3.8.10

The computer which I trained and export the model with:
OS: Rocky Linux 9.3
GPU: Quadro RTX 4000 with CUDA 12.1
onnx: 1.15.1
python: 3.8.18

I exported the model with only one argument(format=‘onnx’), maybe I should export it into engine only? Before I tried, I found some post explaining that tensorRT would be call by the onnx-runtime, so I haven’t tried the engine file yet.

3

Hi,

Do you get the inference results?

Based on your logs, it’s just some warning rather than an error.
The message mainly indicates some algorithms are not available due to the memory limit.

This is common on the memory-limited Jetson platform like Orin Nano.
Thanks.

4

Hi AastaLLL, thanks for your attention.
I did get the inference result, but it took a really long time. Maybe it was the cost of warming up? The first warning log made me worried that the setup of environment, or the parameters for model export. As far as I know, the weights of the exported yolov8 model would be FP32, rather than INT64. Please forgive me for this silly question, I am not a skillful developer on DL.

5

Hi, AastaLLL. I just tried to export model into engine file and perform inference with the same device ( the one with Quadro RTX-4000). When I exported, the same warning :

onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.

came up again. Though I got the engine file successfully, the tensorRT API runtime.deserialize_cuda_engine(serialzied_engine) returned none, and I got a serialization error ( rt::kPLAN_MAGIC_TAG failed.) from runtime.cpp. I’m not sure these things are relevant.

6

Hi,

INT64 might be used in PyTorch for some constants.
However, since the value is usually within the range of INT32, the casting doesn’t really lead to an issue.

The second error related to kPLAN_MAGIC_TAG indicates that you are trying to infer an engine on the environment that differs from the creation.
Please note that you will need to re-convert the TensorRT engine once the software/hardware environment is changed.

Thanks.

8

Hi Im getting the same issue, with the casting taking about 2.5 minutes. Afterwards the model will run fine. Did somebody find a way of performing this cast once and then saving the model? Exporting the model with any of the flags Ultralytics provide (half, dynamic, simplified, int8) did not work for me. On a jetson orin

Orin
Package: nvidia-jetpack
Version: 5.1-b147
Architecture: arm64

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.