Setup TensorRT in DRIVE Orin

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
, ,
1

Please provide the following info (tick the boxes after creating this topic):

Software Version
DRIVE OS 7.0.3
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Thor Developer Kit (940-63960-0010-000)
DRIVE AGX Thor Developer Kit (940-63960-0012-000)
other

Host Machine Version
native Ubuntu Linux 24.04 Host installed with DRIVE OS Docker Containers
other native Ubuntu Linux 22.04 Host installed with DRIVE OS Docker Containers

Issue Description

With DRIVE Orin, I can use TensorRT after flashing, but with Drive Thor, I noticed that TensorRT is not already installed by default after flashing.
Regarding Additional Packages, I assume that both of nv-tensorrt-repo-ubuntu2404-cuda12.8-trt*-d7l-cross-ga-*_amd64.deb and nv-tensorrt-repo-ubuntu2404-cuda12.8-trt*-d7l-target-ga-*_arm64.deb TensorRT packages are necessary according this page. If the use case is engine building and inference directly on Drive Thor, not using the cross-compiler, is it correct that only nv-tensorrt-repo-ubuntu2404-cuda12.8-trt*-d7l-target-ga-*_arm64.deb is needed? If there are any other required packages, please let me know.

Regarding the installation procedure, is my understanding correct that I need to store the package in /drive_flashing and then just follow the “Binding and Flashing the Target Manually” steps again? Or other methods?

Also, where can I find nv-tensorrt-repo-ubuntu2404-cuda12.8-trt*-d7l-target-ga-*_arm64.deb etc on NGC? I searched, but couldn’t find it.

Thank you in advance.
Error String

Logs
Provide logs in text box instead of image.
Please paste the complete application log here. If there are multiple logs, please use multiple text box.

5

Dear @wencheng-lin

Could you check at NVIDIA NGC ?

The arm64.deb files are installers for TensorRT. TRT samples are available on docker at /usr/src/tensort/. You can copy /usr/src/tensort/ folder to target’s home folder. Copy the missing /usr/include/aarch64-linux-gnu/Nv*.h header files from docker to target at /usr/include/aarch64-linux-gnu/

Perform below commads on target to compile the TensorRT sample

nvidia@tegra-ubuntu:~/tensorrt$ cd /lib/aarch64-linux-gnu/
nvidia@tegra-ubuntu:/lib/aarch64-linux-gnu$ sudo ln -s libnvonnxparser.so.10 libnvonnxparser.so
nvidia@tegra-ubuntu:/lib/aarch64-linux-gnu$ sudo ln -s libnvinfer.so.10 libnvinfer.so
nvidia@tegra-ubuntu:/lib/aarch64-linux-gnu$ sudo ln -s libnvinfer_plugin.so.10 libnvinfer_plugin.so
nvidia@tegra-ubuntu:/lib/aarch64-linux-gnu$ cd ~/tensorrt/samples
nvidia@tegra-ubuntu:~/tensorrt/samples$ make -j4
6

Dear @SivaRamaKrishnaNV
Thank you very much for your prompt reply.

I can find these packages at NVIDIA NGC.

Thank you for your instructions. I was able to compile the TensorRT sample /usr/src/tensort/ successfully.
However, how can I use the TensorRT Python API on DRIVE Thor?
I would like to convert my own ONNX model to trt engine using Python script with import tensorrt as trt.

7

Dear @wencheng-lin ,

Check installing nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423_1-1_arm64.deb on target

nvidia@tegra-ubuntu:~/siva$ sudo apt-get update
Get:1 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  InRelease
Ign:1 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  InRelease
Get:2 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  Release [569 B]
Get:2 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  Release [569 B]
Get:3 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  Release.gpg [801 B]
Get:3 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  Release.gpg [801 B]
Ign:3 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  Release.gpg
Get:4 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  Packages [5665 B]
Get:5 http://ports.ubuntu.com/ubuntu-ports noble InRelease [256 kB]
Ign:5 http://ports.ubuntu.com/ubuntu-ports noble InRelease
Get:6 http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease [126 kB]
Ign:6 http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease
Get:7 http://ports.ubuntu.com/ubuntu-ports noble-security InRelease [126 kB]
Ign:7 http://ports.ubuntu.com/ubuntu-ports noble-security InRelease
Reading package lists... Done
W: GPG error: file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  Release: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/_var_nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423_Release
W: The repository 'file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  Release' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: http://ports.ubuntu.com/ubuntu-ports noble InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/ports.ubuntu.com_ubuntu-ports_dists_noble_InRelease
W: The repository 'http://ports.ubuntu.com/ubuntu-ports noble InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/ports.ubuntu.com_ubuntu-ports_dists_noble-updates_InRelease
W: The repository 'http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/noble-updates/InRelease is not valid yet (invalid for another 1d 19h 5min 9s). Updates for this repository will not be applied.
W: GPG error: http://ports.ubuntu.com/ubuntu-ports noble-security InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/ports.ubuntu.com_ubuntu-ports_dists_noble-security_InRelease
W: The repository 'http://ports.ubuntu.com/ubuntu-ports noble-security InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/noble-security/InRelease is not valid yet (invalid for another 1d 19h 3min 44s). Updates for this repository will not be applied.
nvidia@tegra-ubuntu:~/siva$ sudo apt-get install python3-libnvinfer
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  python3-libnvinfer
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/755 kB of archives.
After this operation, 5403 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  python3-libnvinfer
Install these packages without verification? [y/N] y
Get:1 file:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423  python3-libnvinfer 10.10.10.3-1+cuda12.8 [755 kB]
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package python3-libnvinfer.
(Reading database ... 74647 files and directories currently installed.)
Preparing to unpack .../python3-libnvinfer_10.10.10.3-1+cuda12.8_arm64.deb ...
Unpacking python3-libnvinfer (10.10.10.3-1+cuda12.8) ...
Setting up python3-libnvinfer (10.10.10.3-1+cuda12.8) ...
nvidia@tegra-ubuntu:~/siva$ sudo dpkg -l | grep TensorRT
ii  libnvinfer-dispatch10                                                     10.10.10.3-1+cuda12.8                    arm64        TensorRT dispatch runtime library
ii  libnvinfer-lean10                                                         10.10.10.3-1+cuda12.8                    arm64        TensorRT lean runtime library
ii  libnvinfer-plugin10                                                       10.10.10.3-1+cuda12.8                    arm64        TensorRT plugin libraries
ii  libnvinfer-vc-plugin10                                                    10.10.10.3-1+cuda12.8                    arm64        TensorRT vc-plugin library
ii  libnvinfer10                                                              10.10.10.3-1+cuda12.8                    arm64        TensorRT runtime libraries
ii  libnvonnxparsers10                                                        10.10.10.3-1+cuda12.8                    arm64        TensorRT ONNX libraries
ii  python3-libnvinfer                                                        10.10.10.3-1+cuda12.8                    arm64        Python 3 bindings for TensorRT standard runtime
ii  tensorrt-libs                                                             10.10.10.3-1+cuda12.8                    arm64        Meta package for TensorRT runtime libraries
nvidia@tegra-ubuntu:~/siva$ python
Python 3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorrt as trt
>>>
8

@SivaRamaKrishnaNV Thank you.
tensorrt_check.log (5.4 KB)

I am able to import tensorrt at home directory after installing nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423_1-1_arm64.deb on target device.
However, I am unable to run sudo apt-get install python3-libnvinfer.
I would like share the terminal log, which is attached as the [tenosor_check.log] file.

9

Could you check the below contents after installing nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423_1-1_arm64.deb . All the debians are available.

nvidia@tegra-ubuntu:/var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423$ ls
Packages.gz
Release
Release.gpg
c0d91dfc.pub
libnvinfer-bin_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-dispatch-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-dispatch10_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-headers-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-headers-plugin-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-headers-python-plugin-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-lean-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-lean10_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-plugin-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-plugin10_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-samples_10.10.10.3-1+cuda12.8_all.deb
libnvinfer-vc-plugin-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer-vc-plugin10_10.10.10.3-1+cuda12.8_arm64.deb
libnvinfer10_10.10.10.3-1+cuda12.8_arm64.deb
libnvonnxparsers-dev_10.10.10.3-1+cuda12.8_arm64.deb
libnvonnxparsers10_10.10.10.3-1+cuda12.8_arm64.deb
python3-libnvinfer-dev_10.10.10.3-1+cuda12.8_arm64.deb
python3-libnvinfer-dispatch_10.10.10.3-1+cuda12.8_arm64.deb
python3-libnvinfer-lean_10.10.10.3-1+cuda12.8_arm64.deb
python3-libnvinfer_10.10.10.3-1+cuda12.8_arm64.deb
tensorrt-dev_10.10.10.3-1+cuda12.8_arm64.deb
tensorrt-libs_10.10.10.3-1+cuda12.8_arm64.deb
tensorrt_10.10.10.3-1+cuda12.8_arm64.deb
```
10

@SivaRamaKrishnaNV
There are no .deb files in /var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423 on DRIVE Thor device.

I followed the three steps below to install nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423_1-1_arm64.deb. Is there any problem with my procedure? Please correct it if necessary.

Step1:

cd $NV_WORKSPACE/driveinstaller
./driveinstaller --installtype install --pkgpath /drive_flashing/

I confirm that all of debians are generated in /var/nv-tensorrt-repo-ubuntu2404-cuda12.8-trt10.10.10.3-d7l-target-ga-20250423 in container of host PC.

Step2:

cd $NV_WORKSPACE/drive-foundation
./make/bind_partitions -b p3960-10-sw-03 drive_av.linux -p dev_nsr

Step3:

cd $NV_WORKSPACE/drive-foundation/tools/flashtools/bootburn/
./bootburn.py -b p3960-10-sw03 -x /dev/ttyACM2
11

Step 2 and 3 will just reflash the target. You need to install the debian on target.

12

Dear @SivaRamaKrishnaNV

I initially thought that, according to the order described in the manual, in Step 2 and Step 3, the items built in the container of host pc would be updated to target. It turns out that they are installed directly on target as you mention. I have confirmed that I can use TensorRT Python API on my side. Thank you very much.