Is it possible to already use TensorRT 10 on Jestson AGX Orin

[lauchtdaniel]

I try to use some features of TensorRT 10. I use the SDK Manager to set it to Version 8.6 and then set a container with TensorRT version 10 up. I also updated the Cuda Version to the needed version 12.4.

But it seems that I need also the driver version of the grafic-card with version 550 and it looks like there is no grafic-card driver version 550 for Jetson Orin available or did I miss something.
Maybe someone can tell we where I can find the right driver.

[AastaLLL]

Hi,

Orin is an iGPU system so the driver comes with OS rather than a standalone driver.

If you want to use the library on Orin, the library needs to support the Jetson environment (with aarch64-jetson tag).
We do have CUDA 12.4 for Jetson but no TensorRT 10 is available currently.

CUDA Toolkit 12.4 Downloads

Thanks.

[ambrose.maker]

Hello,

I found there is TensorRT 10.0 GA for JetPack with CUDA12.4 downloadable in this link.

Log in

May I know is this work if upgraded CUDA toolkit to 12.4 in JetPack 6.0?

Thank you.

[AastaLLL]

Hi,

Please let us check more details internally and update.

Thanks.

[lauchtdaniel]

Thanks for the responses,
for me I got errors with CUDA 12.4 refering to the grafic card driver version.
Maybe becouse CUDA 12.4 needs grafic driver version 550.
It would be awesome to use TensorRT 10 on Jetson to speed up our interference time in production with quantisation of LLM-Layer and Transformer-Layer.

[AastaLLL]

Hi,

Do you use the CUDA in the link shared above?

CUDA Toolkit 12.4 Downloads

If yes, the package can work correctly after export the compat folder:

Cuda 12.1 driver for Linux aarch64 missing?

Hi, The GPU driver for Jetson is integrated into the OS and the latest version is 11.4. But we support CUDA compatibility from CUDA 12. So CUDA 12 can work with the older GPU driver version. Have you linked the compat folder which allows users to run on an older GPU driver? $ export LD_LIBRARY_PATH=/usr/local/cuda-12.0/compat Below is the document for your reference: Thanks.

Thanks.

[AastaLLL]

Hi,

We have confirmed that TensorRT 10.0 GA can work correctly on Orin+JetPack 6.0GA.
Here are the steps for your reference:

1. CUDA 12.4

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-ubuntu2204.pin
$ sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
$ wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda-tegra-repo-ubuntu2204-12-4-local_12.4.0-1_arm64.deb
$ sudo dpkg -i cuda-tegra-repo-ubuntu2204-12-4-local_12.4.0-1_arm64.deb
$ sudo cp /var/cuda-tegra-repo-ubuntu2204-12-4-local/cuda-*-keyring.gpg /usr/share/keyrings/
$ sudo apt-get update
$ sudo apt-get -y install cuda-toolkit-12-4 cuda-compat-12-4

Edit ~/.bashrc by adding below two lines:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/cuda-[ver]/compat:$LD_LIBRARY_PATH

$ source ~/.bashrc

2. TensorRT 10.0

$ sudo dpkg -i ./nv-tensorrt-local-repo-l4t-10.0.1-cuda-12.4_1.0-1_arm64.deb
$ sudo cp /var/nv-tensorrt-local-repo-l4t-10.0.1-cuda-12.4/nv-tensorrt-local-D8519DC5-keyring.gpg /usr/share/keyrings/
$ sudo apt update
$ sudo apt-get install tensorrt

Verify with below command:

$ dpkg-query -W tensorrt
tensorrt	10.0.1.6-1+cuda12.4

Thanks.

[lauchtdaniel]

Thanks you very much for the solution. It works for me very good and the networks running faster even without doing anything else.
Note: download can be found on:

Log in