CUDA 12 : Insufficient driver version on AGX Orin

johnmansellis · February 10, 2023, 1:44am

Problem :

I am unable to get CUDA 12 running successfully on Jetson AGX Orin.
Device Query :

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

Steps to Reproduce

1. Fresh OS Flash from SDK Manager

2. Verify System Requirements

> lspci | grep -i nvidia
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 229e (rev a1)

> uname -m && cat /etc/*release
aarch64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
# R35 (release), REVISION: 2.1, GCID: 32413640, BOARD: t186ref, EABI: aarch64, DATE: Tue Jan 24 23:38:33 UTC 2023
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"

> gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

> sudo apt-get install linux-headers-$(uname -r)
E: Unable to locate package linux-headers-5.10.104-tegra
E: Couldn't find any package by glob 'linux-headers-5.10.104-tegra'
E: Couldn't find any package by regex 'linux-headers-5.10.104-tegra'

> sudo apt install nvidia-l4t-kernel-headers
nvidia-l4t-kernel-headers is already the newest version (5.10.104-tegra-35.2.1-20230124153320).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

3. Prepare Ubuntu

> sudo apt-key del 7fa2af80
OK
> wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/arm64/cuda-keyring_1.0-1_all.deb
> sudo dpkg -i cuda-keyring_1.0-1_all.deb

4. Update and Install

> sudo apt-get update
> sudo apt-get install cuda
> sudo reboot

5. Post Install Actions

# .bashrc File
# ------ CUDA -------------
export PATH=/usr/local/cuda-12/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-12/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

# --- Links each point to cuda-12.0

6. Build Samples

> cd cuda-samples
make
> cd Samples/1.../deviceQuery/
> ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

If I point my path and LD_LIBRARY_PATH variables to cuda 11.4 instead of 12.0, everything works fine. If either one, or both, is pointing to cuda 12 I get this error.

Cuda toolkit is installed for both 11.4 and 12.0. Jetpack is installed from apt as of today; version 5.1-b147.

AastaLLL · February 10, 2023, 2:30am

Hi,

Could you follow the whole steps mentioned on the below page?
(A cuda-tegra-xxx.deb needs to be installed)

Thanks.

johnmansellis · February 10, 2023, 7:33pm

I followed the instructions exactly. Everything installed without errors. Still getting the same result.

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

CUDA Versions

  int driver=0;
  int runtime=0;

  cudaRuntimeGetVersion(&runtime);
  cudaDriverGetVersion(&driver);

  printf("Cuda Runtime Version = %d\n", runtime);
  printf("Cuda Driver  Version = %d\n", driver);

Output

Cuda Runtime Version = 12000
Cuda Driver  Version = 11040

NVCC

> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:28_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0

CUDA Driver

> apt search cuda-driver
cuda-driver-dev-11-4/stable,now 11.4.298-1 arm64 [installed,automatic]
  CUDA Driver native dev stub library

cuda-driver-dev-11-8/unknown 11.8.89-1 arm64
  CUDA Driver native dev stub library

cuda-driver-dev-12-0/unknown,now 12.0.146-1 arm64 [installed,automatic]
  CUDA Driver native dev stub library

linuxdev · February 10, 2023, 9:03pm

I could be wrong, but I don’t think CUDA 12 is available on Jetsons. If the CUDA 12 you got is from most non-Jetson repositories (even if it is arm64), the integrated GPU (iGPU) of a Jetson is directly wired to the memory controller; the other non-Jetson repositories are intended for discrete PCI GPUs (dGPU).

johnmansellis · February 10, 2023, 9:34pm

@linuxdev, thank you for your feedback. That is something I’ve been wondering about and unsure of. It may just be my naivete but some of the documentation was unclear to me.

System Requirements

For example, the cuda toolkit 12.0 downlod page has a path specifically for aarch64-jetson → native → Ubuntu 20.04. But when I follow the instructions the drivers seem to be installed, but not enabled, or not being used.

On the cuda for tegra page, it lists that cuda 12.0 is compatible, but only through the upgrade package. I’m not sure if the upgrade package is the same as the cuda 12.0 download link, or a different package manager I need to enable and utilize.

The CUDA Installation Guide system requirements lists Arm64 Jetson (igpu) as supported, but it wasn’t clear to me if the “NO” boxes in the table mean that it’s not compatible, or that the system doesn’t come with a version that is compatible, or something else.

EG:

| Distribution          | Kernel1           | Default GCC | GLIBC | GCC2,3 | ICC3 | NVHPC3 | XLC3 | CLANG | Arm C/C++ |
|-----------------------|-------------------|-------------|-------|--------|------|--------|------|-------|-----------|
| L4T4 20.04.z (z <= 5) | 5.10.120-70–tegra | 9.4.0       | 2.31  | NO     | NO   | NO     | NO   | NO    | NO        |

Does this mean that I need to install all the “NO” packages? Or there is no supported version of those packages, so there’s no working version of CUDA 12 on Jetson ? (seems odd to release one that has no supported working path) Are there just certain features I won’t be able to get working?

CUDA Driver 11.4 vs 12.0

Also the cuda-driver-dev-12-0 package is installed, but if getCudaDriverVersion( ) returns 11040, does that imply that the system is using the cuda-driver-dev-11-4 package? Do I need to tell it which cuda driver to use?

Other than setting the PATH and LD_LIBRARY_PATH in my .bashrc file, is there another way I need to instruct the system to use the correct cuda-driver?

linuxdev · February 11, 2023, 7:37pm

This is why I am not certain if my answer is correct. You can check your L4T release with “head -n 1 /etc/nv_tegra_release” (this is what actually gets flashed). L4T releases from R32.x or earlier would all be CUDA 10.x. It isn’t until you get to R34.x when CUDA 11.x becomes available. Again, I could be wrong, but I think that the JetPack/SDK Manager installer software only installs CUDA 11.x. There are arm64 targets which have separate installation packages, but mostly those are not compatible with Jetsons.

Perhaps NVIDIA could answer this: At which L4T release is CUDA 12.x available? Or is this still a “future” release? I might be behind the times and just did not see CUDA 12 added.

AastaLLL · February 13, 2023, 5:50am

Hi,

We start to provide the Jetson package on the website from CUDA 12.
This allows users to use the latest CUDA software without waiting for the next JetPack release.

Some user has used CUDA 12 on JetPack 5.0.2 and it can work well.
However, it seems that there is an issue with JetPack 5.1.

We are going to reproduce this and check internally.
Will share more information with you later.

Thanks.

johnmansellis · February 13, 2023, 5:45pm

@AastaLLL , thank you for clarifying and looking into that. Over the weekend I was able to get it ~working if I set:

export PATH=/usr/local/cuda-12/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/cuda-12.0/compat

Can you explain what the difference is between using the compatibility library and the full cuda-12 library? With Jetpack 5.0.2 is CUDA 12 working in native mode? Or compatibility mode?

Thanks.
John

AastaLLL · February 14, 2023, 4:45am

Hi,

We can reproduce similar behavior.
Same on JetPack 5.0.2 and JetPack 5.1.

We are checking internally about the difference between following two packages.
Will share more info with you after we get a feedback:

/usr/local/cuda-12.0/compat/libcuda.so.1.1
/usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1

Thanks.

AastaLLL · February 22, 2023, 5:14am

Hi,

Sorry for the late reply.

The compat version libcuda.so is required when using new CUDA toolkit components on systems with older base installations.
Please check the below document for more details:

Thanks.

hillct_nvidia.com · February 27, 2023, 3:45pm

@ AastaLLL,
Thanks for the update. I look forward to a time when CUDA driver and runtime releases are better synced up, as this and other similar stack compatibility issues has taken up over half our development ime so far…

I look forward to further updates, improvements in the software release workflow and a final resolution of his issue

AastaLLL · March 1, 2023, 7:42am

Hi,

We release the package from the CUDA website to allow users to get the latest CUDA faster.

Since JetPack is a relatively complicated system with lots of compatibilities.
In general, it takes a longer time to update the library version.

Thanks.

FindHao · March 9, 2023, 8:25pm

Have the same issue here. CUDA 12.1 with tegra release R35.

system · March 23, 2023, 8:26pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to access CUDA repo from CUDA L4t docker container Jetson Xavier NX cuda	39	3541	November 13, 2022
torch.cuda.is_available()=FALSE and [INFO]: Driver not installed... but it IS installed!? Jetson Orin NX cuda , cudnn	6	66	November 7, 2024
CUDA insufficient driver version Jetson TX1	9	8196	October 18, 2021
CUDA driver version is insufficient for CUDA runtime version Jetson AGX Orin cuda , containers , jetson	4	449	July 3, 2024
cudaErrorInsufficientDriver: CUDA driver version is insufficient for CUDA runtime version in docker container Jetson AGX Orin cuda	10	630	June 26, 2024
CUDA is not installed on Jetson Orin Jetson AGX Orin cuda	8	16416	August 10, 2022
Simplifying CUDA Upgrades for NVIDIA Jetson Users Technical Blog	0	365	October 4, 2022
Where is the latest version of CUDA available that could be installed on the jetsons? Jetson Nano cuda	27	2041	May 9, 2023
How to specify which CUDA to use Jetson AGX Orin cudnn	4	1287	November 20, 2023
JetPack L4T 3.2 Release fails to install CUDA Toolkit on host machine. Jetson TX2	24	6849	June 29, 2018

CUDA 12 : Insufficient driver version on AGX Orin

Problem :

Steps to Reproduce

1. Fresh OS Flash from SDK Manager

2. Verify System Requirements

3. Prepare Ubuntu

4. Update and Install

5. Post Install Actions

6. Build Samples

CUDA Versions

Output

NVCC

CUDA Driver

System Requirements

CUDA Driver 11.4 vs 12.0

Related topics