[Fatal] Exception: CUDA error: 35

[ATAineri]

Good afternoon, dear Nvidia! I built my code on Jetson Nano. I installed each Cuda and nvidia-l4t-*32.7.4 package on my Arch Linux (on Jetson nano) and when I try to run this code I encounter:

[Fatal] Exception: CUDA error: 35, in file: /home/alarm/App/_src/Media/Encoder.cu, at line: 125

Then /home/alarm/App/_src/Media/Encoder.cu, at line: 125:
CUDA_CHECK(cudaStreamCreate(&_stream));

Why it can be and how can I surpass this? Or maybe workaround?

[AastaLLL]

Hi,

You can find the CUDA error in our document:

https://docs.nvidia.com/cuda/archive/10.2/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038

cudaErrorInsufficientDriver = 35
This indicates that the installed NVIDIA CUDA driver is older than the CUDA runtime library. This is not a supported configuration. Users should install an updated NVIDIA display driver to allow the application to run.

The error indicates the GPU driver and library are not compatible.
It looks like you use the CUDA package for r32.7.4.

Is your Nano also flashed with r32.7.4?
Thanks.

[ATAineri]

Yes, nano flashed with 32.7.4 and Cuda I installed from repo nvidia-l4t-cuda-32.7.4. Kernel I compiled myself from your git tagged 32.7.4 too. So the version of all components is equal.

[ATAineri]

Maybe the problem is that I compiled kernel that named 4.9.337-dirty, not 4.9.447-tegra? Or something like this. Because I know peoples that installed nvidia-l4t* packages over the arch and they had not encounter such a problem.
Packages I installed from:
https://repo.download.nvidia.com/jetson/
For T210.

[ATAineri]

Maybe you could tell me how or where can I see what is wrong? I want to find two numbers - what version do I need and what version do I have. Then I see this into the /proc or /sys echo them I understood what is wrong, like I need this number to be 53, but it is 0 - something like that. Unfortunately error message lack full info and I cannot understand what the problem…

[ATAineri]

I tried to use sample-rootfs 32.7.4 and encounter the same thing:
[Fatal] Exception: CUDA error: 35, in file: /home/lecor/Build/src/App/_src/Media/Encoder.cu, at line: 125
So I even more confident that it is problem because of I compiled kernel myself and maybe because it has signature 4.4.337-dirty (but if so it will very strange)…

[ATAineri]

Moreover, I built fresh clean 32.7.4 kernel and used fresh clean 32.7.4 sample_rootfs. Downloaded my app, run it and encounter Cuda 35 error.
Now I don’t know how to debug this, I have no clue what to do and why is it happening.

[ATAineri]

A lot of nvidia drivers making as modules, maybe I need to install some specific modules by my hands if it is non installing automatic for some reason?

[AastaLLL]

Hi,

The compatibility is guaranteed with the same JetPack version.
Could you use the OS and CUDA toolkit from the same JetPack version to see if your app can work fine?
(ex. JetPack 4.6.4?)

Thanks.

[ATAineri]

I used cuda and (everything else) from there:
https://repo.download.nvidia.com/jetson/#Jetpack%204.6.x
Purple is that I downloaded and installed in my rootfs:

[AastaLLL]

Hi,

If the OS and package are all from JetPack 4.6.4, could you run a CUDA sample to see if it can work correctly?

For example:

$ cd /usr/local/cuda/samples/0_Simple/vectorAdd
$ sudo make
$ ./vectorAdd

Thanks.

[ATAineri]

Sorry but I cannot download Jetpack 4.6.4. I am using Manjaro as a host system and SDK manager is not working on it because of OS version mismatching:

And I cannot download it through the link because it said that it is possible to download Jetpack 4.6.4 only through the SDK manager and there is no link to it:

Please, could you give me a straight refer to the Jetpack 4.6.4?

[ATAineri]

Please, tell me, how can I see what do the cuda want? There is no nvidia-smi for jetson so I don’t understand how can I see what is wrong. The only message that cuda programs says that Cuda 35 and all. How can I understand what driver do I need, maybe I need some LKM or more options to compile with…
If I only know what Cuda applications searching in rootfs I can fix it, but it don’t tell me that. How can I understand what it needs? Where can I find such information?

[AastaLLL]

Hi,

Sorry for the late update.

Does the SDcard method work for you?
You can find r32.7.4 OS from the link below and run the following command to install components.

JetPack SDK 4.6.4

$ sudo apt update
$ sudo apt install nvidia-jetpack

Thanks.

[ATAineri]

I’m using Arch on target and Manjaro on host, so it will not working)
I solved it - Cuda is not working if I pointing on it with LD_LIBRARY_PATH, only if it was pronounced in ldconfig - it was problem. After pronouncing path to cuda to ldconfig everything worked correctly.