CUDA driver version is insufficient for CUDA runtime version inside docker on Jetson TX2

Autonomous Machines Jetson & Embedded Systems Jetson TX2
,
#1

Hello everyone,

I have some difficulties running a Cuda application inside a docker container on my Jetson TX2. The error message is CUDA Error: CUDA driver version is insufficient for CUDA runtime version The Cuda version is 10.2 but I have no idea where I can find the driver version.

I think there is something simple I don’t understand but I can’t figure out what… ! I am a total beginner on Jetson on Cuda so I must be missing something.

Thank you !

#2

If you cd to “/usr/local”, and run “ls -d cuda*”, then one of them will show a version. That is your current version.

The CUDA version has a strong dependency to the installed L4T which is currently flashed. If you need a different version, then you need to flash. However, keep in mind that nvcc can compile for different compute architectures, e.g., “-gencode arch=” can support different GPUs. Make sure it is really the CUDA version you need to change, and not just a recompile for that GPU.

Here are links to a list of L4T releases (which is what JetPack/SDKM flashes):
https://developer.nvidia.com/embedded/linux-tegra-archive
(you’ll need to go there, log in, and then go there a second time)

Here are links to JetPack releases:
https://developer.nvidia.com/embedded/jetpack-archive

When you drill down to any particular release in that list it will mention which CUDA version it supports. If you simply recompiling for your GPU is not sufficient, then you will need to flash with the JetPack/SDKM release having the proper CUDA version.

#3

Thank you for your answer
I checked my Jetson l4t and cuda versions and they are older than my docker’s one. I am going to downgrade my docker l4t version and see if I still have the same error

#4

running my docker with runtime=nvidia causes the error :

ERROR: for ros-mdt-run Cannot start service ros-mdt-run: OCI runtime create failed: container_linux.go:349: starting container process caused “process_linux.go:449: container init caused “process_linux.go:432: running prestart hook 0 caused \“error running hook: exit status 1, stdout: , stderr: invalid CUDA version: 10-0\\n\”””: unknown

I really don’t understand where is the problem… I am running the r32.3.1 l4t version with CUDA 10, same version on my jetson and inside my docker.

#5

Someone using docker would have to answer this, but do beware that any docker running on a Jetson has to specifically be customized with that in mind. The normal publicly available releases of docker container content for a PC would not be able to work with a Jetson. I think NVIDIA has releases specific to the Jetson, but since I have never used it, I have no way of giving advice on that.

#7

Hi,

Do you fix this issue yet?
Usually, insufficient error comes from the different version between GPU driver and CUDA library.
Please noticed that you need to use the container that built with the same L4T version as your device.

For example,
If you use JetPack4.4 product release, which is rel32-4.3, please use the image that built on r32.4.3.

docker pull nvcr.io/nvidia/l4t-base:r32.4.3

Thanks.

#8

Hi,

I didn’t fix this issue, I checked the version installed on my jetson which is r32.3.1 and I used the same version in my docker image. But I still get the same error. I must have a version problem somewhere.

For now I don’t use docker on my Jetson. If I have some time I will try to make it work and write a feedback here.

Thank you for the answers !