CUDA driver version is insufficient for CUDA runtime version

Hello everyone.

We develop a project on Jetson Xavier AGX. We use cross-compiling to compile our project. Our project uses:

  • OpenCV 3.4.5
  • Caffe
  • MxNet

These all are on the host machine.

  • CUDA 10.2.89
  • CuDNN 8.0.0

These all are on the Xavier and installed with SDK Manager.
Jetpack 4.4 is used.
We linked CUDA and CuDNN paths with using sshfs from Xavier to Host.
We linked all library paths with using sshfs from Host to Xavier.

We compile the program on Host and we execute it on Xavier and we get this error :

E0806 09:08:49.059986  8450 common.cpp:114] Cannot create Cublas handle. Cublas won't be available.
E0806 09:08:49.096447  8450 common.cpp:121] Cannot create Curand generator. Curand won't be available.
F0806 09:08:49.121588  8450 common.cpp:152] Check failed: error == cudaSuccess (35 vs. 0)  CUDA driver version is insufficient for CUDA runtime version

There are some topics about this but they aren’t for Xavier AGX. Some people said that update nvidia drivers and people said that the problem was solved. However, there is no situation like that in Xavier.

What should we do?

Best regards.

A new Jetpack was just released.

You might want to make sure you have the latest version of Jetpack on your building host, and then make sure you re-flash/update the target with the corresponding latest OS image.

If you build with the libraries in the jetpack on the host, and run the corresponding OS image on the target, the version numbers should match. If you end up using some version of software that’s “available” on the host, but not actually part of the Jetpack, then it likely won’t work on the target.

The best way to make sure that your software will work on the target, is to actually build on the target, btw. The AGX Xavier allows you to install a NVME drive, and it has enough RAM and CPU that building on it is acceptable performance. You could, for example, open up the target with VSCode remote over SSH, to have a desktop-hosted IDE yet run the build on the target.

Thanks for your reply and information.

We solved the problem. The problem was that CUDA and CuDNN were newer version but the OS image was 32.3.x which has older Nvidia Drivers according to these versions of CUDA and CuDNN. We reflashed the target with 32.5.1 OS image and then installed CUDA and CuDNN. Problem was solved.