The CMake project reports an error: CUDA driver version is insufficient for CUDA runtime version

Accelerated Computing CUDA CUDA Setup and Installation
1

My device configuration: Ubuntu20.04 && Gefore RTX 3060;
My CUDA Driver version: NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0
My CUDA runtime version: Cuda compilation tools, **release 11.8, V11.8.89 **
Build cuda_11.8.r11.8/compiler.31833905_0

**My CMake Project: **

CUDA Dir’s CMakeLists.txt

set(cuda_include_dir /usr/local/cuda/include)
set(cuda_lib_dir /usr/local/cuda/lib64)
#
include_directories(${cuda_include_dir})
link_directories(${cuda_lib_dir})
#
add_cuda_headers_and_sources (clickhouse_aggregate_functions_cuda .)
add_library (clickhouse_aggregate_functions_cuda SHARED ${clickhouse_aggregate_functions_cuda_sources})    # ${clickhouse_aggregate_functions_cuda_headers}
#
set_target_properties(clickhouse_aggregate_functions_cuda PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
set_target_properties(clickhouse_aggregate_functions_cuda PROPERTIES POSITION_INDEPENDENT_CODE ON)
target_compile_features(clickhouse_aggregate_functions_cuda PUBLIC cuda_std_17)

target_include_directories(clickhouse_aggregate_functions_cuda PUBLIC ${cuda_include_dir})

target_link_libraries(clickhouse_aggregate_functions_cuda PUBLIC ${cuda_lib_dir}/libcudart.so)

Outermost CMakeLists.txt

set(CMAKE_CUDA_ARCHITECTURES 86)
set(CMAKE_CUDA_COMPILER /usr/local/cuda/bin/nvcc)
project(ClickHouse LANGUAGES C CXX ASM CUDA)

set(CMAKE_CUDA_STANDARD 17)
set(CMAKE_CUDA_STANDARD_REQUIRED ON)

But, When I call the CUDA runtime API in the .cu/.cuh/.h file, I get an error “CUDA driver version is insufficient for CUDA runtime version”. Why? Please!

2

Always, always, always, this error means you should update your GPU driver to the latest for your GPU.

The latest driver for RTX 3060 is newer than the 525.89.02 driver that you have installed. You can get a 530.xx driver here.

I won’t be able to tell you why, since you’ve not given a complete description of what you are doing. There is some aspect of your toolchain that is creating binaries that are “newer” than what is supported by your 525.89.02 driver. (Since CUDA 12.1 is out, and so are 530.xx drivers, presumably you have something that expects that.)

1 Like
3

Thanks for your answer, I have updated the driver and runtime version to the latest but still face the same error.

Below, I’ll describe what I’m doing:
I’m trying to link CUDA code as a dynamic library into a CMake/C++ project (ClickHouse source).

The project structure is as follows:
ClickHouse
├── …
├── src
├── CMakeLists.txt
├── parallelization(CUDA Dir)
├── CMakeLists.txt
├── cudaTest.cu
└── cudaTest.cuh
└── CMakeLists.txt(Outermost)

Both CMakeLists.txt contexts are same as my first question.
I created a new directory (parallelization) under the source src directory to place the CUDA code, but any CUDA runtime API call for any .cu/.cuh/.h file under src will report the error ‘CUDA driver version is insufficient for CUDA runtime version’.
After updating the driver and runtime, the result of cudaGetDriver/RuntimeVersion is as follows:
CUDA Driver Version / Runtime Version 0.0 / 12.1.

Also: I customized a simple project like ClickHouse and ran the CUDA API described above. In the same software and hardware environment, it can run correctly without any error.

My System Envs: Ubuntu20.04、GeForce RTX 3060
My Toolchain: cmake3.24.2（clion bundle）、LLVM_15.x clang&&clang++
My Environment Variable: export PATH=$PATH:/usr/local/cuda-12.1/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.1/lib64
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda-12.1/lib64
nvidia-smi
Tue Mar 21 11:37:02 2023
±--------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02 Driver Version: 530.30.02 CUDA Version: 12.1 |
|-----------------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3060 L… Off| 00000000:01:00.0 Off | N/A |
| N/A 40C P0 N/A / 115W| 6MiB / 6144MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+

±--------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1121 G /usr/lib/xorg/Xorg 4MiB |
±--------------------------------------------------------------------------------------+

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

If it was an error caused by ClickHouse’s complex CMake project, could you provide a little idea of what might you be able to do? Or is it a problem caused by Clang15? Or what caused the error?

If you need more information, you can let me know.Thank you for your patience and guidance!

4

So that driver install is broken, or else you have a problem with the actual libcudart or libcuda on your machine, or else the clang calls into libcuda or libcudart are messed up.

I don’t work with clang very much. I suggest validating your cuda install using the methodology described in the linux install guide, not using clion, not using cmake. If that doesn’t report the same error, then there is a problem with your other toolchain components. If that does report the same error, then there is a problem with your CUDA install.

You don’t mention what clang version you are using and the only supported version for the latest CUDA toolkit (12.1) is 15.0. I don’t know what 15.x means.

And if we go back to CUDA 11.8, the only supported clang version is 14.0.

So I think you need to scrub your install carefully.

5

Thank you very much! With your prompt, I solved the error for the following reasons:

ClickHouse prohibits the use of dynamically loaded library methods defined in <dlfcn.h> header file, so it defines functions with the same signature in the main.cpp to prevent this behavior.

It may be that custom dlopen, dlsym and other functions are incorrectly called, causing cuda to report an error.

So, it’s not CUDA’s problem. You don’t need to reply to this message, I just want to give a hint to those who came after me. Thank you again for your patient guidance!

1 Like