Hi everyone. I’m trying to install CUDNN on Ubuntu 22.04 system with two Nvidia Tesla K80 GPUs totalling 24GB of GPU memory. I have installed:
- nvidia driver 470
- cuda toolkit 11.8
- cudnn 8.9.7.29 (libcudnn8, libcudnn8-dev, libcudnn8-samples)
CUDA seems to have been installed properly because I could successfully compile and run the following test:
cuda-samples/Samples/1_Utilities/deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 2 CUDA Capable device(s)
Device 0: "Tesla K80"
CUDA Driver Version / Runtime Version 11.4 / 11.8
CUDA Capability Major/Minor version number: 3.7
Total amount of global memory: 11441 MBytes (11997020160 bytes)
(013) Multiprocessors, (192) CUDA Cores/MP: 2496 CUDA Cores
GPU Max Clock rate: 562 MHz (0.56 GHz)
Memory Clock rate: 2505 Mhz
Memory Bus Width: 384-bit
L2 Cache Size: 1572864 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total shared memory per multiprocessor: 114688 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Enabled
Device supports Unified Addressing (UVA): Yes
Device supports Managed Memory: Yes
Device supports Compute Preemption: No
Supports Cooperative Kernel Launch: No
Supports MultiDevice Co-op Kernel Launch: No
Device PCI Domain ID / Bus ID / location ID: 0 / 5 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Device 1: "Tesla K80"
CUDA Driver Version / Runtime Version 11.4 / 11.8
CUDA Capability Major/Minor version number: 3.7
Total amount of global memory: 11441 MBytes (11997020160 bytes)
(013) Multiprocessors, (192) CUDA Cores/MP: 2496 CUDA Cores
GPU Max Clock rate: 562 MHz (0.56 GHz)
Memory Clock rate: 2505 Mhz
Memory Bus Width: 384-bit
L2 Cache Size: 1572864 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total shared memory per multiprocessor: 114688 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Enabled
Device supports Unified Addressing (UVA): Yes
Device supports Managed Memory: Yes
Device supports Compute Preemption: No
Supports Cooperative Kernel Launch: No
Supports MultiDevice Co-op Kernel Launch: No
Device PCI Domain ID / Bus ID / location ID: 0 / 6 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from Tesla K80 (GPU0) -> Tesla K80 (GPU1) : Yes
> Peer access from Tesla K80 (GPU1) -> Tesla K80 (GPU0) : Yes
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.4, CUDA Runtime Version = 11.8, NumDevs = 2
Result = PASS
However, the following CUDDN test fails:
cudnn_samples_v8/mnistCUDNN
Executing: mnistCUDNN
cudnnGetVersion() : 8907 , CUDNN_VERSION from cudnn.h : 8907 (8.9.7)
Host compiler version : GCC 11.4.0
There are 2 CUDA capable devices on your machine :
device 0 : sms 13 Capabilities 3.7, SmClock 562.0 Mhz, MemSize (Mb) 11441, MemClock 2505.0 Mhz, Ecc=1, boardGroupID=0
device 1 : sms 13 Capabilities 3.7, SmClock 562.0 Mhz, MemSize (Mb) 11441, MemClock 2505.0 Mhz, Ecc=1, boardGroupID=0
Using device 0
Testing single precision
ERROR: cudnn failure (CUDNN_STATUS_ARCH_MISMATCH) in mnistCUDNN.cpp:414
Aborting...
I suspect it has something to do with the relatively old driver? I did try installing newer drivers like version 535 or 550, but those drivers wouldn’t work. I got a message in the log saying my GPU only runs on the 470.x legacy driver. Still, according to the documentation, the 470 driver should work with CUDA 11.x. I also tried looking for CUDNN version documentation but I couldn’t find it.
I have tried installing CUDA toolkit 11.4 too, but Ubuntu 22 lacks the required dependency. Please help me solve this issue. Thank you very much.