JAX program gives CUDNN_STATUS_ARCH_MISMATCH error

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

I am trying to run a simple JAX program and I get this error

2023-03-03 13:58:10.931419: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:429] Could not create cudnn handle: CUDNN_STATUS_ARCH_MISMATCH
2023-03-03 13:58:10.989189: E external/org_tensorflow/tensorflow/compiler/xla/status_macros.cc:57] INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:627) dnn != nullptr 
*** Begin stack trace ***

I have a Tesla K40c GPU card . I have installed CUDA Toolkit 11.4 . The CUDnn version from the cudnn_version.h file is 8800

The first few lines of the deviceQuery is as follows

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Tesla K40c"
  CUDA Driver Version / Runtime Version          11.4 / 11.4
  CUDA Capability Major/Minor version number:    3.5
  Total amount of global memory:                 11441 MBytes (11997020160 bytes)
  (015) Multiprocessors, (192) CUDA Cores/MP:    2880 CUDA Cores
  GPU Max Clock rate:                            745 MHz (0.75 GHz)
  Memory Clock rate:                             3004 Mhz

A search on Stack Overflow indicated that the CUDA toolkit and the CUDNN version might have mismatches . But I specifically selected the CUDNN version compatible "cuDNN 8.8.0 for CUDA 11.x "

Any pointers will be helpful

I am not able to understand this page’s compatibility matrix

1 Like
2

Hi @hvram1

This is correct as per the document,
However have you also verified the CUDA Compatibility matrix?

Thanks