TensorFlow loads the wrong cuDNN version

I installed CUDA 12.2 and cuDNN 8.9.5 from nVidia’s website on Ubuntu 22.04 under WSL using the Ubuntu repositories. I then built TensorFlow 2.14 from source under this environment (using nvcc rather than the default clang). Now when I try to use TensorFlow, I get the following error:

2023-10-16 14:24:54.912883: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:435] Loaded runtime CuDNN library: 8.7.0 but source was compiled with: 8.9.5. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.

I have no idea how it is finding version 8.7.0 at runtime. I checked the files /usr/include/cudnn_version.h and /usr/include/x86_64-linux-gnu/cudnn_version_v8.h, and both say the version is 8.9.5. Does anyone know what might be going on?

Solved my own issue. cuDNN and other libraries had been previously installed by pip, probably from when I installed the TensorFlow prebuilt binaries before building it myself:

$ pip list | grep nvidia
nvidia-cublas-cu11           11.11.3.6
nvidia-cuda-cupti-cu11       11.8.87
nvidia-cuda-nvcc-cu11        11.8.89
nvidia-cuda-runtime-cu11     11.8.89
nvidia-cudnn-cu11            8.7.0.84
nvidia-cufft-cu11            10.9.0.58
nvidia-curand-cu11           10.3.0.86
nvidia-cusolver-cu11         11.4.1.48
nvidia-cusparse-cu11         11.7.5.86
nvidia-nccl-cu11             2.16.5

I uninstalled these packages through pip and that solved the problem!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.