TensorFlow loads the wrong cuDNN version

[andrew.riachi]

I installed CUDA 12.2 and cuDNN 8.9.5 from nVidia’s website on Ubuntu 22.04 under WSL using the Ubuntu repositories. I then built TensorFlow 2.14 from source under this environment (using nvcc rather than the default clang). Now when I try to use TensorFlow, I get the following error:

2023-10-16 14:24:54.912883: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:435] Loaded runtime CuDNN library: 8.7.0 but source was compiled with: 8.9.5. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.

I have no idea how it is finding version 8.7.0 at runtime. I checked the files /usr/include/cudnn_version.h and /usr/include/x86_64-linux-gnu/cudnn_version_v8.h, and both say the version is 8.9.5. Does anyone know what might be going on?

[andrew.riachi]

Solved my own issue. cuDNN and other libraries had been previously installed by pip, probably from when I installed the TensorFlow prebuilt binaries before building it myself:

$ pip list | grep nvidia
nvidia-cublas-cu11           11.11.3.6
nvidia-cuda-cupti-cu11       11.8.87
nvidia-cuda-nvcc-cu11        11.8.89
nvidia-cuda-runtime-cu11     11.8.89
nvidia-cudnn-cu11            8.7.0.84
nvidia-cufft-cu11            10.9.0.58
nvidia-curand-cu11           10.3.0.86
nvidia-cusolver-cu11         11.4.1.48
nvidia-cusparse-cu11         11.7.5.86
nvidia-nccl-cu11             2.16.5

I uninstalled these packages through pip and that solved the problem!