Compactibility of GPU driver and CUda

Description

DNN library is not found.
[[{{node model/conv1_conv/Conv2D}}]] [Op:__inference_predict_function_14645] is the error that I have encountered

Environment

TensorRT Version:
GPU Type: NVIDIA RTX A4000
Nvidia Driver Version: 545.23.08.
CUDA Version: `12.3
CUDNN Version: 8.9.6
Operating System + Version: Ubuntu 22.04.3 LTS
Python Version (if applicable): Python 3.9.18
TensorFlow Version (if applicable): 2.15.0
PyTorch Version (if applicable):

Please include:
2024-04-09 10:24:15.344526: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:447] Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.9.4. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2024-04-09 10:24:15.345197: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at conv_ops_impl.h:1199 : UNIMPLEMENTED: DNN library is not found.
Traceback (most recent call last):

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/spyder_kernels/py3compat.py”, line 356, in compat_exec
exec(code, globals, locals)

File “/home/gcekcse/Downloads/oct_flora.py”, line 429, in
x_train_features, y_train_binary, y_train_multiclass, x_val_features, y_val_binary, y_val_multiclass= load_and_preprocess_dataset(datagen_train, datagen_val, num_clusters, layer_name, max_iterations, num_flowers, model,num_multiclass_classes,images_per_class)

File “/home/gcekcse/Downloads/oct_flora.py”, line 256, in load_and_preprocess_dataset
x_train_features = feature_extraction(model, x_train_segmented, layer_name)

File “/home/gcekcse/Downloads/oct_flora.py”, line 190, in feature_extraction
extracted_features, _ = model.predict(img_array)

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py”, line 70, in error_handler
raise e.with_traceback(filtered_tb) from None

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/tensorflow/python/eager/execute.py”, line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,

UnimplementedError: Graph execution error:

Detected at node model/conv1_conv/Conv2D defined at (most recent call last):
File “/home/gcekcse/anaconda3/lib/python3.9/runpy.py”, line 197, in _run_module_as_main

File “/home/gcekcse/anaconda3/lib/python3.9/runpy.py”, line 87, in _run_code

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/spyder_kernels/console/main.py”, line 24, in

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/spyder_kernels/console/start.py”, line 330, in main

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/ipykernel/kernelapp.py”, line 736, in start

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/tornado/platform/asyncio.py”, line 195, in start

File “/home/gcekcse/anaconda3/lib/python3.9/asyncio/base_events.py”, line 601, in run_forever

File “/home/gcekcse/anaconda3/lib/python3.9/asyncio/base_events.py”, line 1905, in _run_once

File “/home/gcekcse/anaconda3/lib/python3.9/asyncio/events.py”, line 80, in _run

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/ipykernel/kernelbase.py”, line 516, in dispatch_queue

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/ipykernel/kernelbase.py”, line 505, in process_one

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/ipykernel/kernelbase.py”, line 412, in dispatch_shell

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/ipykernel/kernelbase.py”, line 740, in execute_request

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/ipykernel/ipkernel.py”, line 429, in do_execute

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/ipykernel/zmqshell.py”, line 546, in run_cell

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py”, line 2914, in run_cell

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py”, line 2960, in _run_cell

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/IPython/core/async_helpers.py”, line 78, in _pseudo_sync_runner

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py”, line 3185, in run_cell_async

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py”, line 3377, in run_ast_nodes

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py”, line 3457, in run_code

File “/tmp/ipykernel_17133/4149869158.py”, line 1, in

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/spyder_kernels/customize/spydercustomize.py”, line 528, in runfile

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/spyder_kernels/customize/spydercustomize.py”, line 615, in _exec_file

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/spyder_kernels/customize/spydercustomize.py”, line 473, in exec_code

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/spyder_kernels/py3compat.py”, line 356, in compat_exec

File “/home/gcekcse/Downloads/oct_flora.py”, line 429, in

File “/home/gcekcse/Downloads/oct_flora.py”, line 256, in load_and_preprocess_dataset

File “/home/gcekcse/Downloads/oct_flora.py”, line 190, in feature_extraction

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py”, line 65, in error_handler

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/training.py”, line 2655, in predict

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/training.py”, line 2440, in predict_function

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/training.py”, line 2425, in step_function

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/training.py”, line 2413, in run_step

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/training.py”, line 2381, in predict_step

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py”, line 65, in error_handler

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/training.py”, line 590, in call

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py”, line 65, in error_handler

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/base_layer.py”, line 1149, in call

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py”, line 96, in error_handler

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/functional.py”, line 515, in call

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/functional.py”, line 672, in _run_internal_graph

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py”, line 65, in error_handler

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/engine/base_layer.py”, line 1149, in call

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py”, line 96, in error_handler

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/layers/convolutional/base_conv.py”, line 290, in call

File “/home/gcekcse/anaconda3/lib/python3.9/site-packages/keras/src/layers/convolutional/base_conv.py”, line 262, in convolution_op

DNN library is not found.
[[{{node model/conv1_conv/Conv2D}}]] [Op:__inference_predict_function_14645]

This is the error encountered. Why such errors are coming??

That seems it cannot find the cudnn implementation. Basically, we have a generic dnn class, and we have cuda_dnn or rocm_dnn implementations. The error message seems to be from the generic dnn.

Can you please for the cudnn implementation