Hi! We are having problems running a docker built on the image nvcr.io/nvidia/tensorflow:24.05-tf2-py3. Our docker runs a tensorflow based model trained on the same version as the one present in the image yet when it tries to infer no algorithm is found to solve certain layers.
The error is the following:
No algorithm worked! Error messages:
Profiling failure on CUDNN engine eng1{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): ‘status’
Profiling failure on CUDNN engine eng28{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): ‘status’
Profiling failure on CUDNN engine eng0{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): ‘status’
Profiling failure on CUDNN engine eng54{k2=1,k12=-1,k13=0,k14=3,k15=0,k17=3,k18=1,k23=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): ‘status’
Profiling failure on CUDNN engine eng41{k2=0,k12=-1,k13=2,k14=3,k15=0,k17=3,k18=1,k22=0,k23=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): ‘status’
Profiling failure on CUDNN engine eng53{k2=5,k13=1,k14=2,k18=0,k23=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): ‘status’
Profiling failure on CUDNN engine eng56{k2=6,k12=-1,k13=0,k14=1,k15=0,k17=3,k18=1,k22=0,k23=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): ‘status’
[[{{node model/conv1/Conv2D}}]] [Op:__inference_predict_function_2796]
The problem might be related to the NVIDIA drivers but we are not sure.
Any help or advise would be welcomed.