"ERROR: Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED" inside nvidia tensorflow docker

Hi,

I confront error when trying to run TensorRT convert in Nvidia TensorFlow docker container 20.02-tf2-py3.

Error Message:

2020-03-04 08:53:15.344585: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:736] Building a new TensorRT engine for TRTEngineOp_1 with input shapes: [[1,13,13,3,2], [1,13,13,3,2], [1,13,13,3,1], [1,13,13,3,80], [1,13,13,3,2]]
2020-03-04 08:53:15.344591: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:736] Building a new TensorRT engine for TRTEngineOp_3 with input shapes: [[1,26,26,3,2], [1,26,26,3,2], [1,26,26,3,1], [1,26,26,3,80], [1,26,26,3,2]]
2020-03-04 08:53:15.344587: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:736] Building a new TensorRT engine for TRTEngineOp_2 with input shapes: [[1,52,52,3,2], [1,52,52,3,2], [1,52,52,3,1], [1,52,52,3,80], [1,52,52,3,2]]
2020-03-04 08:53:15.348113: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2020-03-04 08:53:15.348508: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2020-03-04 08:53:15.348871: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2020-03-04 08:53:15.351238: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2020-03-04 08:53:15.351243: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2020-03-04 08:53:15.351246: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
../rtSafe/safeContext.cpp (110) - cuBLAS Error in initializeCommonContext: 3 (Could not initialize cublas, please check cuda installation.)
2020-03-04 08:53:15.351263: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:751] Engine creation for TRTEngineOp_2 failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
2020-03-04 08:53:15.351263: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:751] Engine creation for TRTEngineOp_3 failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
2020-03-04 08:53:15.351275: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:751] Engine creation for TRTEngineOp_1 failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
2020-03-04 08:53:15.487527: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:736] Building a new TensorRT engine for TRTEngineOp_0 with input shapes: [[5,416,416,3]]
2020-03-04 08:53:16.092505: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2020-03-04 08:53:16.092716: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 1 (Could not initialize cudnn, please check cudnn installation.)
2020-03-04 08:53:16.093253: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:751] Engine creation for TRTEngineOp_0 failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
2020-03-04 08:53:16.333042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-04 08:53:16.333630: E tensorflow/stream_executor/cuda/cuda_dnn.cc:330] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2020-03-04 08:53:16.333660: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Possibly insufficient driver version: 440.64.0
2020-03-04 08:53:16.334278: E tensorflow/stream_executor/cuda/cuda_dnn.cc:330] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2020-03-04 08:53:16.334295: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Possibly insufficient driver version: 440.64.0
2020-03-04 08:53:16.334922: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_op.cc:318 : Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node StatefulPartitionedCall/model/conv2d/Conv2D}}]]
2020-03-04 08:53:16.334935: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node StatefulPartitionedCall/model/conv2d/Conv2D}}]]
	 [[TRTEngineOp_0]]

Host Driver Version: 440.64.0
Host OS: Ubuntu 16.04
Nvidia TensorFlow docker version: 20.02
GPU: Geforce 2080

On host system, there exists CUDA 10.0, I am not sure whether it relates to the problem or not.

Did you solve this error? I have the same problem

I had the exact problem. Cudnn is complaining about insufficient driver.

Host OS: Ubuntu 20.04
Host Driver Version: 440.82+really.440.64-0ubuntu6
CUDA: 10.2

It could that you have the latest cudnn 7.6.5 installed. I have success with cudnn 7.5.4. It works flawlessly.