Tensorflow_gpu R - could not create cudnn handle

Hi all,

I know this problem has arisen on here but nobody seems to discuss this problem with regards to R; all I see are python issues and solutions.

I am running the GPU version of tensorflow as a backend to the Keras library in Rstudio to use for deep learning - specifically convnets.

Software / hardware:

This is my nvidia-smi output:

nvidia-smi
Tue Jul 09 11:05:43 2019
±----------------------------------------------------------------------------+
| NVIDIA-SMI 425.25 Driver Version: 425.25 CUDA Version: 10.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 2060 WDDM | 00000000:01:00.0 Off | N/A |
| N/A 41C P8 4W / N/A | 5256MiB / 6144MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 13156 C+G …eam\bin\cef\cef.win7\steamwebhelper.exe N/A |
| 0 16016 C …tensorflow_gpu\Library\bin\rsession.exe N/A |
±----------------------------------------------------------------------------+

Hardware:
GPU - NVidia RTX 2060
CPU - Intel I7 8750H
RAM - 16GB

Software:
OS - Windows 10 Home
GPU Drivers - 425.25
Cuda - 10.1
Cudnn - 7.6
TF - 1.13.1 (GPU version)
Keras - 2.2.4
Python - 3.6.8
Rstudio - 1.1.456
Base R - 3.6.1

When I attempt to execute the model I get the following error message:

2019-07-09 10:56:49.551567: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED

The full traceback is:

2019-07-09 10:56:45.085345: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-07-09 10:56:45.328872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.89GiB
2019-07-09 10:56:45.329507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-09 10:56:46.571437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-09 10:56:46.571725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-07-09 10:56:46.571847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-07-09 10:56:46.573101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4620 MB memory) → physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-07-09 10:56:49.550878: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2019-07-09 10:56:49.551567: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
Show Traceback

Rerun with Debug
Error in py_call_impl(callable, dots$args, dots$keywords) :
UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv2d_9/Conv2D}}]]
[[{{node ConstantFoldingCtrl/loss/activation_5_loss/broadcast_weights/assert_broadcastable/AssertGuard/Switch_0}}]]

I have also tried doing the same using Ubuntu 18.04 and get the following error:

2019-07-09 11:22:49.942666: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use:
SSE4.1 SSE4.2 AVX AVX2 FMA
2019-07-09 11:22:49.966646: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2208000000 Hz
2019-07-09 11:22:49.967415:
I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5611f27510e0 executing computations on platform Host. Devices:
2019-07-09 11:22:49.967431:
I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-07-09 11:22:50.051695:
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node,
so returning NUMA node zero
2019-07-09 11:22:50.051950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name:
GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
totalMemory: 5.79GiB freeMemory: 5.43GiB
2019-07-09 11:22:50.051963:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-09 11:22:50.052384:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-09 11:22:50.052392:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-07-09 11:22:50.052396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:
N
2019-07-09 11:22:50.052445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5259 MB memory)
→ physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-07-09 11:22:50.053557: I tensorflow/compiler/xla/service/service.cc:150]
XLA service 0x5611f28776a0 executing computations on platform CUDA. Devices:
2019-07-09 11:22:50.053569: I tensorflow/compiler/xla/service/service.cc:158]
StreamExecutor device (0): GeForce RTX 2060, Compute Capability 7.5
2019-07-09 11:22:54.658481: I tensorflow/stream_executor/dso_loader.cc:152]
successfully opened CUDA library libcublas.so.10.0 locally
2019-07-09 11:23:28.099635: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334]
Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-07-09 11:23:28.114124: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334]
Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Show Traceback

Rerun with Debug
Error in py_call_impl(callable, dots$args, dots$keywords) :

UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

 [[{{node conv2d/Conv2D}}]]
 [[{{node loss/mul}}]] 

Does anyone have any ideas for how to resolve this?
Thanks

do you have some solutions? i have the same question