CuDNN error while fitting CNN

lifewanter · May 22, 2018, 7:46am

My system parameters:
OS: Windows 10 64bit
TensorFlow GPU: 1.5.0 (installed with pip)
Python Version: 3.6.4
IDE : Spyder 3.2.6
CUDA: v9.0
CuDNN: v7.0.5 for CUDA 9.0
GPU: GeForce GTX 960M 4GB
NVIDIA drivers: 397.64

So, I was trying to run some CNN (and also CapsNet where the first layer is just a convolution layer). And it keeps crushing with the error below. From what I found on the internet I decided to reinstall cuda and cudnn. And the code with CNN worked. Though capsnet still didn’t. I googled more and found out that cuda reset your drivers after installation, so, I needed to update my drivers. I had something like 380 or 384. Don’t remember exactly. I installed 397. And now they both don’t work again. I’m desperate. I don’t know what to do anymore.
The CNN code works fine on CPU though.
Someone also suggested to delete .nv folder in home directory. I couldn’t find that .nv folder.

Error given by code with CNN:

2018 15:50:40.136802: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018 15:50:40.925004: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.34GiB
2018 15:50:40.930269: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) ‑> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2018 15:50:47.834229: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.866263: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.905931: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.907669: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.909351: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.911059: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:48.585303: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018 15:50:48.586924: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018 15:50:48.588290: F C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream‑>parent()‑>GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)
2018 15:53:45.749902: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018 15:53:46.246273: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.34GiB
2018 15:53:46.248497: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) ‑> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2018 15:53:49.372376: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.373772: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.377767: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.379147: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.380568: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.382145: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.646183: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018 15:53:49.647436: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018 15:53:49.648615: F C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream‑>parent()‑>GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)

Error given by capsnet code:

2018 15:49:02.623151: W C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\bfc_allocator.cc:277] *****************************************___________________________________________________________
2018 15:49:02.624700: W C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1198] Resource exhausted: OOM when allocating tensor with shape[24,6,6,3,3,32,32,4,4,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2018 16:22:12.617066: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018 16:22:13.091828: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.34GiB
2018 16:22:13.094135: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) ‑> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2018 16:23:36.501624: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018 16:23:36.504510: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018 16:23:36.507233: F C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream‑>parent()‑>GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)

adnanz_2002 · January 12, 2019, 3:21pm

Having similar kind of problem,

[I 20:15:32.795 NotebookApp] Saving file at /Untitled6.ipynb
2019-01-12 20:15:54.621580: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-01-12 20:15:54.626759: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-01-12 20:15:54.632440: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-01-12 20:15:54.636430: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-01-12 20:15:54.642570: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-01-12 20:15:54.646620: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-01-12 20:15:56.280068: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2019-01-12 20:15:56.284640: F C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)

is there any solution please share

or.bennatan · May 17, 2020, 4:47am

I have a similar issue.
Was there any response from NVidia on this issue?
The problem disappears only on full machine startup, and this is the reason I think it is chip related. That is, closing pycharm and starting all over doesn’t make it go away.
C:\Users\orben\PycharmProjects\envs\IKnowFirst\Scripts\python.exe “C:\Program Files\JetBrains\PyCharm Community Edition 2019.3.2\plugins\python-ce\helpers\pydev\pydevd.py” --multiproc --qt-support=auto --client 127.0.0.1 --port 59562 --file C:/Users/orben/PycharmProjects/IKnowFirst/generic_trade_success/RegressTodayPredictionFile.py C:\Users\orben\PycharmProjects\IKnowFirst\deeplearningsimulate.json

pydev debugger: process 9316 is connecting

Connected to pydev debugger (build 201.7223.92)

C:\Users\orben\PycharmProjects\envs\IKnowFirst\lib\site-packages\sklearn\externals\joblib_init_.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.

warnings.warn(msg, category=FutureWarning)

Using TensorFlow backend.

2020-05-17 06:40:48.083194: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll

processing ikf file IKForecast_stocks_top_10_SP500_17_May_2020.xls

2020-05-17 06:40:58.892326: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll

2020-05-17 06:40:58.992314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:

pciBusID: 0000:01:00.0 name: GeForce GT 730 computeCapability: 3.5

coreClock: 0.9015GHz coreCount: 2 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 37.33GiB/s

2020-05-17 06:40:58.992669: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll

2020-05-17 06:40:58.999471: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll

2020-05-17 06:40:59.003427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll

2020-05-17 06:40:59.005460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll

2020-05-17 06:40:59.012072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll

2020-05-17 06:40:59.016325: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll

2020-05-17 06:40:59.026620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll

2020-05-17 06:40:59.027401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0

2020-05-17 06:40:59.027952: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2

2020-05-17 06:40:59.028715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:

pciBusID: 0000:01:00.0 name: GeForce GT 730 computeCapability: 3.5

coreClock: 0.9015GHz coreCount: 2 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 37.33GiB/s

2020-05-17 06:40:59.029105: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll

2020-05-17 06:40:59.029291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll

2020-05-17 06:40:59.029495: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll

2020-05-17 06:40:59.029685: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll

2020-05-17 06:40:59.029868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll

2020-05-17 06:40:59.030053: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll

2020-05-17 06:40:59.030234: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll

2020-05-17 06:40:59.030899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0

2020-05-17 06:40:59.687974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:

2020-05-17 06:40:59.688176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0

2020-05-17 06:40:59.688310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N

2020-05-17 06:40:59.689324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1406 MB memory) → physical GPU (device: 0, name: GeForce GT 730, pci bus id: 0000:01:00.0, compute capability: 3.5)

(120, 47)

2020-05-17 06:41:01.054440: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll

2020-05-17 06:41:01.299283: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

2020-05-17 06:41:01.299614: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

2020-05-17 06:41:01.299881: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

2020-05-17 06:41:01.300115: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

2020-05-17 06:41:01.300352: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

2020-05-17 06:41:01.301086: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

2020-05-17 06:41:01.301286: W tensorflow/stream_executor/stream.cc:2041] attempting to perform BLAS operation using StreamExecutor without BLAS support

2020-05-17 06:41:01.301544: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Internal: Blas GEMM launch failed : a.shape=(32, 47), b.shape=(47, 70), m=32, n=70, k=47

[[{{node dense_1/MatMul}}]]

Traceback (most recent call last):

File “C:\Users\orben\PycharmProjects\envs\IKnowFirst\lib\site-packages\tensorflow_core\python\eager\function.py”, line 545, in call

ctx=ctx)

File “C:\Users\orben\PycharmProjects\envs\IKnowFirst\lib\site-packages\tensorflow_core\python\eager\execute.py”, line 67, in quick_execute

six.raise_from(core._status_to_exception(e.code, message), None)

File “”, line 3, in raise_from

tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(32, 47), b.shape=(47, 70), m=32, n=70, k=47

[[node dense_1/MatMul (defined at \Users\orben\PycharmProjects\envs\IKnowFirst\lib\site-packages\keras\backend\tensorflow_backend.py:3009) ]] [Op:__inference_keras_scratch_graph_1389]

Function call stack:

keras_scratch_graph

Process finished with exit code -1