Use gpu for tensorflow, crashes

Accelerated Computing CUDA CUDA Setup and Installation
#1

New to gpu computing, have installed a gtx 1660ti with compute capability of 7.5. I’m using on ubuntu 21.10, have set up nvidia-toolkit, cudnn, tensorflow, tensorflow-gpu in a conda env, all appears to work fine: 1 gpu visible, built with cudnn 11.6.r11.6, tf version 2.8.0, python version 3.7.10 all in conda env running on a jupyter notebook. All seems to run fine until I attempt to train a model and then I get this error message:

2022-03-19 04:42:48.005029: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8302

and then the kernel just locks up and crashes. BTW the code worked prior to installing gpu, when it simply used cpu. Is this simply a version mismatch somewhere between python, tensorflow, tensorflow-gpu, cudnn versions or something more sinister? Thx. J.

4 Likes
#2

Hi, I have the exactly same problem, I am using tensorflow 2.8.0 with CudNN 11.2 and python 3.7.6 with one GPU visible. The code also worked before I set up the Toolkits and and environment variables, but now it crashes after the same message.

2022󈚨󈚫 15:45:05.860344: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8400

I would really appreciate if some one could help out with this issue, or maybe if @jimmaasuk you have found a solution for this problem,

Best wishes,
Bartosz

5 Likes
#3

me too , i would appreciate it vary much

#4

Same exact problem here, running on tf 2.8 and python 3.9 in PyCharm with a Conda env. Would very much appreciate some help with this.

#5

I have the same problem, the environment is py3.8.10 tf2.8 keras 2.8 output log: Epoch 1 / 30 2022-04-23 11:25:22.138221: I tensorflow/stream_executor/cuda/cuda_dnn.cc: 368] Loaded cuDNN version 8400,

#6

Hallo,
I had the same issue, although I do not know how to fix the problem, I have downgraded to tensorflow and keras 2.4.0 and to CUDA version 11.0 and cuDNN 8.0 and now I can finally work without issues.
Best luck

#7

me too.
same issue.
GTX1060, tf 2.8( using V1 API) , python 3.9 , CUDA11.6, cuDnn_windows_8.4.0.27

the code worked well with CPU, untill cuDnn was installed.

2022-05-03 23:36:26.225210: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8400

Process finished with exit code -1073740791 (0xC0000409)

1 Like
#8

My issue was resolved.

Nvidia driver, cuDnn version, python, CUDA…… everything is ok.

Root cause: a wrong version tensorflow(only for CPU) was installed in pycharm. I removed it, and installed tensorflow-gpu version

#9

Hello,
I’m experiencing the very same issue right now, same GPU model.

@lx6636 your resolution didn’t help me :(

@jimmaasuk (or anyone else in this post) - did any of you find a solution for that??

#10

Resolved by installing Zlib and add it to path

#11

Hello, you can try this:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

Tensorflow was slower, but it will working.