] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: invalid argument


My environment is:
windows 10, VS2017, GPU 2080Ti, GPU driver 441.12, CUDA 10.2.88, cudnn version cudnn-10.1-windows10-x64-v7.6.3.30.

  1. On my computer, I could correctly run the weights model of deep learning neural network on Python with GPU dirver, CUDA and cudnn.
  2. on the same computer, I could correctly run the same weights model on VS2017 on CPU without GPU.
  3. without changing anything, I just config corresponding libs and dlls, and tried to run the weights model on GPU. I encountered the following issue:
2020-03-09 10:59:02.700225: I tensorflow/core/platform/] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-03-09 10:59:02.715150: I tensorflow/stream_executor/platform/default/] Successfully opened dynamic library nvcuda.dll
2020-03-09 10:59:02.754034: I tensorflow/core/common_runtime/gpu/] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:01:00.0
2020-03-09 10:59:02.762206: I tensorflow/stream_executor/platform/default/] GPU libraries are statically linked, skip dlopen check.
2020-03-09 10:59:02.766599: I tensorflow/core/common_runtime/gpu/] Adding visible gpu devices: 0
2020-03-09 10:59:02.886320: I tensorflow/core/common_runtime/gpu/] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-09 10:59:02.890206: I tensorflow/core/common_runtime/gpu/]      0
2020-03-09 10:59:02.893006: I tensorflow/core/common_runtime/gpu/] 0:   N
2020-03-09 10:59:02.895875: I tensorflow/core/common_runtime/gpu/] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5632 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-03-09 10:59:04.349761: F tensorflow/stream_executor/cuda/] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: invalid argument

Same. Did you solve this problem?
My environment is tf 1.15.3 + CUDA10.0 + cudnn 7.4 and my GPU is 2060. My laptop works completely fine with tf 2.2 + CUDA 10.1 + cudnn 7.4 and I guess it’s because an inconsistency between my GPU driver version and tf 1.15.3.
If you have found any way to solve this, please help me out too, thanks a lot!

I solved it long time ago.
I roughly remember it might because both cuda and cudnn version don’t match with your tensorflow. Choose a proper version of cuda and cudnn which match your tensorflow might solve your issue.