cuda_driver.cc:175] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: invalid argument

Accelerated Computing CUDA CUDA Setup and Installation
1

Hi,

My environment is:
windows 10, VS2017, GPU 2080Ti, GPU driver 441.12, CUDA 10.2.88, cudnn version cudnn-10.1-windows10-x64-v7.6.3.30.

  1. On my computer, I could correctly run the weights model of deep learning neural network on Python with GPU dirver, CUDA and cudnn.
  2. on the same computer, I could correctly run the same weights model on VS2017 on CPU without GPU.
  3. without changing anything, I just config corresponding libs and dlls, and tried to run the weights model on GPU. I encountered the following issue:
2020-03-09 10:59:02.700225: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-03-09 10:59:02.715150: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2020-03-09 10:59:02.754034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:01:00.0
2020-03-09 10:59:02.762206: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-03-09 10:59:02.766599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-03-09 10:59:02.886320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-09 10:59:02.890206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2020-03-09 10:59:02.893006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2020-03-09 10:59:02.895875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5632 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-03-09 10:59:04.349761: F tensorflow/stream_executor/cuda/cuda_driver.cc:175] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: invalid argument
3 Likes
2

Same. Did you solve this problem?
My environment is tf 1.15.3 + CUDA10.0 + cudnn 7.4 and my GPU is 2060. My laptop works completely fine with tf 2.2 + CUDA 10.1 + cudnn 7.4 and I guess it’s because an inconsistency between my GPU driver version and tf 1.15.3.
If you have found any way to solve this, please help me out too, thanks a lot!

3

I solved it long time ago.
I roughly remember it might because both cuda and cudnn version don’t match with your tensorflow. Choose a proper version of cuda and cudnn which match your tensorflow might solve your issue.