I tried cuda 9.0 and it worked，but when i run yolo-v3 model to train my own model, it report an error:
failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
When i run the mnist dataset ， it worked perfectly well.
Where is the problem
D:\anaconda\python.exe D:/keras-yolo3/train.py
D:\anaconda\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
2018-10-16 16:40:21.107744: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-10-16 16:40:21.443178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.53GiB
2018-10-16 16:40:21.443542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2018-10-16 16:40:22.278990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-16 16:40:22.279198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] 0
2018-10-16 16:40:22.279321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0: N
2018-10-16 16:40:22.279544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6270 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
Create YOLOv3 model with 9 anchors and 13 classes.
D:\anaconda\lib\site-packages\keras\engine\saving.py:1140: UserWarning: Skipping loading of weights for layer conv2d_59 due to mismatch in shape ((1, 1, 1024, 54) vs (255, 1024, 1, 1)).
weight_values[i].shape))
D:\anaconda\lib\site-packages\keras\engine\saving.py:1140: UserWarning: Skipping loading of weights for layer conv2d_59 due to mismatch in shape ((54,) vs (255,)).
weight_values[i].shape))
D:\anaconda\lib\site-packages\keras\engine\saving.py:1140: UserWarning: Skipping loading of weights for layer conv2d_67 due to mismatch in shape ((1, 1, 512, 54) vs (255, 512, 1, 1)).
weight_values[i].shape))
D:\anaconda\lib\site-packages\keras\engine\saving.py:1140: UserWarning: Skipping loading of weights for layer conv2d_67 due to mismatch in shape ((54,) vs (255,)).
weight_values[i].shape))
D:\anaconda\lib\site-packages\keras\engine\saving.py:1140: UserWarning: Skipping loading of weights for layer conv2d_75 due to mismatch in shape ((1, 1, 256, 54) vs (255, 256, 1, 1)).
weight_values[i].shape))
D:\anaconda\lib\site-packages\keras\engine\saving.py:1140: UserWarning: Skipping loading of weights for layer conv2d_75 due to mismatch in shape ((54,) vs (255,)).
weight_values[i].shape))
Load weights model_data/yolo_weights.h5.
Freeze the first 249 layers of total 252 layers.
Train on 468 samples, val on 51 samples, with batch size 2.
Epoch 1/800
2018-10-16 16:40:35.133313: E tensorflow/stream_executor/cuda/cuda_blas.cc:652] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
File "D:/keras-yolo3/train.py", line 217, in <module>
_main()
File "D:/keras-yolo3/train.py", line 84, in _main
epochs=800
File "D:\anaconda\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "D:\anaconda\lib\site-packages\keras\engine\training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "D:\anaconda\lib\site-packages\keras\engine\training_generator.py", line 217, in fit_generator
class_weight=class_weight)
File "D:\anaconda\lib\site-packages\keras\engine\training.py", line 1217, in train_on_batch
outputs = self.train_function(ins)
File "D:\anaconda\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "D:\anaconda\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "D:\anaconda\lib\site-packages\tensorflow\python\client\session.py", line 1399, in __call__
run_metadata_ptr)
File "D:\anaconda\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 526, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=86528, n=32, k=64
[[{{node conv2d_3/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu, conv2d_3/kernel/read)]]
[[{{node yolo_loss/while_1/strided_slice/stack/_2895}} = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3114_yolo_loss/while_1/strided_slice/stack", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopyolo_loss/while_1/strided_slice_1/stack_2/_2805)]]
I wonder if cuda 9.2 can work
According to the CUDA release notes, CUDA 10 added support for the Turing architecture (compute capability 7.5). E.g. here:
https://docs.nvidia.com/cuda/cuda-memcheck/index.html#release-notes
If it were me, I would use CUDA 10
I just successed !
cuda 9.2
cudnn 7.3.1.20
most important is that tensorflow seems not support cuda 9.2+ ,so i find a tensorflow.whl to install locally.
[url]https://github.com/fo40225/tensorflow-windows-wheel.git[/url]
If it changed to cuda 10 , which tensorflow version should be?
1.110 ?
According to my colleague, i will report an error “CUDNN STATUS ALLOC FAILED”
I think CUDA 10 is too new to be available in an official TF wheel at this point, although this will probably change in the future.
Therefore, to use CUDA 10 at the moment with TF on linux, my suggestion would be to build TF from sources. This is documented elsewhere and I would not try to document the process here.
On windows I am successfully using CUDA 10, Anaconda with python 3.6, and the wheel files here with an EVGA RTX 2080
could you point me in the right direction please i have an rtx 2070
The latest versions of TF support CUDA 10.0
CUDA 10.0 can work with rtx 2070
how can i install it…im having trouble
after installing cuda and cudnn can i just run the conda install tensorflow-gpu in my env?
i did that im getting errors
ubuntu 18.04, rtx 2070, nvidia driver: 410.48, cuda 10.0.130, cudnn 7.4.2.24, tensorflow-gpu 1.13.1
i uninstall cudnn, cuda, nvidia driver and reboot the computer before reinstall
( reference: GitHub - mjiUST/driver_cuda_cudnn: instruction for the installation of the Nvidia driver + cuda + cudnn )
reinstall nvidia driver and cuda with runfile (local) from https://developer.nvidia.com/cuda-10.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal
install cudnn and tensorflow-gpu
and solve the cuDNN failed to initialize error by setting up allow_growth = True in tensorflow session config
( reference1: Error : Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. · Issue #24828 · tensorflow/tensorflow · GitHub )
( reference2: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR · Issue #24496 · tensorflow/tensorflow · GitHub )
and then tensorflow-gpu works well :)
here are some notes too :)
https://gitlab.com/tainvecs/note/tree/master/ubuntu-note
“solve the cuDNN failed to initialize error by setting up allow_growth = True”
Do you get the same success with Cudnn 7.5?
So regarding the original question, can RTX2080 support CUDA 9?
Been using CUDA 10 & it works, but I would like to know if using CUDA 9 is possible too.
It should be possible. Keep your current driver. Install CUDA 9.
When you compile CUDA codes, you will have to make sure that all codes (and libraries) are compiled with PTX so they can JIT-compile to cc7.5
This requirement would apply to anything else you use or install (like if you install tensorflow binaries, for example). They and any libraries either have to have compatible SASS or compatible PTX for Turing.
The only way to know for sure in a given case (software stack) is to try it.
But certainly if you are compiling your own codes, and you compile with PTX on CUDA 9, those will run on Turing.
CUDA 10.0 is the release that formally added support for Turing:
[url]Release Notes :: CUDA Toolkit Documentation
This means with CUDA 10.0 or later, you can specify a cc7.5 target when compiling device code, and thereby include Turing SASS in your binary.
Having said that, if you go back and read the first post in this thread, it may make more sense to you. The only answer to whether it works or not is “it depends on exactly what software you are using, and how it was built”
I am using CUDA 11.3 version and still i am facing this error. I am using RTX 2060