While training the network, I faced this error and I am notable to train.
I am using CUDA-8.0, cuDNN-v.5.0 ,Ubuntu-14.04(64-bit)
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:105] Couldn’t open CUDA library libcudnn.so. LD_LIBRARY_PATH:
I tensorflow/stream_executor/cuda/cuda_dnn.cc:3448] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
name: GeForce GTX 980 Ti
major: 5 minor: 2 memoryClockRate (GHz) 1.2405
pciBusID 0000:01:00.0
Total memory: 5.93GiB
Free memory: 5.72GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) → (device: 0, name: GeForce GTX 980 Ti, pci bus id: 0000:01:00.0)
Any solutions is appreciated.
You need to download the cuDNN library (separately from the CUDA toolkit - it’s not part of the CUDA toolkit) and you need to put it in a location that the runtime linker can find - usually on a path that is specified in your LD_LIBRARY_PATH environment variable.
I have done it, but still I am getting this error.
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
E tensorflow/core/common_runtime/direct_session.cc:132] Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuCtxCreate: CUDA_ERROR_OUT_OF_MEMORY; total memory reported: 18446744073709551615
Traceback (most recent call last):
File “./OverFIt_Iteration_1.py”, line 346, in
initial_weights=initial_weights)
File “./OverFIt_Iteration_1.py”, line 305, in train
with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
File “/home/gpu/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1138, in init
super(Session, self).init(target, graph, config=config)
File “/home/gpu/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 502, in init
self._session = tf_session.TF_NewSession(opts, status)
File “/home/gpu/anaconda3/envs/tensorflow/lib/python3.5/contextlib.py”, line 66, in exit
next(self.gen)
File “/home/gpu/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/errors.py”, line 463, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.InternalError: Failed to create session.
Can anyone help me out.