CUDA 9.0 ImportError: libcublas.so.8.0

Accelerated Computing CUDA CUDA Setup and Installation
1

Hi,

Currently working on some neural nets stuff with python 3.6.3, until now I was using tensorflow cpu and recently decided to switch to -GPU version. Followed all the steps from official websites in this order:

  1. Installation of CUDA Toolkit (9.0)
  2. Installation of cuDNN v7.0
  3. Installation of libcupti-dev library (7.5.18-0ubuntu1)
  4. Installation of Tensorflow-GPU

Of course, after step one and two I have verified the installation using ./deviceQuery and ./bandwidthTest, both of them passed correctly. Also verified cuDNN using ./mnistCUDNN, test also passed.

Installed tensorflow-gpu, tried to verify this using sample code from the website (both python3 Shell and IDE Spyder) but unfortunately no luck.

The stack trace from the shell:

Traceback (most recent call last):
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/albert/anaconda3/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/albert/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 72, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/albert/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/albert/anaconda3/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/albert/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

The import error shows that libcublas.so.8.0 is missing, however in both cuda/lib64 and cuda-9.0/lib64 the file does exists (newer version libcublas.so.9.0). I have checked the environmental variables and all seems to be perfectly correct. Tried removing everything that is associated with nvidia from my linux including drivers, cuda toolkit, cudnn etc. and installing it all over again several times but I always end up getting the same error.

Any suggestions what could possibly go wrong or how to fix this?

Regards,

Albert

2

currently, the binary installable versions of TF-GPU require CUDA 8. This will probably change in the future, but for those binaries, you can’t satisfy the requirement with CUDA 9.

If you really want to use CUDA 9/CUDNN 7, you will need to build TF from sources. Otherwise install CUDA 8 and point TF at that. CUDA 8 can be installed alongside CUDA 9 and you don’t need to modify the driver; the CUDA 9 GPU driver will work with CUDA 8 install.

3

Alright, brilliant, I’ll try that solution, lastly with the cuDNN do I have to downgrade to v6 for it to work with CUDA 8?

4

cudnn has specific versions designed to work with specific CUDA versions. Please take a look at the cudnn download page, it will identify what is available for each cuda version.

5

Where are the older CUDA downloads located? From the download link only the latest CUDA 9 is available. I need to install CUDA 8 for the same reasons listed in this post.

6

google “cuda toolkit archive”

7

Thank you. I failed at searching on my own, but with your keywords I found it:

8

@txbob thanks for the info.

Would you please explain how to “point” TF at a particular version of CUDA?

Thanks.

9

If you’re using a current binary installable version of TF (say, TF 1.4), it should be already linked against CUDA 8. You don’t really have an option to “point” it at anything else. In that case, follow the instructions in the linux install guide to make sure that CUDA 8 has been properly setup in the PATH and LD_LIBRARY_PATH environment variables, and TF should find what it needs from CUDA 8.

10

Thanks @txbob, that makes sense…

Realized that /usr/local/cuda/lib64, which is in LD_LIBRARY_PATH, was symlinked to /usr/local/cuda-9.0

Changed symlink to map /usr/local/cuda to /usr/local/cuda-8.0, and now I can import tensforflow without a problem.

11

I installed cuda toolkit 8.0. And in my /usr/local folder, the cuda-8.0/ and cuda/ folders exist. Then, I put
export PATH=/usr/local/cuda-8.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/extras/CUPTI/lib64:$LD_LIBRARY_PATH
in /etc/profile.
However, it doesn’t work.
When importing tensorflow, it gives back:
libcublas.so.9.0: cannot open shared object file: No such file or directory

12

Your TF is looking for CUDA 9.0. So you’ll need to install CUDA 9.0 (if you havent already) and update your PATH and LD_LIBRARY_PATH to use it. Or get a version of TF linked against CUDA 8

13

However, from the discussion of tensorflow’s github, currently, tensorflow does not support cuda-9.0 currently. Please see the link as below.

Another issue is that, I just installed cuda-8.0, but why when I run ./deviceQuery, it gives back cuda driver version 9.0

14

Certainly that is true if you installed TF using a binary install method, such as pip

However if you build TF from sources, it’s possible for it to be linked against some other CUDA version. Anyway I have no idea what you did or what is happening exactly, but this:

certainly, without a doubt, indicates that something is looking for i.e. linked against CUDA 9.0.

The CUDA driver version is reflective of the GPU driver version that is installed. For example if you have a r384 GPU driver installed, the CUDA driver API interface within that GPU driver supports CUDA 9 (but it also supports CUDA 8, and prior CUDA versions as well). This is normal and not a problem.

15

Thanks so far.

16

When I use pip to install tf-nightly-gpu, i meet the same problem. How did you solve this problem? Thanks!

17

I solved this problem.
Here are what I have done, hope it will help you.
First, you need to check your cuda version. Use nvcc -V, if it shows cuda 9.0, then you have to uninstall it, because TF doesn’t support this version currently. Forget about install from source, because it is even harder.

Then, install cuda 8.0 and cudnn 6.0 instead. One trick is that, it’s better for your to type yum (or apt-get on ubuntu) install cuda-8-0 instead of install cuda.

After that, just run pip install tensorflow-gpu.

This works well for me. Hope it can help you.

18

try
$ sudo apt-get install cuda-8-0