insufficient CUDA driver version (1080ti, Ubuntu 18.04, 390.78, CUDA 9.1.85, cuDNN 7.1.3 for 9.1)

Accelerated Computing CUDA CUDA Setup and Installation
1

Good day everyone.

I’m trying to sep up CUDA in my PC (1080ti, Ubuntu 18.04) to get Keras-TF work with my GPU. I’ve done the following:

  • Purged installed drivers

  • Installed Nvidia 390.78 drivers (I think it’s the latest for my 1080ti) with:

  • Installed CUDA Toolkit 9.1 for Ubuntu 17.04 (since there’s no 18.04 option), runfile local version. I didn’t install the driver, though, since it was older. I did install the main toolkit and the 3 available patches

  • Installed cuDNN v7.1.3 (April 17, 2018) for CUDA 9.1, since it was the newer available for CUDA 9.1

  • Installed libcupti

  • Edited my bash file with:

export PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

  • Updated (more like checked for the last version of) tensorflow-gpu and keras-gpu, which I had. I use conda, if case it means anything here.

  • Open a jupyter notebook and run:

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
  • To get:
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
<ipython-input-2-57062467377b> in <module>()
----> 1 print(device_lib.list_local_devices())

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/device_lib.py in list_local_devices(session_config)
     39   return [
     40       _convert(s)
---> 41       for s in pywrap_tensorflow.list_devices(session_config=session_config)
     42   ]

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py in list_devices(session_config)
   1677                                           status)
   1678     else:
-> 1679       return ListDevices(status)
   1680 
   1681 

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
    517             None, None,
    518             compat.as_text(c_api.TF_Message(self.status.status)),
--> 519             c_api.TF_GetCode(self.status.status))
    520     # Delete the underlying status object from memory otherwise it stays alive
    521     # as there is a reference to status from this from the traceback due to

InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version

I installed those versions since I thought they were the latest I could set in my PC (dunno if I should use CUDA 9.0 with cuDNN 7.3.0 instead, feel free to give me some advice in that regard).

What can I do? Is there any more info you would need to give me a hand?

Thanks for your time. Sorry if I should be able to get it down or if it’s a dumb question, but I can’t make it work after reading quite a bunch of posts over here…

2

CUDA 10 is the first CUDA version with support for Ubuntu 18.04. If you want to use Ubuntu 18.04, that’s the CUDA version you would want to install. Your other option is to use CUDA 9.x with Ubuntu 17.04.

My advice is to always check the Linux Installation Guide first, pick one of the supported platforms listed there, and precede with that. Trying to mix and match software components in unsupported ways is something that is best left to ninja-level experts. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html

Each CUDA version comes with a matching driver package (you can install newer drivers later if you wish). The minimum driver versions required for each CUDA version are listed in table 1 in the release notes: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html

3

Oh, I see. I saw those notes, that was the reason I stuck with CUDA 9.1, because by default it seemed that the higher driver version I could install was 390.

Would it be possible, then, to install a higher driver version with my 1080ti? That is, 410.48+. If it isn’t, is my only possibility to install Ubuntu 17.04 and work from there?

Thanks for your advice.

4

While it is difficult to make remote diagnoses (ask your car mechanic about that when you get the chance), the proximate cause of your problems seems to be the use of CUDA 9.x with an operating system that is not supported by that version of CUDA: Either upgrade your CUDA version or downgrade your OS version, so that you wind up with a combination that is listed as supported in NVIDIA documentation.

Newer drivers support older hardware (unless the hardware is very old: current drivers support GPUs with compute capability 3.0 or higher, i.e. Kepler architecture and younger) and older CUDA versions. The GTX 1080 Ti has been around for a while; any CUDA version >= 8.0 and any driver version >= 367.48 (I think, no guarantees) should work just fine. You could certainly pull the latest drivers available for any supported operating system platform that you use.

5

The kernels associated with the most recent linux kernels e.g. in Ubuntu 18.04 and Fedora 27/28 require very recent GPU driver versions.

It’s likely that your driver install was not done correctly.

As njuffa already pointed out, this should not be an issue if you use CUDA 10 on a clean load of Ubuntu 18.04, and follow the instructions in the linux install guide carefully, and install only from NVIDIA supplied sources.

6

Yay! I installed CUDA 10 with 410.48 and it works perfectly! Thanks everyone :)