Ubuntu 18.04 Quadro P2000 "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver"

Graphics / Linux Linux
1

Hello,

I have tried many different ways to install Nvidia Driver but they all result in command nvidia-smi returning “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver”. I have made sure to prime-select nvidia and reboot but the error persists.

For my last attempt I made sure to purge everything nvidia and cuda related and then I have followed exactly the instructions for package manager installation on Ubuntu (see Installation Guide Linux :: CUDA Toolkit Documentation). This installs cuda 10-1 and nvidia driver 418 but problem persists with no communication with driver.

Some info about my system:

phil@p5530:~$ lspci | grep -i nvidia
01:00.0 3D controller: NVIDIA Corporation GP107GLM [Quadro P2000 Mobile] (rev a1)

phil@p5530:~$ uname -m && cat /etc/*release
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

phil@p5530:~$ uname -r
4.15.0-47-generic

Any help would be greatly appreciated, thanks!
nvidia-bug-report.log.gz (95.8 KB)

2

Quick troubleshooting:

  • make sure secure boot is disabled in bios
  • run
grep nvidia /etc/modprobe.d/* /lib/modprobe.d/*

to find a file containing

blacklist nvidia

and remove it,
then run

sudo update-initramfs -u

and reboot. If the problem persists, please run nvidia-bug-report.sh as root and attach the resulting .gz file to your post. Hovering the mouse over an existing post of yours will reveal a paperclip icon.
https://devtalk.nvidia.com/default/topic/1043347/announcements/attaching-files-to-forum-topics-posts/

3

Thanks for the quick response!

  • Secure boot was already disabled
  • I deleted the blacklist file, ran sudo update-initramfs -u and rebooted. Issue persists

I have attached the nvidia-bug-report in my original post.

4

There’s a problem with your compiler

module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 1, loc 00000000c0f0aa22, val ffffffffc1a5eb50

Did you set it to llvm/clang? It needs to be set to the system compiler (gcc 7.3.0). Afterwards, purge and reinstall the driver.

5

I do not remember specifying the compiler during installation. During which step of the cuda Ubuntu package manager installation would I specify this?

My default gcc is 7.3

phil@p5530:~/Documents$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.3.0-27ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
6

Please post the output of

update-alternatives --list cc

and

update-alternatives --display cc
7

Awesome! you fixed it.

The problem was my /usr/bin/cc was provided by clang and not gcc-7.

The solution was to update-alternative for cc so that it is provided by gcc-7, purge all cuda and nvidia packages and then reinstall everything following the Cuda linux installation instructions.

Thank you so much for the help, have a good weekend!

phil@p5530:~$ nvidia-smi
Sat May  4 14:06:41 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.56       Driver Version: 418.56       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P2000        Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8    N/A /  N/A |    118MiB /  4042MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1635      G   /usr/lib/xorg/Xorg                           118MiB |
+-----------------------------------------------------------------------------+