NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

So there was some glitch in CUDA8 on the server, nvidia drivers couldn’t be identified. I purged nvidia, then kernel was downgraded to 116, then upgraded back to 119. I reinstalled driver (nvidia 384), and reinstalled cuda I keep getting these errors:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

lspci | grep -i nvidia

ubuntu-drivers devices | grep nv
driver   : nvidia-384 - third-party free
driver   : nvidia-390 - third-party free recommended


modinfo nvidia-384
filename:       /lib/modules/4.4.0-119-generic/updates/dkms/nvidia_384.ko
alias:          char-major-195-*
version:        384.130
supported:      external
license:        NVIDIA
srcversion:     586B4E0CF4083A6BC41A33B
alias:          pci:v000010DEd00000E00sv*sd*bc04sc80i00*
alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
vermagic:       4.4.0-119-generic SMP mod_unload modversions 
parm:           NVreg_Mobile:int
parm:           NVreg_ResmanDebugLevel:int
parm:           NVreg_RmLogonRC:int
parm:           NVreg_ModifyDeviceFiles:int
parm:           NVreg_DeviceFileUID:int
parm:           NVreg_DeviceFileGID:int
parm:           NVreg_DeviceFileMode:int
parm:           NVreg_UpdateMemoryTypes:int
parm:           NVreg_InitializeSystemMemoryAllocations:int
parm:           NVreg_UsePageAttributeTable:int
parm:           NVreg_MapRegistersEarly:int
parm:           NVreg_RegisterForACPIEvents:int
parm:           NVreg_CheckPCIConfigSpace:int
parm:           NVreg_EnablePCIeGen3:int
parm:           NVreg_EnableMSI:int
parm:           NVreg_TCEBypassMode:int
parm:           NVreg_UseThreadedInterrupts:int
parm:           NVreg_EnableStreamMemOPs:int
parm:           NVreg_MemoryPoolSize:int
parm:           NVreg_RegistryDwords:charp
parm:           NVreg_RegistryDwordsPerDevice:charp
parm:           NVreg_RmMsg:charp
parm:           NVreg_AssignGpus:charp


    sudo modprobe nvidia_384
modprobe: ERROR: could not insert 'nvidia_384': Exec format error

I’m not sure what to make of it all. I reinstalled the drivers and CUDA a few times now, rebooted but the problem is the same. What should I do?

If that is returning nothing, then it means that the GPUs are not responding at a hardware level, typically.

Try a reboot or better yet a power cycle, followed by running that command again. If still nothing, there is a HW problem.

Apologies, the text didn’t attach at the top of the post:

lspci|grep 'NVIDIA'
03:00.0 VGA compatible controller: NVIDIA Corporation GF119 [NVS 315] (rev a1)
03:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
04:00.0 3D controller: NVIDIA Corporation GK110BGL [Tesla K40m] (rev a1)