GPU not found - NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

Accelerated Computing CUDA CUDA Setup and Installation
1

My GPU was working fine and suddenly the following happened. This happened twice where I have had to terminate my virtual machines and re-install cuda/python environment. I would install cuda, check nvidia-smi, cat /proc/driver/nvidia/version and nvcc --version returning the expected outputs. Now, without prior warning:

The results of the checking nvidia:

  1. nvidia-smi
    gives:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
  1. cat /proc/driver/nvidia/version
    gives:
cat: /proc/driver/nvidia/version: No such file or directory
  1. nvcc --version
    gives:
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Thu_Jun_11_22:26:38_PDT_2020
Cuda compilation tools, release 11.0, V11.0.194
Build cuda_11.0_bu.TC445_37.28540450_0

Other information that might be helpful. dpkg -l | grep nvidia
ii libnvidia-cfg1-450:amd64 450.51.05-0ubuntu1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-450 450.51.05-0ubuntu1 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-418:amd64 430.50-0ubuntu3 amd64 Transitional package for libnvidia-compute-430
ii libnvidia-compute-430:amd64 440.100-0ubuntu0.20.04.1 amd64 Transitional package for libnvidia-compute-440
ii libnvidia-compute-440:amd64 450.119.03-0ubuntu0.20.04.1 amd64 Transitional package for libnvidia-compute-450
ii libnvidia-compute-450:amd64 450.51.05-0ubuntu1 amd64 NVIDIA libcompute package
rc libnvidia-compute-450-server:amd64 450.142.00-0ubuntu0.20.04.1 amd64 NVIDIA libcompute package
ii libnvidia-decode-450:amd64 450.51.05-0ubuntu1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-450:amd64 450.51.05-0ubuntu1 amd64 NVENC Video Encoding runtime library
ii libnvidia-extra-450:amd64 450.51.05-0ubuntu1 amd64 Extra libraries for the NVIDIA driver
ii libnvidia-fbc1-450:amd64 450.51.05-0ubuntu1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-450:amd64 450.51.05-0ubuntu1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-ifr1-450:amd64 450.51.05-0ubuntu1 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime library
ii nvidia-compute-utils-450 450.51.05-0ubuntu1 amd64 NVIDIA compute utilities
ii nvidia-dkms-450 450.51.05-0ubuntu1 amd64 NVIDIA DKMS package
ii nvidia-driver-450 450.51.05-0ubuntu1 amd64 NVIDIA driver metapackage
ii nvidia-kernel-common-450 450.51.05-0ubuntu1 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-450 450.51.05-0ubuntu1 amd64 NVIDIA kernel source package
ii nvidia-modprobe 450.51.05-0ubuntu1 amd64 Load the NVIDIA kernel driver and create device files
ii nvidia-prime 0.8.16~0.20.04.1 all Tools to enable NVIDIA’s Prime
ii nvidia-settings 450.51.05-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-450 450.51.05-0ubuntu1 amd64 NVIDIA driver support binaries
ii screen-resolution-extra 0.18build1 all Extension for the nvidia-settings control panel
ii xserver-xorg-video-nvidia-450 450.51.05-0ubuntu1 amd64 NVIDIA binary Xorg driver

lspci | grep -i nvidia
00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)

2

nvidia-settings
gives:
ERROR: NVIDIA driver is not loaded

dkms status
gives:
nvidia, 450.51.05, 5.4.0-1045-aws, x86_64: installed

3

Hi @omar.qusous,
This forum branch is dedicated to CUDA-GDB tool support. Your question might be more suitable for CUDA Setup and Installation - NVIDIA Developer Forums forum branch. I have moved your topic there.