nvidia-smi "No devices were found" error

All of a sudden started having the issue after a reboot about a week ago as well, using Ubuntu 20.04.

Things i’ve tried so far:

  • Upgrading Ubuntu 20.04
  • Upgrading/Downgrading NVIDIA driver via run file (450.66 / 455.38 / 440.100)
  • Nouveau driver is blacklisted, i tried running another initramfs to update the kernel.

I then purged everything e.g

sudo apt-get remove nvidia-* xserver-xorg-* && sudo apt-get purge nvidia-* xserver-xorg-* && sudo apt-get autoclean && sudo apt-get autoremove

Shutdown, and tested the card in a different system where it ran perfectly. Loaded it up to 100% for 2 hours and completed tasks fine. I then put the card back into my Ubuntu Server and reinstalled, this time via aptitude

sudo apt-get install dkms build-essential linux-headers-$(uname -r)
sudo nano /etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidiafb
alias nouveau off
echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
sudo apt-get install nvidia-headless-450-server nvidia-utils-450-server nvidia-container-runtime nvidia-container-toolkit nvidia-docker2

Still have the same issue

$ lspci | grep NVIDIA
07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
07:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

$ dmesg | grep NVRM
[ 1.569623] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 450.80.02 Wed Sep 23 01:13:39 UTC 2020
[ 65.390214] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x26:0xffff:1266)
[ 65.390403] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0

$ dmesg | grep NVIDIA
[ 1.450276] nvidia: module license ‘NVIDIA’ taints kernel.
[ 1.569623] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 450.80.02 Wed Sep 23 01:13:39 UTC 2020
[ 1.572949] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 450.80.02 Wed Sep 23 00:48:09 UTC 2020

$ cat /var/log/kern.log | grep taint
Nov 1 14:13:49 mediabox kernel: [ 1.450269] nvidia: loading out-of-tree module taints kernel.
Nov 1 14:13:49 mediabox kernel: [ 1.450276] nvidia: module license ‘NVIDIA’ taints kernel.
Nov 1 14:13:49 mediabox kernel: [ 1.450278] Disabling lock debugging due to kernel taint
Nov 1 14:13:49 mediabox kernel: [ 1.460693] nvidia: module verification failed: signature and/or required key missing - tainting kernel

$ sudo nvidia-smi
No devices were found