NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver - Ubuntu

I had a problem with cuda and nvidia driver compatibility issue on Ubuntu Linux OS, so I gave the following commands to install latest drivers
(I am logged in using SSH through remote PC)
$ sudo ubuntu-drivers install

$ sudo reboot

I get error after nvidia-smi as follows
$ nvidia-smi

NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
I have also shared nvidia-bug-report here nvidia-bug-report
I am giving below outputs from various commands if it might help in diagnosis.

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

$nvidia-settings -v
nvidia-settings: version 595.58.03
The NVIDIA Settings tool.

$dpkg -l | grep nvidia-driver
iU nvidia-driver-580 580.126.20-1ubuntu1 amd64 NVIDIA driver metapackage

$dkms status

shows nothing

$ lspci -k

shows following about NVIDIA

01:00.0 3D controller: NVIDIA Corporation GA100GL [A30 PCIe] (rev a1)
Subsystem: NVIDIA Corporation GA100GL [A30 PCIe]
Kernel modules: nvidiafb, nouveau
02:00.0 Non-Volatile memory controller: Device 025e:f1ab (rev 03)
Subsystem: Device 025e:3910
Kernel driver in use: nvme
Kernel modules: nvme
04:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Ethernet Controller I225-V
Kernel driver in use: igc
Kernel modules: igc
05:00.0 VGA compatible controller: NVIDIA Corporation Device 2702 (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device 4144
Kernel modules: nvidiafb, nouveau
05:00.1 Audio device: NVIDIA Corporation Device 22bb (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device 4144
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

$ grep EE /var/log/Xorg.0.log

[ 88733.068] Current Operating System: Linux user-MS-7E07 6.5.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2 x86_64
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[ 88733.908] (EE) NVIDIA(GPU-1): The NVIDIA GPU at PCI:1:0:0 is not supported by the 535.161.07
[ 88733.908] (EE) NVIDIA(GPU-1): NVIDIA driver.
[ 88733.908] (EE) NVIDIA(GPU-1): Failed to initialize the NVIDIA graphics device!
[ 88733.908] (EE) NVIDIA(G0): Failing initialization of X screen
[ 88734.033] (EE) NVIDIA(GPU-1): The NVIDIA GPU at PCI:1:0:0 is not supported by the 535.161.07
[ 88734.033] (EE) NVIDIA(GPU-1): NVIDIA driver.
[ 88734.033] (EE) NVIDIA(GPU-1): Failed to initialize the NVIDIA graphics device!
[ 88734.205] (II) Initializing extension MIT-SCREEN-SAVER

​$ sudo journalctl -b 0 -p err

Mar 30 08:59:14 user-MS-7E07 gdm-launch-environment][3352029]: GLib-GObject: g_object_unref: assertion ‘G_IS_OBJECT (object)’ failed

I looked at similar posts before, but I still cannot figure out what is the issue in my case. I’d really appreciate if you could help in finding what causes this failure.
Thank you in advance.

Hi @sine.arc,

Let’s have a closer look at the things you posted above.
First driver version:

nvidia-settings: version 595.58.03

Second driver version, the one ubuntu-drivers install installed:

iU nvidia-driver-580 580.126.20-1ubuntu1 amd64 NVIDIA driver metapackage

Third driver version, old leftover:

[ 88733.908] (EE) NVIDIA(GPU-1): The NVIDIA GPU at PCI:1:0:0 is not supported by the 535.161.07

Fourth driver modules by Nouveau:

Subsystem: NVIDIA Corporation GA100GL [A30 PCIe]
Kernel modules: nvidiafb, nouveau

I highly recommend a fresh reinstall of Ubuntu and following the detailed CUDA installation instructions.

Thanks!