Issue getting nvidia-smi to find devices

I can’t seem to get nvidia-smi to detect any of the cards; especially the P40. I downloaded and installed the data center driver and did a nvidia-bug-report.sh. the log file is attached.

nvidia-bug-report.log (187.1 KB)

Hi there @user106385 and welcome to the NVIDIA developer forums.

I have read only the beginning of the log, but already see several problems.

WARNING: The NVIDIA Quadro FX 580 GPU installed in this system is supported through the NVIDIA 340.xx legacy Linux graphics drivers.  Please visit http://www.nvidia.com/object/unix.html for more information.  The 525.60.13 NVIDIA Linux graphics driver will ignore this GPU.
ERROR: Unable to load the kernel module 'nvidia.ko'.  This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or if another driver, such as nouveau, is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA device(s), or no NVIDIA device installed in this system is supported by this NVIDIA Linux graphics driver release.
/proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  545.23.06  Sun Oct 15 17:43:11 UTC 2023
GCC version:  gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
-> Kernel messages:
               NVRM:  visit http://www.nvidia.com/object/unix.html for more
               NVRM:  information.  The 525.60.13 NVIDIA driver will ignore
               NVRM:  this GPU.  Continuing probe...
[ 1457.174231] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
               NVRM: BAR1 is 0M @ 0x0 (PCI:0000:03:00.0)
[ 1457.174235] nvidia: probe of 0000:03:00.0 failed with error -1
[ 1457.174248] NVRM: ignoring the legacy GPU 0000:04:00.0
[ 1457.174250] nvidia: probe of 0000:04:00.0 failed with error -1
[ 1457.174267] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 1457.174269] NVRM: None of the NVIDIA devices were initialized.

You should not mix these types of GPUs in one system. Remove the old Quadro, enable Resizable BAR in your system BIOS, purge all NVIDIA drivers from your system, reinstall the latest driver for P40 and try again, don’t mix driver 525 and 545.

Thanks!