Only one of two RTX 3090s detected with nvidia-smi (Ubuntu 20.04)

I got two rtx 3090 cards for my deep learning setup. Both are connected to my X570 motherboard (which supports dual gpu SLI). I know both GPU cards are working because when I connect them individually they can be detected with nvidia-smi from their respective PCIE slots, it’s just nvidia-smi doesn’t recognize both of them together at the same time. Also in windows 10, 2x RTX 3090 cards are detected by the Geforce Experience software, it is ubuntu that is detecting only 1 out of the 2 cards.

In the terminal, with lspci |grep -i vga:
2d:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1)
2e:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1)

with dmesg | grep NVRM:
[ 2.582063] NVRM: The NVIDIA probe routine was not called for 1 device(s).
[ 2.582065] NVRM: This can occur when a driver such as:
NVRM: nouveau, rivafb, nvidiafb or rivatv
NVRM: was loaded and obtained ownership of the NVIDIA device(s).
[ 2.582065] NVRM: Try unloading the conflicting kernel module (and/or
NVRM: reconfigure your kernel without the conflicting
NVRM: driver(s)), then try loading the NVIDIA kernel module
NVRM: again.
[ 2.582066] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 455.38 Thu Oct 22 06:06:59 UTC 2020

with nvidia-smi:
±----------------------------------------------------------------------------+
| NVIDIA-SMI 455.38 Driver Version: 455.38 CUDA Version: 11.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 3090 Off | 00000000:2D:00.0 On | N/A |
| 30% 43C P8 14W / 350W | 230MiB / 24245MiB | 2% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1078      G   /usr/lib/xorg/Xorg                107MiB |
|    0   N/A  N/A      1544      G   /usr/bin/gnome-shell              116MiB |
|    0   N/A  N/A      2342      G   /usr/lib/firefox/firefox            4MiB |
+-----------------------------------------------------------------------------+

I’ve reinstalled the driver via:
sudo apt install nvidia-driver-455

and also have cuda 11.1 installed.

Can anyone help me get both cards recognized by nvidia-smi? Thank you~!

After one day of trying various things, I finally got it to work. Here is the solution:

sudo nano /etc/modprobe.d/blacklist.conf
add the following line to the end of the file:
blacklist nvidiafb
blacklist nouveau

save file and run the following 2 lines in terminal:
sudo depmod -ae
sudo update-initramfs -u

woo hoo it works!

nvidia-smi
Sun Nov 22 00:37:11 2020
±----------------------------------------------------------------------------+
| NVIDIA-SMI 455.23.04 Driver Version: 455.23.04 CUDA Version: 11.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 3090 Off | 00000000:2D:00.0 On | N/A |
| 0% 39C P8 13W / 350W | 330MiB / 24265MiB | 1% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 GeForce RTX 3090 Off | 00000000:2E:00.0 Off | N/A |
| 0% 35C P8 8W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1104 G /usr/lib/xorg/Xorg 177MiB |
| 0 N/A N/A 1474 G /usr/bin/gnome-shell 145MiB |
| 0 N/A N/A 2929 G /usr/lib/firefox/firefox 4MiB |
| 1 N/A N/A 1104 G /usr/lib/xorg/Xorg 4MiB |
| 1 N/A N/A 1474 G /usr/bin/gnome-shell 0MiB |
| 1 N/A N/A 2929 G /usr/lib/firefox/firefox 0MiB |
±----------------------------------------------------------------------------+

upgrade driver version worked for me
I use the version of 470.57.02