Ubuntu - NVLink not working with two RTX 3090

Hi!

I have two RTX 3090 installed. Both are properly connected via NVLink. Unfortunately, NVLink does not work.

Any ideas?

I have compiled some outputs below.

All the best,
Tristan

NVIDIA SMI NVLink:

$ nvidia-smi nvlink -s
GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-631e6a76-d7c7-0e5b-c011-6e20f573017c)
	 Link 0: <inactive>
	 Link 1: <inactive>
	 Link 2: <inactive>
	 Link 3: <inactive>
GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-15971343-c0e3-5be9-19fb-c1c4425ec80c)
	 Link 0: <inactive>
	 Link 1: <inactive>
	 Link 2: <inactive>
	 Link 3: <inactive>

Ubuntu distribution:

$ cat /etc/issue
Ubuntu 20.04.4 LTS \n \l

NVIDIA-SMI:

$ nvidia-smi
Sat Sep  3 15:59:22 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:0A:00.0 Off |                  N/A |
| 30%   32C    P8    12W / 350W |   2261MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:0B:00.0 Off |                  N/A |
| 30%   31C    P8    18W / 350W |      3MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       897      C   /usr/bin/python3                  837MiB |
|    0   N/A  N/A      2471      C   /usr/bin/python3                 1421MiB |
+-----------------------------------------------------------------------------+

NVIDIA cuda compiler driver:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

NVIDIA driver:

$ apt list --installed | grep nvidia-driver

nvidia-driver-515/unknown,now 515.65.01-0ubuntu1 amd64 [installed,automatic]
1 Like