Nvdia Quadro GP100 on Ubuntu 22.04

I got a new pc, and am having troubles getting my extra graphics cards to be detected by PyTorch and nvidia-smi. I’ve looked around and can’t find articles with the same issue

I’m using Ubuntu 22.04

I have a 4060 ti and two Quadro GP100s connected by a NVLink bridge

nvidia-smi detects my 4060 just fine and it outputs graphics, but not the quadros. lspci detects one quadro though, so they seem to be registered by the system. However, I can’t get any graphics output from them.

The nvidia driver installed is 560.35.03, which Nvidia says supports the GP100s.

I found this saying the GP100 is certified for Ubuntu 20.04, but could not find a similar page for 22.04. Do I need to switch to 20.04 to use my GP100s?

Does anyone know how to fix this? Is this somehow due to the NVlink bridge? Is there other software I need to install?

I set up my drivers automatically with

   sudo ubuntu-drivers install

and I installed the cuda toolkit with

    sudo apt install nvidia-cuda-toolkit

Below is the output of lspci | grep VGA

01:00.0 VGA compatible controller: NVIDIA Corporation Device 2803 (rev a1)
16:00.0 VGA compatible controller: NVIDIA Corporation GP100GL [Quadro GP100] (rev a1)
17:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 164e (rev c9)

Could there be an issue since only one Quadro is detected by lspci?

Below is the output of nvidia-smi

  +-----------------------------------------------------------------------------------------+
  | NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
  |-----------------------------------------+------------------------+----------------------+
  | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
  |                                         |                        |               MIG M. |
  |=========================================+========================+======================|
  |   0  NVIDIA GeForce RTX 4060 Ti     Off |   00000000:01:00.0  On |                  N/A |
  |  0%   38C    P8             10W /  160W |     446MiB /   8188MiB |     14%      Default |
  |                                         |                        |                  N/A |
  +-----------------------------------------+------------------------+----------------------+
                                                                                           
  +-----------------------------------------------------------------------------------------+
  | Processes:                                                                              |
  |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
  |        ID   ID                                                               Usage      |
  |=========================================================================================|
  |    0   N/A  N/A      1675      G   /usr/lib/xorg/Xorg                            294MiB |
  |    0   N/A  N/A      1952      G   /usr/bin/gnome-shell                           97MiB |
  +-----------------------------------------------------------------------------------------+