Proxmox 9.0.10 passthrough to guest Debian 13 of Quadro P520, detected but nvidia-smi can't find it

I have installed Proxmox 9.0.10 and configured PCI passthrough to a guest VM running Debian 13. I first followed the guide https://linuxconfig.org/debian-13-nvidia-driver-installation on the HOST, and I it looked like it worked: I could find it with nvidia-smi.

I then realized that I wanted it on the GUEST, uninstalled the nvidia-driver and blacklisted it on the HOST.

I did passthrough to the GUEST. On the GUESTI am able to detect it, but nvidia-smi can’t find it. I have tried searching for how to identify my problem and am stuck. I hope someone on this forum can help.

I don’t have X running, I want to use the card for Plex transcoding.

Here is my nvidia-bug-report from the GUEST:

nvidia-bug-report.log.gz (65.4 KB)

In addition I tried executing some commands that could give some useful information:

On the HOST

root@host:~# grep 'GRUB_CMDLINE_LINUX_DEFAULT' /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream pcie_acs_override=multifunction nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off"

and both ~# lsmod | grep nvidia and lsmod | grep nouv return nothing.

root@host:~# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Coffee Lake HOST and DRAM Controller [8086:3e34] (rev 0c)
00:02.0 VGA compatible controller [0300]: Intel Corporation WhiskeyLake-U GT2 [UHD Graphics 620] [8086:3ea0] (rev 02)
00:04.0 Signal processing controller [1180]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem [8086:1903] (rev 0c)
00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Point-LP Thermal Controller [8086:9df9] (rev 30)
00:14.0 USB controller [0c03]: Intel Corporation Cannon Point-LP USB 3.1 xHCI Controller [8086:9ded] (rev 30)
00:14.2 RAM memory [0500]: Intel Corporation Cannon Point-LP Shared SRAM [8086:9def] (rev 30)
00:14.3 Network controller [0280]: Intel Corporation Cannon Point-LP CNVi [Wireless-AC] [8086:9df0] (rev 30)
00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Point-LP Serial IO I2C Controller #0 [8086:9de8] (rev 30)
00:16.0 Communication controller [0780]: Intel Corporation Cannon Point-LP MEI Controller #1 [8086:9de0] (rev 30)
00:16.3 Serial controller [0700]: Intel Corporation Cannon Point-LP Keyboard and Text (KT) Redirection [8086:9de3] (rev 30)
00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Point-LP PCI Express Root Port #1 [8086:9db8] (rev f0)
00:1c.4 PCI bridge [0604]: Intel Corporation Cannon Point-LP PCI Express Root Port #5 [8086:9dbc] (rev f0)
00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Point-LP PCI Express Root Port #9 [8086:9db0] (rev f0)
00:1d.4 PCI bridge [0604]: Intel Corporation Cannon Point-LP PCI Express Root Port #13 [8086:9db4] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Cannon Point-LP LPC Controller [8086:9d84] (rev 30)
00:1f.3 Audio device [0403]: Intel Corporation Cannon Point-LP High Definition Audio Controller [8086:9dc8] (rev 30)
00:1f.4 SMBus [0c05]: Intel Corporation Cannon Point-LP SMBus Controller [8086:9da3] (rev 30)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Point-LP SPI Controller [8086:9da4] (rev 30)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (6) I219-LM [8086:15bd] (rev 30)
01:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS522A PCI Express Card Reader [10ec:522a] (rev 01)
02:00.0 PCI bridge [0604]: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] [8086:15c0] (rev 01)
03:00.0 PCI bridge [0604]: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] [8086:15c0] (rev 01)
03:01.0 PCI bridge [0604]: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] [8086:15c0] (rev 01)
03:02.0 PCI bridge [0604]: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] [8086:15c0] (rev 01)
04:00.0 System peripheral [0880]: Intel Corporation JHL6240 Thunderbolt 3 NHI (Low Power) [Alpine Ridge LP 2016] [8086:15bf] (rev 01)
3a:00.0 USB controller [0c03]: Intel Corporation JHL6240 Thunderbolt 3 USB 3.1 Controller (Low Power) [Alpine Ridge LP 2016] [8086:15c1] (rev 01)
3c:00.0 3D controller [0302]: NVIDIA Corporation GP108GLM [Quadro P520] [10de:1d34] (rev a1)
3d:00.0 Non-Volatile memory controller [0108]: Sandisk Corp SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD [15b7:5006]

The guest VM config is:

root@host:~# qm config 107
agent: 1
balloon: 1024
boot: order=scsi0;ide2;net0
cores: 4
cpu: x86-64-v2-AES
hostpci0: 0000:3c:00,pcie=1
ide2: local:iso/debian-13.1.0-amd64-netinst.iso,media=cdrom,size=783M
machine: q35
memory: 8196
meta: creation-qemu=10.0.2,ctime=1758649136
name: guest
net0: virtio=BC:24:11:90:CD:36,bridge=vmbr1,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-107-disk-0,iothread=1,size=50G
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=3e4c5f06-f1e6-463e-b375-703a5a1c5400
sockets: 1
vga: std
virtiofs0: medier
vmgenid: 3d24a132-e86a-44ab-a99e-22b0dc6d48d7

nvidia-detect:

root@host:~# nvidia-detect
Detected NVIDIA GPUs:
3c:00.0 3D controller [0302]: NVIDIA Corporation GP108GLM [Quadro P520] [10de:1d34] (rev a1)

Checking card:  NVIDIA Corporation GP108GLM [Quadro P520] (rev a1)
Your card is supported by all driver versions.
Your card is also supported by the Tesla 535 drivers series.
It is recommended to install the
    nvidia-driver
package.

nvidia-smi:

root@yoda:~# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

The above should be what I want since I don’t want the HOST controlling the Nvidia card.

On the GUEST

root@guest:~# grep ‘GRUB_CMDLINE_LINUX_DEFAULT’ /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet console=tty console=ttyS0,115200 intel_iommu=on iommu=pt"
root@guest:~# lsmod | grep nvidia
nvidia_uvm           4915200  0
nvidia_drm            118784  0
nvidia_modeset       1605632  1 nvidia_drm
video                  81920  1 nvidia_modeset
nvidia              60702720  2 nvidia_uvm,nvidia_modeset
drm_ttm_helper         16384  4 bochs,drm_vram_helper,nvidia_drm
drm_kms_helper        253952  4 bochs,drm_vram_helper,drm_ttm_helper,nvidia_drm
drm                   774144  8 drm_kms_helper,bochs,drm_vram_helper,nvidia,drm_ttm_helper,nvidia_drm,ttm

while lsmod | grep nouv returns nothing

root@guest:~# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0]
00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02)
00:02.0 Mass storage controller [0180]: Red Hat, Inc. Virtio 1.0 file system [1af4:105a] (rev 01)
00:1a.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03)
00:1a.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03)
00:1a.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03)
00:1a.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03)
00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03)
00:1c.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c]
00:1c.1 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c]
00:1c.2 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c]
00:1c.3 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c]
00:1d.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03)
00:1d.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03)
00:1d.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03)
00:1d.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03)
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 92)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02)
00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02)
01:00.0 3D controller [0302]: NVIDIA Corporation GP108GLM [Quadro P520] [10de:1d34] (rev a1)
05:01.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
05:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
05:03.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
05:04.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
06:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002]
06:08.0 Communication controller [0780]: Red Hat, Inc. Virtio console [1af4:1003]
06:12.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
09:01.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio SCSI [1af4:1004]

nvidia-detect:

root@guest:~# nvidia-detect
Detected NVIDIA GPUs:
01:00.0 3D controller [0302]: NVIDIA Corporation GP108GLM [Quadro P520] [10de:1d34] (rev a1)

Checking card:  NVIDIA Corporation GP108GLM [Quadro P520] (rev a1)
Your card is supported by all driver versions.
Your card is also supported by the Tesla 535 drivers series.
It is recommended to install the
    nvidia-driver
package.

nvidia-smi:

root@guest:~# nvidia-smi
No devices were found

I installed the nvidia-driver with the command

apt install nvidia-driver nvidia-kernel-dkms

This is the nvidia-driver I have installed

root@guest:~# apt show nvidia-driver
Package: nvidia-driver
Version: 550.163.01-2
Priority: optional
Section: non-free/x11
Source: nvidia-graphics-drivers
Maintainer: Debian NVIDIA Maintainers <pkg-nvidia-devel@lists.alioth.debian.org>
Installed-Size: 1.478 kB
Provides: nvidia-driver-any, nvidia-glx-any
Pre-Depends: nvidia-installer-cleanup, nvidia-legacy-check (>= 495)
Depends: nvidia-driver-libs (= 550.163.01-2), nvidia-driver-bin (= 550.163.01-2), xserver-xorg-video-nvidia (= 550.163.01-2), nvidia-vdpau-driver (= 550.163.01-2), nvidia-alternative (= 550.16>
Recommends: nvidia-settings (>= 550), libnvidia-cfg1 (= 550.163.01-2), nvidia-persistenced (>= 550)
Suggests: nvidia-kernel-dkms (>= 550.163.01) | nvidia-kernel-source (>= 550.163.01) | nvidia-open-kernel-source (>= 550.163.01)
Homepage: https://www.nvidia.com
Download-Size: 531 kB
APT-Manual-Installed: yes
APT-Sources: http://deb.debian.org/debian trixie/non-free amd64 Packages
Description: NVIDIA metapackage
 This metapackage depends on the NVIDIA binary driver and libraries
 that provide optimized hardware acceleration of
 OpenGL/GLX/EGL/GLES/Vulkan applications via a direct-rendering X Server.
 .
 Please see the nvidia-kernel-dkms
 (nvidia-open-kernel-dkms) or nvidia-kernel-source
 (nvidia-open-kernel-source) packages
 for building the kernel module required by this package.
 This will provide nvidia-kernel-550.163.01
 (nvidia-open-kernel-550.163.01).
 .
 This version only supports GeForce, NVS, Quadro, RTX, Tesla, ... GPUs based on
 the Maxwell, Pascal, Volta, Turing, Ampere, Ada, Hopper or newer architectures.
 Look at the legacy driver or Tesla driver packages for older cards.
 .
 See /usr/share/doc/nvidia-driver/README.txt.gz
 for a complete list of supported GPUs and PCI IDs.
 .
 Building the kernel module has been tested up to Linux 6.15.