Can't passthrough a quadro p2200 to a Ubuntu 20 LTS server VM in proxmox

I put together a new home server using old threadripper parts and an interesting X399 board from Asrock that included dual 10g networking and an IPMI equivalent. Hardware:

  • TR 1920x
  • ASRock Rack X399D8A-2T
  • Seasonic 800w focus plus
  • 8xSAS drives ( Seagate Constellation E3 3TB)
  • 2xSamsung 970 Evo plus nvme m.2 (mirror 500GB )
  • 1xSamsung 860 Evo SATA ( Cache )
  • LSI SAS 9210-8i
  • PNY Quadro P2200

Installed Proxmox, created a TrueNas Core VM, passthrough the LSI controller and the small SSD created another zfs pool over there and created network shares to communicate, so far so good.

Then I followed “Craft Computing Youtube Guide on This” IDK if links are permitted here, so I will not link, but searching for plex on proxmox is likely to find you his video. I followed these steps.

  • Created Ubuntu server LTS 20 VM
  • Enabled repository updating as per Plex Guide
  • Installed Plex
  • Added libraries from the share

— GPU passthrough Journey begins here —

ON HOST (Proxmox)

  • Included “quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off” /etc/kernel/cmdline ( I use systemd not grub )
  • proxmox-boot-tool refresh
  • included the following in /etc/modules:
    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd
  • IOMMU interrupt remaping:
    echo “options vfio_iommu_type1 allow_unsafe_interrupts=1” > /etc/modprobe.d/iommu_unsafe_interrupts.conf
    echo “options kvm ignore_msrs=1” > /etc/modprobe.d/kvm.conf
  • Blacklist drivers
    echo “blacklist radeon” >> /etc/modprobe.d/blacklist.conf
    echo “blacklist nouveau” >> /etc/modprobe.d/blacklist.conf
    echo “blacklist nvidia” >> /etc/modprobe.d/blacklist.conf
  • reboot
  • added GPU to VFIO ( I made sure the I got the correct Id’s)
    echo “options vfio-pci ids=10de:1c31,10de:10f1 disable_vga=1”> /etc/modprobe.d/vfio.conf
  • update-initramfs -u
    -reboot

ON GUEST ( Ubuntu LTS 20.04.2 live server amd64 )

  • Download nVidia Drivers
    sudo apt update
    wget https : // international.download.nvidia.com /XFree86/Linux-x86_64 /###.##.##/NVIDIA-Linux-x86_64- ###.##.## . run
    sudo chmod +x NVIDIA-Linux-x86_64-###.##.##.run
  • Disable Nouveau drivers in kernel-
    sudo bash -c “echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf”
    sudo bash -c “echo options nouveau modset=0 >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf”
    sudo update-initramfs -u
  • reboot

ON HOST (PROXMOX)

  • Hide VM identifiers from Nvidia
    SSH into ProxMox
    cd /etc/PvE/qemu-server
    nano ###.conf (# is the VM identifier of your Plex server)
    Modify cpu line…
    cpu: host,hidden=1
    Save file and exit.
    Start Plex VM.
    Nvidia-smi to confirm GPU is working.

—THE PROBLEM—

At this lspci -v inside the VM returns the PCIe device and returns the correct driver in use, but…

nvidia-msi
No devices Found

I’ve also changed the bios to UEFI ( OMVF )
and turned off SR-IOV on bare metal bios, since I read this was a good idea because I’m passing through a single GPU

At this point, I don’t know if nvdia-msi is a good testing tool or if there are better ways to make sure that it is working

If there are other commands or config files I should look like pls let me know and how to test if the gpu is working or not.

My Plex VM stops Playing videos that need transcoding if I enable HW accel.

Finally, after 10 min of a black screen, it gave up and started transcoding on the CPU with this error message.
[Transcode] [FFMPEG] - → CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected

I just wanted to make an update. I was finally able to solve the GPU passthrough issue. The last thing I needed for it to work was to change the primary graphics setting in bios from PCIe to internal VGA; now this disables my ability to connect a monitor to the server, but I can still access the bios screen if needed and thanks to it being proxmox, I can access the console in the web UI directly so no issues there. But yeah, passthrough is working now without any problems.