nvidia gtx1060 kvm passthrough

Graphics / Linux Linux
1

Hi,

I’m currently trying to setup a passthrough of my nvidia gtx 1060 mobile (from a dell inspiron 7000 series) to a windows guest.

lspci

01:00.0 VGA compatible controller: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile 3GB] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau

I enabled iommu :

dmesg|grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=0b58f681-aba7-4bdd-983a-c3fb9e69201e rw cryptdevice=/dev/nvme0n1p2:cryptroot cryptkey=rootfs:/crypto/root.key intel_iommu=on iommu=pt nouveau.blacklist=1 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=0b58f681-aba7-4bdd-983a-c3fb9e69201e rw cryptdevice=/dev/nvme0n1p2:cryptroot cryptkey=rootfs:/crypto/root.key intel_iommu=on iommu=pt nouveau.blacklist=1 quiet
[    0.000000] DMAR: IOMMU enabled

and here are the iommu group where my gtx belong

IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile 3GB] [10de:1c20] (rev a1)

I added the devices to the vfio.conf in modules configurations.

Using virt-manager I can see my gpu in the available pci devices. but booting my vm with no vga I just get no screen on my virtual machine. when I use the qxl video driver I can see my gpu using GPU-Z inside the host, but the nvidia driver won’t install and my card is seen as a basic microsoft video card.

I want to know if there as workaround to do to perform this passthrough correctly, or if there is known bug that I should be aware of.

any help would be welcome, please let me know if I can provide more usefull information

2

Do you get a picture on the monitor that is connected to the nvidia card using the ms driver?

3

See this for Optimus quirks: [url]https://gist.github.com/Misairu-G/616f7b2756c488148b7309addc940b28[/url]

4

I tried to plug a monitor on the hdmi port (which I guess is managed by the nvidia) with no result… but in fact I also have this issue outside of my virtual machine : whenever I plugged an hdmi cable its never detected.
I can only plug monitor on hdmi with an usb-c adapter… do you think both issues can be related ?

I think that usb-c and my computer monitor are managed by my intel graphic chipset, and hdmi managed by nvidia. But I don’t know how to verify this “theory”

5

thanks for the link generix, will have a look at it and come back if this solve my issues(or not)

6

Hi, after following the steps from your tutorial(with some adaptation as I’m not on ubuntu) I did not manage to successfully boot my virtual machine.

I progressed a little bit : I can now use my gpu on my host and binding/unbinding the vfio driver.

the verification step after the bumblebee installation :

cat /proc/acpi/bbswitch 
0000:01:00.0 OFF
 
optirun cat /proc/acpi/bbswitch 
0000:01:00.0 ON

optirun nvidia-smi
Mon Feb 12 14:24:39 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   42C    P0    22W /  N/A |      7MiB /  6078MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1494      G   /usr/lib/xorg-server/Xorg                      5MiB |
+-----------------------------------------------------------------------------+

here is the script I use to launch my virtual machine

echo "[*] binding dGPU to vfio driver"
echo "10de 1c20" > "/sys/bus/pci/drivers/vfio-pci/new_id"

echo "[*] enabling huge pages"
echo 5000 |tee /proc/sys/vm/nr_hugepages

echo "[*] launching VM"
qemu-system-x86_64 \
  -name "Windows10-QEMU" \
  -machine type=q35,accel=kvm \
  -global ICH9-LPC.disable_s3=1 \
  -global ICH9-LPC.disable_s4=1 \
  -cpu host,kvm=off,hv_vapic,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vendor_id=12alphanum \
  -smp 8,sockets=1,cores=4,threads=2 \
  -m 8G \
  -mem-path /dev/hugepages \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vnc 127.0.0.1:1 \
  -device qxl,bus=pcie.0,addr=1c.2 \
  -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -k en-us \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,x-pci-sub-device-id=1969,x-pci-sub-vendor-id=4136,multifunction=on,romfile=vbios.rom,x-vga=true \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/ovmf/ovmf_code_x64.bin  \
  -drive if=pflash,format=raw,file=WIN_VARS.fd \
  -boot menu=on \
  -boot order=c \
  -drive id=disk0,if=virtio,cache=none,format=raw,file=win10-gaming.img \
  -drive file=Win10_1709_French_x64.iso,media=cdrom,index=1 \
  -drive file=virtio-win.iso,media=cdrom,index=2 \
  -net nic -net user,smb=/shared/

the vm boot without any errors, no logs are created. But when I connect to the host using vncviewer I always get the message “guest has not initialized the display (yet)”.

One fact is that I did not manage to dump my vbios (input/output error). i’m using a vbios from a msi gtx1060 mobile, the device vendor and system id are the same than on my dGPU, but the subsystem ID aren’t the same. I did not manage to find the vbios of the gtx1060 mobile with dell subsystem :

optirun lspci -nnk -s 01:00.0
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile 3GB] [10de:1c20] (rev a1)
	Subsystem: Dell GP106M [GeForce GTX 1060 Mobile] [1028:0802]
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

I have no idea what I can try now… I really want to make this passthrough work. I have no idea how to debug this as qemu don’t show me any errors or warning. Any help is welcome

7

Why not try with a Linux guest with enabled sshd first? Easier to debug if there’s a problem with vbios.