GTX 1080 & KVM PCI passthrough to guest

Accelerated Computing CUDA CUDA Setup and Installation
1

I’m trying to passthrough GTX 1080 to instance (virtual machine) under KVM (qemu).

My command line (for qemu) is:

/usr/bin/qemu-system-x86_64 -name test -S -machine pc-i440fx-trusty,accel=kvm,usb=off -cpu host -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a86d92f8-53e4-4014-b9e2-b6170a52d608 -smbios type=1,serial=4c4c4544-0058-3410-8057-c6c04f573032,uuid=a86d92f8-53e4-4014-b9e2-b6170a52d608 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/a86d92f8-53e4-4014-b9e2-b6170a52d608.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/a86d92f8-53e4-4014-b9e2-b6170a52d608/disk,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:85:43:44,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/a86d92f8-53e4-4014-b9e2-b6170a52d608/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device pci-assign,configfd=26,host=42:00.0,id=hostdev0,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

Device visible on host:
42:00.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 119e
Flags: bus master, fast devsel, latency 0, IRQ 72
Memory at d4000000 (32-bit, non-prefetchable)
Memory at 3bff0000000 (64-bit, prefetchable)
Memory at 3bfee000000 (64-bit, prefetchable)
I/O ports at d000
Expansion ROM at d5000000 [disabled]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] #19
Kernel driver in use: pci-stub

I can see device inside virtual machine:

00:05.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1)

and I can install/compile driver (tried both 367.35 & 370.23, .run blobs from nvidia.com). They install, but not recognized by nvidia-smi:

nvidia-smi

Unable to determine the device handle for GPU 0000:00:05.0: Unknown Error

dmesg shows:

[ 1093.280138] nvidia 0000:00:05.0: irq 45 for MSI/MSI-X
[ 1093.303496] NVRM: RmInitAdapter failed! (0x23:0x56:451)
[ 1093.303503] NVRM: rm_init_adapter failed for device bearing minor number 0
[ 1093.304132] nvidia 0000:00:05.0: irq 45 for MSI/MSI-X
[ 1093.335032] NVRM: RmInitAdapter failed! (0x23:0x56:451)
[ 1093.335037] NVRM: rm_init_adapter failed for device bearing minor number 0

nouveau blacklisted, nvidia & nvidia-uvm loaded:

lsmod |egrep ‘nvidia|nou’

nvidia_uvm 710792 0
nvidia_drm 14707 0
nvidia_modeset 773069 1 nvidia_drm
nvidia 11908960 2 nvidia_modeset,nvidia_uvm
drm 303102 3 ttm,drm_kms_helper,nvidia_drm

guest cmdline: BOOT_IMAGE=/boot/vmlinuz-3.13.0-93-generic root=LABEL=cloudimg-rootfs ro nofb nomodeset vga=normal console=tty0 console=ttyS0,115200 no_timer_check nofb nomodeset vga=normal

host cmdline:
initrd=initrd.gz intel_iommu=on boot=live config nosplash nopersistent noprompt noautologin quickreboot BOOT_IMAGE=vmlinuz (some pxe options omitted).

I’ve tried it on Ubuntu 16.04 (linux 4.6) and ubuntu 14.04 (linux 3.13) - same behavior.

Any idea how to configure qemu/libvirt?

2

I don’t think so GTX 1080 is supported for passthrough. May you need to use Qaudro GPU. Please check with OS vendor is this supported config?

3

we recently acquired a small development machine running an Asus X99/Deluxe-II, with a GTX1080 GPU…we were going to try and build some GPU enabled appliances (both linux and windows vm’s)…

the mobo supports all the needed Intel Vt-x extensions, and in fact we have linux (centos, ubuntu) and windows (v7 & 10) base installations that run with the card seemingly fine (although not done much with them yet)…

Is the 1080 a non-starter here? should we move to a Quadro card? disappointing, since the 1080 is quite the piece of hardware.

4

You shouldn’t have any trouble using the GTX 1080 in the base OS.
As already stated, it is not supported for passthrough into a VM, and I don’t think you’ll get that working.

5

well thats disappointing.

which way should we go - a GRID type card, or one of the older GeForce cards that are supported, to get virtualized guests to work with passthrough hardware?

I’ll do some research on both, but anything you all could share would be great context.

6

Txbox is wrong, and I got it to work.

7

Imgur: The magic of the Internet Either Nvidia is censoring/deleting it/or something is wrong with my account. Hopefully you have email notifications on and will get this.

8

Hey kvasko -
That sounds like a lot of craziness…we are just trying to use their stong hardware…

What very driver are you using?

Are you running both linux and windows guests in KVM?

9

Hey kvasko -
That sounds like a lot of craziness…we are just trying to use their stong hardware…

What very driver are you using?

Are you running both linux and windows guests in KVM?

10

Completely understand, me too :).

I’m only running on Linux (host and guest) currently. I’m using the latest NVidia driver on my VM guests (you shouldn’t have the driver installed on the host, but be using vfio-pci or pci-stub as the driver on the host).

Use the following guide to get started https://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/. Key steps, 1) Make sure you are blacklisting the GPUs on the host correctly (e.g. make sure the GPU is driver is bound to pci-stub (if newer system vfio-pci)). You can check this with “lspci -vnnn | grep -i driver” if it shows that pci-stub is in use (and not NVidia or nouveau) you should be good to go on doing the passthrough to KVM. 2) create your kvm guest VM, at that point you should have a configuration file for the VM, edit that config and make sure to change the -cpu flag to something like “-cpu ,kvm=no”. For example mine was “-cpu Haswell-NoTSX,kvm=no”.

11

Interesting…we are new to kvm - is that option turning off both processor AND chipset virtualization detection?

12

From my understanding instead of the host passing to the VM information so the guest VM can tell that it is in a VM, the option disables that and there is no way for the VM to tell it isn’t running on “baremetal”.

See this.

https://www.redhat.com/archives/libvir-list/2014-August/msg00512.html

13

ah yes we are in violent agreement - I should not have said detection, but “exposure”

I read the post about the cpu flag - very interesting.

thanks again for sharing

we will be getting after it probably next week!