Vfio passthrough to ubuntu20.04 guest GeForce RTX 2070 - 440.100 - RmInitAdapter failed! (0x23:0x56:515)

Graphics / Linux Linux
1

Hi,

I have a host machine, Hardware name: System manufacturer System Product Name/ROG STRIX Z390-E GAMING, BIOS 1502 02/21/2020, bios enabled VT-d, 5.8.0 mainline kernel with vfio-pci modules loaded for the host nvidia card,

I’m running the host kernel on the inbuilt intel graphics adapter.

I’m created a ubuntu-20.04 amd64 host-passthrough guest. The guest kernel is failing to load the driver. I’m not sure where to go from here.

[    3.537108] [drm] [nvidia-drm] [GPU ID 0x00000700] Loading driver
[    3.537110] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:07:00.0 on minor 0
[    3.561282] nvidia-uvm: Loaded the UVM driver, major device number 236.
[    4.100322] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input5
[    4.100356] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input6
[    4.100393] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input7
[    4.100423] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input8
[    5.059112] audit: type=1400 audit(1597567787.684:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=647 comm="apparmor_parser"
[    5.059231] audit: type=1400 audit(1597567787.684:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ippusbxd" pid=651 comm="apparmor_parser"
[    5.059418] audit: type=1400 audit(1597567787.684:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=649 comm="apparmor_parser"
[    5.059420] audit: type=1400 audit(1597567787.684:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=649 comm="ap
parmor_parser"
[    5.059421] audit: type=1400 audit(1597567787.684:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=650 comm="apparmor_parser"
[    5.059422] audit: type=1400 audit(1597567787.684:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=650 comm="apparmor_parser"
[    5.059423] audit: type=1400 audit(1597567787.684:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=650 comm="apparmor_parser"
[    5.060042] audit: type=1400 audit(1597567787.684:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=653 comm="apparmor_parser"
[    5.060044] audit: type=1400 audit(1597567787.684:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=653 comm="apparmor_parser"
[    5.332680] kauditd_printk_skb: 34 callbacks suppressed
[    5.332681] audit: type=1400 audit(1597567787.956:45): apparmor="DENIED" operation="capable" profile="/usr/sbin/cups-browsed" pid=781 comm="cups-browsed" capability=23  capname="sys_nice"
[    6.146010] Lockdown: Xorg: raw io port access is restricted; see man kernel_lockdown.7
[    6.179765] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:515)
[    6.179845] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[    6.225947] Lockdown: Xorg: raw io port access is restricted; see man kernel_lockdown.7
[    6.296582] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:515)
[    6.296671] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0

nvidia-bug-report.log.gz (93.6 KB)
libvirt definition of guest ubunt2004.xml.gz (1.6 KB)
iommu groups on host (2.4 KB)

2

Small correction to above, I was using secure boot:

virsh dumpxml ubuntu20.04  | grep CODE
    <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>

After extracting rom:

# echo 1 >/sys/bus/pci/devices/0000\:01\:00.0/rom 
# cat /sys/bus/pci/devices/0000\:01\:00.0/rom  > ~dan/nvidia-bios-extracted.rom
# file ~dan/nvidia-bios-extracted.rom
/home/dan/nvidia-bios-extracted.rom: BIOS (ia32) ROM Ext. IBM comp. Video (117*512)
# echo 0 >/sys/bus/pci/devices/0000\:01\:00.0/rom
# mv ~dan/nvidia-bios-extracted.rom /var/lib/libvirt/vbios/

Fixing selinux permissions so qemu can load it:

#  semanage fcontext -a -t virt_image_t '/var/lib/libvirt/vbios(/.*)?'
# restorecon -R -v /var/lib/libvirt/vbios/

And placing a “rom bar” for the nvidia device in the xml

# virsh dumpxml ubuntu20.04  | grep -C 3 'rom bar'
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom bar='on' file='/var/lib/libvirt/vbios/nvidia-bios-extracted.rom'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>

I’ve got a working driver and screen. After some menial setup like mice and keyboard I should be right.