(EE) no screens found(EE) whyle trying to enable the nvidia driver on Debian 11 emulated with bhyve

Hello.

I’m trying to enable the NVIDIA driver in Debian 11,that I have emulated with bhyve on FreeBSD 13r. This is the error I get while I’m logged inside Debian using ssh from FreeBSD :

root@marietto:~# startx


X.Org X Server 1.20.11
X Protocol Version 11, Revision 0
Build Operating System: linux Debian
Current Operating System: Linux marietto 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.10.0-8-amd64 root=UUID=c25a24a3-12f2-4653-a275-9f43924d6c3b ro quiet
Build Date: 13 April 2021  04:07:31PM
xorg-server 2:1.20.11-1 (https://www.debian.org/support) 
Current version of pixman: 0.40.0
    Before reporting problems, check http://wiki.x.org
    to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
    (++) from command line, (!!) notice, (II) informational,
    (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Sun Sep  5 15:48:08 2021
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using config directory: "/etc/X11/xorg.conf.d"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
(EE) 
Fatal server error:
(EE) no screens found(EE) 
(EE) 
Please consult the The X.Org Foundation support 
     at http://wiki.x.org
 for help. 
(EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
(EE) 
(EE) Server terminated with error (1). Closing log file.
^Cxinit: giving up
xinit: unable to connect to X server: Connection refused
xinit: unexpected signal 2

This is what lspci says :

root@marietto:~# lspci

00:00.0 Host bridge: Network Appliance Corporation Device 1275
00:01.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:02.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
00:02.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
00:02.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
00:02.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)
00:06.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:1e.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
00:1f.0 ISA bridge: Intel Corporation Z390 Chipset LPC/eSPI Controller (rev 10)

as u can see the nvidia graphic card is passed through correctly from FreeBSD to Debian 11.

What I want to appurate is the cause of the problem. If it belongs to bhyve or to a wrong configuration of debian 11 / Xorg / or the graphic driver. Anyway,debian 11 is able to detect correctly the graphic card :

root@marietto:~# nvidia-detect

Detected NVIDIA GPUs:
00:02.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)

Checking card:  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
Your card is supported by the default drivers.
Your card is also supported by the Tesla 460 drivers series.
Your card is also supported by the Tesla 450 drivers series.
Your card is also supported by the Tesla 418 drivers series.
It is recommended to install the nvidia-driver package.

but :

root@marietto:~# nvidia-smi
No devices were found

What I did to get that output ? I’ve followed this guide :

https://wiki.debian.org/NvidiaGraphicsDrivers#Tesla_Drivers

So,I did the following :

root@marietto:~# install nvidia-driver
root@marietto:~# nvidia-xconfig

and it has created a xorg.conf file like this :

# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig:  version 460.32.03


Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    Screen      1  "Screen0 (1)" RightOf "Screen0"
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
EndSection

Section "Files"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Unknown"
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:0:2:0"
    Screen          0
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Screen0 (1)"
    Device         "Device0 (1)"
    Monitor        "Monitor0"
    DefaultDepth    24
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

What else ? just ask. I really want to use my nvidia graphic card from within debian 11 os installed with bhyve. Take in consideration that it works if I use the nouveau driver. yes,my monitor,attached to the HDMI port of the graphic card turns on if I use the nouveau driver but it doesn’t if I use the nvidia driver. (I even tried with the proprietary nvidia driver)

dmesg :

[   14.697445] NVRM: GPU 0000:00:02.0: RmInitAdapter failed! (0x24:0x65:1248)

[    1.948773] nvidia: loading out-of-tree module taints kernel.
[    1.948781] nvidia: module license 'NVIDIA' taints kernel.
[    1.956323] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    1.967795] nvidia-nvlink: Nvlink Core is being initialized, major device number 248
[    1.968329] nvidia 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    2.112279] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  460.91.03  Fri Jul  2 05:43:38 UTC 2021
[    2.123865] audit: type=1400 audit(1630855752.267:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=383 comm="apparmor_parser"
[    2.123868] audit: type=1400 audit(1630855752.267:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=383 comm="apparmor_parser"
[    2.204737] [drm] [nvidia-drm] [GPU ID 0x00000002] Loading driver
[    2.204739] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:00:02.0 on minor 0

[    0.847082] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[    0.847082] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.968329] nvidia 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem

I don’t think this will work with bhyve and any other guest OS than BSD.
AFAIK, bhyve is missing the option to hide the hypervisor from the guest which is necessary to install the Linux or Windows driver on VMs that are unsupported.
https://smartos.topicbox.com/groups/smartos-discuss/Ta62589718c26804e
Please also check the slides from this article for some additional problems:
https://www.phoronix.com/scan.php?page=news_item&px=Bhyve-GPU-Passthrough-2019

I’ve modified the bhyve source code,I’ve added a bit and I’ve hidden the hypervisor to the nvidia driver for windows 10. I’ve also installed RwEverything in my Windows VM, I’ve opened the command window, typed in “cpuid 1” and checked that ECX bit 32 was set…and yes,as expected, ecx bit 31 was unset. But this is not enough. It still does not work. what else is missing ? In addition,I read that it wasn’t even necessary,because since I’m running windows 10,nvidia made available the working drivers to enable the passthru for this OS,but despite this,it is not working here,on FreeBSD with bhyve.

Please post the output of
sudo lspci -d “10de:*” -v -xxx

root@marietto-BHYVE:~# sudo lspci -d “10de:*” -v -xxx

00:03.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ZOTAC International (MCO) Ltd. TU102 [GeForce RTX 2080 Ti]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at d2000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (32-bit, prefetchable) [size=256M]
Memory at d0000000 (32-bit, prefetchable) [size=32M]
I/O ports at 2080 [size=128]
Expansion ROM at d4100000 [virtual] [disabled] [size=1M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
00: de 10 04 1e 07 00 10 00 a1 00 00 03 00 00 00 00
10: 00 00 00 d2 08 00 00 c0 00 00 00 00 08 00 00 d0
20: 00 00 00 00 81 20 00 00 00 00 00 00 da 19 03 25
30: 00 00 00 00 60 00 00 00 00 00 00 00 0b 01 00 00
40: da 19 03 25 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 01 00 00 00 ce d6 23 00 00 00 00 00
60: 01 68 c3 49 08 00 00 00 05 78 80 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 10 00 12 00 e1 8d 2c 11
80: 10 29 00 00 03 3d 45 00 43 01 81 10 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 13 00 04 00
a0: 00 00 00 00 0e 00 00 00 03 00 1f 00 00 00 00 00
b0: 00 00 00 00 09 00 14 01 00 00 10 80 00 00 00 00
c0: 03 00 00 00 00 00 00 00 11 00 05 00 00 00 b9 00
d0: 00 00 ba 00 00 00 00 00 00 00 00 00 da 19 03 25
e0: da 19 03 25 03 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:04.0 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at d4050000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00: de 10 f7 10 07 00 18 00 a1 00 03 04 10 00 00 00
10: 00 00 05 d4 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 da 19 03 25
30: 00 00 00 00 60 00 00 00 00 00 00 00 0a 02 00 00
40: da 19 03 25 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 ce d6 23 00 00 00 00 00
60: 01 68 03 00 08 00 00 00 05 78 80 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 10 00 02 00 e1 8d 2c 01
80: 10 29 09 00 03 3d 45 00 43 01 81 10 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 13 00 04 00
a0: 00 00 00 00 0e 00 00 00 00 00 01 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:05.0 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI])
Subsystem: ZOTAC International (MCO) Ltd. TU102 USB 3.1 Host Controller
Flags: fast devsel, IRQ 39
Memory at d4000000 (32-bit, prefetchable) [size=256K]
Memory at d4040000 (32-bit, prefetchable) [size=64K]
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [b4] Power Management version 3
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
00: de 10 d6 1a 13 04 10 00 a1 30 03 0c 10 00 00 00
10: 08 00 00 d4 00 00 00 00 00 00 00 00 08 00 04 d4
20: 00 00 00 00 00 00 00 00 00 00 00 00 da 19 03 25
30: 00 00 00 00 68 00 00 00 00 00 00 00 0b 03 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 31 60 00 00 00 00 00 00 05 78 81 00 00 30 e0 fe
70: 00 00 00 00 24 00 00 00 10 b4 02 00 e0 8d 2c 01
80: 10 29 0b 00 03 3d 45 00 43 00 81 10 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 13 00 04 00
a0: 00 00 00 00 0e 00 00 00 00 00 01 00 00 00 00 00
b0: 00 00 00 00 01 00 43 48 0b 01 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:06.0 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 USB Type-C UCSI Controller
Flags: bus master, fast devsel, latency 0, IRQ 37
Memory at d4058000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [b4] Power Management version 3
Kernel driver in use: nvidia-gpu
Kernel modules: i2c_nvidia_gpu
00: de 10 d7 1a 07 04 10 00 a1 00 80 0c 10 00 00 00
10: 00 80 05 d4 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 da 19 03 25
30: 00 00 00 00 68 00 00 00 00 00 00 00 0a 04 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 05 78 81 00 00 10 e0 fe
70: 00 00 00 00 24 00 00 00 10 b4 02 00 e0 8d 2c 01
80: 10 29 09 00 03 3d 45 00 43 00 81 10 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 13 00 04 00
a0: 00 00 00 00 0e 00 00 00 00 00 01 00 00 00 00 00
b0: 00 00 00 00 01 00 43 48 0b 01 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The topology of the devices isn’t really usable, I guess. Apart from sitting at the root bus, the sub-devices are no subfunctions but own devices.

what you suggest me to do ? Take in consideration that it works when I pass it through on Linux with qemu and kvm. What’s the magic trick under the hood of qemu + kvm ? It is not a special graphic card,eh. It’s a card that has been bought by many people. Tell me what kind of solutions on the bhyve source code I could try to implement.

How is the topology on the host BSD OS?
I guess I’d first try to only pass through the gpu function and see where it gets me.

can you send me your email address to my email address (marietto2008@gmail.com) ? I’m working on this issue with some developers. I would like to add you in the discussion. thanks.

on FreeBSD host :

bhyve -S -c 4 -m 8G -w -H
-s 0,hostbridge
-s 1,ahci-cd,/home/marietto/Downloads/OS/ubuntu-21.04-desktop-amd64.iso
-s 2,virtio-blk,/mnt/da3s1/OS/bhyve/ub2104.img
-s 3,passthru,1/0/0,rom=TU102.rom
#-s 4,passthru,1/0/1
#-s 5,passthru,1/0/2
#-s 6,passthru,1/0/3
-s 8,virtio-net,tap0
-s 30,xhci,tablet
-s 31,lpc
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900,wait
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
vm0

on Ubuntu guest :

root@marietto-BHYVE:~# sudo lspci -d “10de:*” -v -xxx

lspci: -d: Invalid vendor ID
root@marietto-BHYVE:~# sudo lspci -d “10de:*” -v -xxx
00:03.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ZOTAC International (MCO) Ltd. TU102 [GeForce RTX 2080 Ti]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at d2000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (32-bit, prefetchable) [size=256M]
Memory at d0000000 (32-bit, prefetchable) [size=32M]
I/O ports at 2080 [size=128]
Expansion ROM at d4100000 [virtual] [disabled] [size=1M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
00: de 10 04 1e 07 00 10 00 a1 00 00 03 00 00 00 00
10: 00 00 00 d2 08 00 00 c0 00 00 00 00 08 00 00 d0
20: 00 00 00 00 81 20 00 00 00 00 00 00 da 19 03 25
30: 00 00 00 00 60 00 00 00 00 00 00 00 0b 01 00 00
40: da 19 03 25 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 01 00 00 00 ce d6 23 00 00 00 00 00
60: 01 68 c3 49 08 00 00 00 05 78 80 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 10 00 12 00 e1 8d 2c 11
80: 10 29 00 00 03 3d 45 00 43 01 81 10 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 13 00 04 00
a0: 00 00 00 00 0e 00 00 00 03 00 1f 00 00 00 00 00
b0: 00 00 00 00 09 00 14 01 00 00 10 80 00 00 00 00
c0: 03 00 00 00 00 00 00 00 11 00 05 00 00 00 b9 00
d0: 00 00 ba 00 00 00 00 00 00 00 00 00 da 19 03 25
e0: da 19 03 25 03 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

dmesg :

[ 30.348911] nvidia: loading out-of-tree module taints kernel.
[ 30.348921] nvidia: module license ‘NVIDIA’ taints kernel.
[ 30.348922] Disabling lock debugging due to kernel taint
[ 30.365128] nvidia-nvlink: Nvlink Core is being initialized, major device number 236

[ 30.365987] nvidia 0000:00:03.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 30.482659] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 470.74 Mon Sep 13 23:09:15 UTC 2021
[ 31.293963] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 470.74 Mon Sep 13 22:59:50 UTC 2021
[ 33.352258] [drm] [nvidia-drm] [GPU ID 0x00000003] Loading driver
[ 33.352260] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:00:03.0 on minor 0
[ 39.051832] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
[ 39.054079] nvidia-uvm: Loaded the UVM driver, major device number 234.
[ 52.849581] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 52.849811] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 56.965416] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 56.965631] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 67.893602] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 67.893842] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 72.009630] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 72.009853] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 76.125596] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 76.125831] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 80.242953] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 80.243185] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 89.082584] rfkill: input handler disabled
[ 98.906276] rfkill: input handler enabled
[ 103.118662] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 103.118887] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 107.236046] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 107.236268] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 117.318192] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 117.318409] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[ 121.434584] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[ 121.434846] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0

root@marietto:/home/marietto/Desktop/bhyve # ./os-uefi-ub2104.sh

VM:vm0 is not created.
fbuf frame buffer base: 0xa54a00000 [sz 16777216]

1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
rdmsr to register 0x140 on vcpu 0
wrmsr to register 0x140(0) on vcpu 0
rdmsr to register 0x140 on vcpu 1
wrmsr to register 0x140(0) on vcpu 1
rdmsr to register 0x140 on vcpu 2
wrmsr to register 0x140(0) on vcpu 2
rdmsr to register 0x140 on vcpu 3
wrmsr to register 0x140(0) on vcpu 3
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) unmap
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) unmap
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) unmap
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) unmap
1/0/0 modify_mmio 0x0000000096000000 → 0x00000000d2000000 (size 880000) map
1/0/0 modify_mmio 0x0000000096881000 → 0x00000000d2881000 (size 77f000) map
1/0/0 modify_mmio 0x0000000060000000 → 0x00000000c0000000 (size 10000000) map
1/0/0 modify_mmio 0x0000000094000000 → 0x00000000d0000000 (size 2000000) map
rdmsr to register 0x64e on vcpu 3
rdmsr to register 0x34 on vcpu 3
wrmsr to register 0xe01(0x2000001f) on vcpu 0
wrmsr to register 0xe01(0x2000001f) on vcpu 0
wrmsr to register 0xe01(0x2000001f) on vcpu 1
wrmsr to register 0xe01(0x2000001f) on vcpu 1
wrmsr to register 0xe01(0x2000001f) on vcpu 2
wrmsr to register 0xe01(0x2000001f) on vcpu 2
wrmsr to register 0xe01(0x2000001f) on vcpu 3
wrmsr to register 0xe01(0x2000001f) on vcpu 3
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88
rdmsr to register 0x3fc on vcpu 1
rdmsr to register 0x3fd on vcpu 1
rdmsr to register 0x3fe on vcpu 1
rdmsr to register 0x60d on vcpu 1
rdmsr to register 0x3f8 on vcpu 1
rdmsr to register 0x3f9 on vcpu 1
rdmsr to register 0x3fa on vcpu 1
rdmsr to register 0x630 on vcpu 1
rdmsr to register 0x631 on vcpu 1
rdmsr to register 0x632 on vcpu 1
rdmsr to register 0x64d on vcpu 0
rdmsr to register 0x64d on vcpu 0

Glancing at the bhyve manual, to properly passthrough the sub-functions,
-s 3:0,passthru,1/0/0
-s 3:1,passthru,1/0/1
-s 3:2,passthru,1/0/2
-s 3:3,passthru,1/0/3
should be used.
Though that shouldn’t matter since the main gpu function alone doesn’t allow the driver to init.
Please set the nvidia module option
NVreg_ResmanDebugLevel=0
in the guest to maybe get some more info.

the bhyve command that I have used are the following :

bhyve -S -c 4 -m 8G -w -H \
-s 0,hostbridge \
#-s 1,ahci-cd,/home/marietto/Downloads/OS/ubuntu-21.04-desktop-amd64.iso \
-s 2,virtio-blk,/mnt/da3s1/OS/bhyve/ub2104.img \
-s 3:0,passthru,1/0/0 \
-s 3:1,passthru,1/0/1 \
-s 3:2,passthru,1/0/2 \
-s 3:3,passthru,1/0/3 \
-s 8,virtio-net,tap0 \                   
-s 30,xhci,tablet \
-s 31,lpc \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900,wait \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \

here I have used the acso patch and u can see how is the dmesg logs produced adding on /etc/modprobe.d/nvidia.conf the line options nvidia NVreg_ResmanDebugLevel=0

https://pastebin.ubuntu.com/p/fGS8JN87xk/

On debian 11 without the acso patch :

https://pastebin.ubuntu.com/p/67g5zbpYzf/

In the log from debian 11 without patch, the driver seems to load fine, no errors.

NVRM: probing 0x10de 0x1e04, class 0x30000
[ 20.098164] nvidia 0000:00:03.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 20.098232] NVRM: PCI:0000:00:03.0 (10de:1e04): BAR0 @ 0xd2000000 (16MB)
[ 20.099092] NVRM: PCI:0000:00:03.0 (10de:1e04): BAR1 @ 0xc0000000 (256MB)
[ 20.216355] NVRM: PBI is not supported for GPU 0000:00:03.0
[ 20.217091] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 470.57.02 Tue Jul 13 16:14:05 UTC 2021
[ 20.365496] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 470.57.02 Tue Jul 13 16:06:24 UTC 2021
[ 20.399966] [drm:nv_drm_probe_devices [nvidia_drm]] Registering device for NVIDIA GPU ID 0x083
[ 20.400097] [drm:drm_minor_register [drm]]
[ 20.400535] [drm:drm_minor_register [drm]] new minor registered 128
[ 20.400576] [drm:drm_minor_register [drm]]
[ 20.400900] [drm:drm_minor_register [drm]] new minor registered 0
[ 20.400904] [drm] [nvidia-drm] [GPU ID 0x00000003] Loading driver
[ 20.400909] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:00:03.0 on minor 0

Using ACS override in a guest is pretty much senseless anyway.

I know,debian fooled me too. It seems everything ok but it does not work. When I do “nvidia-smi” the usual error appears. Now I’m disabling the acso patch on ubuntu and I will create the log to show you what happens without.

this is the log of ubuntu without the acso patch enabled :

https://pastebin.ubuntu.com/p/9YP2cm32q5/

instead this is what the log says when I do “nvidia-smi” on debian (here I have the framebuffer enabled):

when I disable the framebuffer,it says :

root@marietto:~# nvidia-smi
No devices were found
[   20.663210] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.1/sound/card0/input
6
[   20.663350] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.1/sound/card0/input
7
[   20.663815] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.1/sound/card0/input
8
[   20.664009] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:03.1/sound/card0/input
9
[   20.664116] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:03.1/sound/card0/inpu
t10
[   20.664229] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:03.1/sound/card0/inpu
t11
[   20.664343] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:03.1/sound/card0/inpu
t12
[   25.461894] nvidia: loading out-of-tree module taints kernel.
[   25.461956] nvidia: module license 'NVIDIA' taints kernel.
[   25.461962] Disabling lock debugging due to kernel taint
[   25.489773] nvidia: module verification failed: signature and/or required key missing - tainting
kernel
[   25.515913] nvidia-nvlink: Nvlink Core is being initialized, major device number 246
[   25.515918] NVRM: probing 0x10de 0x1e04, class 0x30000
[   25.516802] nvidia 0000:00:03.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns
=io+mem
[   25.516858] NVRM: PCI:0000:00:03.0 (10de:1e04): BAR0 @ 0xd2000000 (16MB)
[   25.516861] NVRM: PCI:0000:00:03.0 (10de:1e04): BAR1 @ 0xc0000000 (256MB)
[   25.632001] NVRM: PBI is not supported for GPU 0000:00:03.0
[   25.632052] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  470.57.02  Tue Jul 13 16:14:05 UTC 20
21
[   26.452867] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  470.57.
02  Tue Jul 13 16:06:24 UTC 2021
[   26.599906] [drm:nv_drm_probe_devices [nvidia_drm]] Registering device for NVIDIA GPU ID 0x083
[   26.599933] [drm:drm_minor_register [drm]]  
[   26.600015] [drm:drm_minor_register [drm]] new minor registered 128
[   26.600022] [drm:drm_minor_register [drm]]  
[   26.600070] [drm:drm_minor_register [drm]] new minor registered 0
[   26.600071] [drm] [nvidia-drm] [GPU ID 0x00000003] Loading driver
[   26.600071] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:00:03.0 on minor 0
[   58.213306] NVRM: nvidia_open...
[   58.213343] NVRM: nvidia_ctl_open
[   58.213527] NVRM: ioctl(0xd2, 0x4010b80, 0x48)
[   58.213761] NVRM: ioctl(0xd6, 0x4010c10, 0x8)
[   58.213772] NVRM: ioctl(0xc8, 0x7aed9f40, 0x900)
[   58.213788] NVRM: ioctl(0x2b, 0x4010ce0, 0x20)
[   58.221308] NVRM: ioctl(0x2b, 0x4010c50, 0x28)
[   58.221709] NVRM: ioctl(0x2b, 0x4010c50, 0x28)
[   58.221749] NVRM: ioctl(0x2a, 0x4010c50, 0x20)
[   58.221780] NVRM: ioctl(0x2a, 0x4010c50, 0x20)
[   58.221970] NVRM: ioctl(0x2a, 0x4010c50, 0x20)
[   58.222002] NVRM: ioctl(0x2a, 0x4010c50, 0x20)
[   58.222320] NVRM: nvidia_open...
[   58.222349] NVRM: GPU 0000:00:03.0: Opening GPU with minor number 0
[   58.225375] NVRM: GPU 0000:00:03.0: RmInitAdapter
[   58.225390] NVRM: GPU 0000:00:03.0: RmSetupRegisters for 0x10de:0x1e04
[   58.225398] NVRM: GPU 0000:00:03.0: pci config info:
[   58.225407] NVRM: GPU 0000:00:03.0:    registers look  like: 00000000aee6691b 000000001652004aNVR
M: GPU 0000:00:03.0:    fb        looks like: 000000007bff11a5 00000000b398ef6bNVRM: GPU 0000:00:03.
0: Successfully mapped framebuffer and registers
[   58.226293] NVRM: GPU 0000:00:03.0: final mappings:
[   58.226302] NVRM: GPU 0000:00:03.0:     regs: 00000000aee6691b 000000001652004a 0x000000001c80e5f
c
[   58.365431] NVRM: VM: nv_alloc_pages: 81 pages
[   58.365434] NVRM: VM:    contig 0  cache_type 1
[   58.365438] NVRM: VM: 3243363344: 81 pages
[   58.365715] NVRM: VM: nv_alloc_pages:3479: 0x0000000084326545, 81 page(s), count = 1, flags = 0x0
0000004, page_table = 0x000000000a0bb4c0
[   58.365833] NVRM: VM: nv_alloc_pages: 6 pages
[   58.365835] NVRM: VM:    contig 1  cache_type 1
[   58.365836] NVRM: VM: nv_alloc_contig_pages: 6 pages
[   58.365892] NVRM: VM: nv_alloc_pages:3479: 0x00000000a5061344, 6 page(s), count = 1, flags = 0x00
000005, page_table = 0x0000000046b762f8
[   58.365896] NVRM: VM: nv_alloc_pages: 2 pages
[   58.365897] NVRM: VM:    contig 1  cache_type 1
[   58.365899] NVRM: VM: nv_alloc_contig_pages: 2 pages
[   58.365921] NVRM: VM: nv_alloc_pages:3479: 0x00000000b2cae0b1, 2 page(s), count = 1, flags = 0x00
000005, page_table = 0x00000000fc71cacb
[   58.366212] NVRM: GPU 0000:00:03.0: OS reports GPU is primary VGA
[   58.366214] NVRM: GPU 0000:00:03.0:  is primary VGA
[   58.366956] NVRM: VM: nv_alloc_pages: 3 pages
[   58.366958] NVRM: VM:    contig 1  cache_type 1
[   58.366959] NVRM: VM: nv_alloc_contig_pages: 3 pages
[   58.367005] NVRM: VM: nv_alloc_pages:3479: 0x00000000703034df, 3 page(s), count = 1, flags = 0x00
000005, page_table = 0x0000000053a2a913
[   58.367027] NVRM: VM: nv_alloc_pages: 1 pages
[   58.367028] NVRM: VM:    contig 1  cache_type 1
[   58.367029] NVRM: VM: nv_alloc_contig_pages: 1 pages
[   58.367083] NVRM: VM: nv_alloc_pages:3479: 0x00000000cd7a9d45, 1 page(s), count = 1, flags = 0x00
000005, page_table = 0x00000000bcc0c0db
[   59.678540] tsc: Refined TSC clocksource calibration: 3597.407 MHz
[   59.678552] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x33dac1e8801, max_idle_ns: 44
0795330329 ns
[   60.234213] clocksource: Switched to clocksource tsc
[   62.234501] NVRM: VM: nv_free_pages: 0x3
[   62.234507] NVRM: VM: nv_free_pages:3502: 0x00000000703034df, 3 page(s), count = 1, flags = 0x000
00005, page_table = 0x0000000053a2a913
[   62.234509] NVRM: VM: nv_free_contig_pages: 3 pages
[   62.234628] NVRM: VM: nv_free_pages: 0x6
[   62.234630] NVRM: VM: nv_free_pages:3502: 0x00000000a5061344, 6 page(s), count = 1, flags = 0x000
00005, page_table = 0x0000000046b762f8
[   62.234631] NVRM: VM: nv_free_contig_pages: 6 pages
[   62.234671] NVRM: VM: nv_free_pages: 0x2
[   62.234672] NVRM: VM: nv_free_pages:3502: 0x00000000b2cae0b1, 2 page(s), count = 1, flags = 0x000
00005, page_table = 0x00000000fc71cacb
[   62.234674] NVRM: VM: nv_free_contig_pages: 2 pages
[   62.234693] NVRM: VM: nv_free_pages: 0x51
[   62.234694] NVRM: VM: nv_free_pages:3502: 0x0000000084326545, 81 page(s), count = 1, flags = 0x00
000004, page_table = 0x000000000a0bb4c0
[   62.234696] NVRM: VM: nv_free_system_pages: 81 pages
[   62.234789] NVRM: VM: nv_free_pages: 0x1
[   62.234791] NVRM: VM: nv_free_pages:3502: 0x00000000cd7a9d45, 1 page(s), count = 1, flags = 0x000
00005, page_table = 0x00000000bcc0c0db
[   62.234792] NVRM: VM: nv_free_contig_pages: 1 pages
[   62.234881] NVRM: GPU 0000:00:03.0: Tearing down registers
[   62.234883] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[   62.235097] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[   62.351795] NVRM: PBI is not supported for GPU 0000:00:03.0
[   62.351834] NVRM: nvidia_open...
[   62.351839] NVRM: GPU 0000:00:03.0: Opening GPU with minor number 0
[   62.352354] NVRM: GPU 0000:00:03.0: RmInitAdapter
[   62.352356] NVRM: GPU 0000:00:03.0: RmSetupRegisters for 0x10de:0x1e04
[   62.352358] NVRM: GPU 0000:00:03.0: pci config info:
[   62.352359] NVRM: GPU 0000:00:03.0:    registers look  like: 00000000aee6691b 000000001652004aNVR
M: GPU 0000:00:03.0:    fb        looks like: 000000007bff11a5 00000000b398ef6bNVRM: GPU 0000:00:03.
0: Successfully mapped framebuffer and registers
[   62.352369] NVRM: GPU 0000:00:03.0: final mappings:
[   62.352370] NVRM: GPU 0000:00:03.0:     regs: 00000000aee6691b 000000001652004a 0x000000005c075b4
7
[   62.354333] NVRM: VM: nv_alloc_pages: 81 pages
[   62.354335] NVRM: VM:    contig 0  cache_type 1
[   62.354339] NVRM: VM: 3243363344: 81 pages
[   62.354592] NVRM: VM: nv_alloc_pages:3479: 0x0000000052cb0d7f, 81 page(s), count = 1, flags = 0x0
0000004, page_table = 0x00000000d64f3528
[   62.354647] NVRM: VM: nv_alloc_pages: 6 pages
[   62.354648] NVRM: VM:    contig 1  cache_type 1
[   62.354649] NVRM: VM: nv_alloc_contig_pages: 6 pages
[   62.354702] NVRM: VM: nv_alloc_pages:3479: 0x00000000400212a1, 6 page(s), count = 1, flags = 0x00
000005, page_table = 0x00000000920fab31
[   62.354706] NVRM: VM: nv_alloc_pages: 2 pages
[   62.354706] NVRM: VM:    contig 1  cache_type 1
[   62.354708] NVRM: VM: nv_alloc_contig_pages: 2 pages
[   62.354730] NVRM: VM: nv_alloc_pages:3479: 0x000000005c321e41, 2 page(s), count = 1, flags = 0x00
000005, page_table = 0x00000000af042b8f
[   62.354996] NVRM: GPU 0000:00:03.0: OS reports GPU is primary VGA
[   62.354998] NVRM: GPU 0000:00:03.0:  is primary VGA
[   62.355677] NVRM: VM: nv_alloc_pages: 3 pages
[   62.355679] NVRM: VM:    contig 1  cache_type 1
[   62.355681] NVRM: VM: nv_alloc_contig_pages: 3 pages
[   62.355725] NVRM: VM: nv_alloc_pages:3479: 0x000000007982855e, 3 page(s), count = 1, flags = 0x00
000005, page_table = 0x00000000e32c47bd
[   62.355742] NVRM: VM: nv_alloc_pages: 1 pages
[   62.355744] NVRM: VM:    contig 1  cache_type 1
[   62.355745] NVRM: VM: nv_alloc_contig_pages: 1 pages
[   62.355783] NVRM: VM: nv_alloc_pages:3479: 0x00000000bb1a459b, 1 page(s), count = 1, flags = 0x00
000005, page_table = 0x000000009577ba2a
[   66.350707] NVRM: VM: nv_free_pages: 0x3
[   66.350713] NVRM: VM: nv_free_pages:3502: 0x000000007982855e, 3 page(s), count = 1, flags = 0x000
00005, page_table = 0x00000000e32c47bd
[   66.350716] NVRM: VM: nv_free_contig_pages: 3 pages
[   66.350836] NVRM: VM: nv_free_pages: 0x6
[   66.350838] NVRM: VM: nv_free_pages:3502: 0x00000000400212a1, 6 page(s), count = 1, flags = 0x000
00005, page_table = 0x00000000920fab31
[   66.350840] NVRM: VM: nv_free_contig_pages: 6 pages
[   66.350876] NVRM: VM: nv_free_pages: 0x2
[   66.350877] NVRM: VM: nv_free_pages:3502: 0x000000005c321e41, 2 page(s), count = 1, flags = 0x000
00005, page_table = 0x00000000af042b8f
[   66.350879] NVRM: VM: nv_free_contig_pages: 2 pages
[   66.350898] NVRM: VM: nv_free_pages: 0x51
[   66.350900] NVRM: VM: nv_free_pages:3502: 0x0000000052cb0d7f, 81 page(s), count = 1, flags = 0x00
000004, page_table = 0x00000000d64f3528
[   66.350901] NVRM: VM: nv_free_system_pages: 81 pages
[   66.350997] NVRM: VM: nv_free_pages: 0x1
[   66.350999] NVRM: VM: nv_free_pages:3502: 0x00000000bb1a459b, 1 page(s), count = 1, flags = 0x000
00005, page_table = 0x000000009577ba2a
[   66.351001] NVRM: VM: nv_free_contig_pages: 1 pages
[   66.351087] NVRM: GPU 0000:00:03.0: Tearing down registers
[   66.351090] NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x23:0x65:1204)
[   66.351301] NVRM: GPU 0000:00:03.0: rm_init_adapter failed, device minor number 0
[   66.468646] NVRM: PBI is not supported for GPU 0000:00:03.0
[   66.468693] NVRM: ioctl(0xd1, 0x4010b54, 0xc)
[   66.468707] NVRM: ioctl(0x2a, 0x400e7e0, 0x20)
[   66.468766] NVRM: ioctl(0x2a, 0x4010e40, 0x20)
[   66.468772] NVRM: ioctl(0x29, 0x4010eb0, 0x10)
[   66.468793] NVRM: GPU 0000:00:00.0: nvidia_close on GPU with minor number 255
[   66.468794] NVRM: nvidia_ctl_close

This looks like the gpu can be initialized fine but breaks on throttling up (pcie speed/width). Please try with a 460 driver setting the nvidia module option
NVreg_RegistryDwords=“OverrideMaxPerf=0x1”
Don’t know if that only keeps clocks down or also pcie speed, though.

root@marietto:~# uname -a

Linux marietto 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux

sudo nano /etc/modprobe.d/99-nvidia-registry.conf

options nvidia NVreg_RegistryDwords="OverrideMaxPerf=0x1"
options nvidia NVreg_ResmanDebugLevel=0

sudo update-initramfs -u

added to : nano /etc/apt/sources.list :

deb http://deb.debian.org/debian/ bullseye main contrib non-free

root@marietto:~# apt update
root@marietto:~# apt install nvidia-driver firmware-misc-nonfree

nvidia-driver is already at version (460.73.01-1~bpo10+1).
firmware-misc-nonfree is already at version (20210315-3).

root@marietto:~# apt install nvidia-smi
installed.
update-initramfs: Generating /boot/initrd.img-5.10.0-9-amd64

root@marietto:~# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

root@marietto:~# modinfo nvidia
modinfo: ERROR: Module nvidia not found.

Are u there @generix ? I’ve reinstalled debian 11 with the nvidia driver 460,but this time on linux,with qemu + kvm and …

root@marietto-BHYVE:/home/marietto# lspci

00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 05)
00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.4 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.5 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.6 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.7 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
02:00.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)
03:00.0 Communication controller: Red Hat, Inc. Virtio console (rev 01)
04:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
05:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon (rev 01)
06:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG (rev 01)
07:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
08:00.0 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
09:00.0 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
0a:00.0 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

root@marietto-BHYVE:/home/marietto# modinfo nvidia

modinfo: ERROR: Module nvidia not found.

root@marietto-BHYVE:/home/marietto# nvidia-smi

Unable to determine the device handle for GPU 0000:07:00.0: Unknown Error

interesting : on qemu+kvm I have the same error that I have on bhyve :

[    3.610073] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input6
[    3.610368] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input7
[    3.610401] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input8
[    3.610427] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input9
[    3.610457] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input10
[    3.610489] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input11
[    3.610518] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:02.7/0000:08:00.0/sound/card1/input12
[    3.725450] intel_pmc_core intel_pmc_core.0:  initialized
[    3.864688] [drm] [nvidia-drm] [GPU ID 0x00000700] Loading driver
[    3.864690] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:07:00.0 on minor 1
[    3.925046] NVRM: nvidia_open...
[    3.925052] NVRM: nvidia_ctl_open
[    3.925084] NVRM: ioctl(0xd2, 0xe17e9b10, 0x48)
[    3.925095] NVRM: ioctl(0xd6, 0xe17e9ba0, 0x8)
[    3.925097] NVRM: ioctl(0xc8, 0xe1b96ac0, 0x900)
[    3.925098] NVRM: ioctl(0x2b, 0xe17e9c70, 0x20)
[    3.925130] NVRM: ioctl(0x2a, 0xe17e9c40, 0x20)
[    3.925137] NVRM: ioctl(0x2a, 0xe17e9c40, 0x20)
[    3.925141] NVRM: ioctl(0x29, 0xe17e9c50, 0x10)
[    3.925164] NVRM: GPU 0000:00:00.0: nvidia_close on GPU with minor number 255
[    3.925164] NVRM: nvidia_ctl_close
[    3.925256] NVRM: nvidia_open...
[    3.925258] NVRM: nvidia_ctl_open
[    3.925269] NVRM: ioctl(0xd2, 0xe17e9980, 0x48)
[    3.925274] NVRM: ioctl(0xd6, 0xe17e9a10, 0x8)
[    3.925275] NVRM: ioctl(0xc8, 0xe1b96ac0, 0x900)
[    3.925276] NVRM: ioctl(0x2b, 0xe17e9ae0, 0x20)
[    3.925282] NVRM: ioctl(0x2a, 0xe17e9ab0, 0x20)
[    3.925299] NVRM: ioctl(0x2a, 0xe17e9ab0, 0x20)
[    3.925301] NVRM: ioctl(0x2a, 0xe17e9ab0, 0x20)
[    3.930392] NVRM: nvidia_open...
[    3.930398] NVRM: GPU 0000:07:00.0: Opening GPU with minor number 0
[    3.968808] NVRM: GPU 0000:07:00.0: RmInitAdapter
[    3.968810] NVRM: GPU 0000:07:00.0: RmSetupRegisters for 0x10de:0x1e04
[    3.968810] NVRM: GPU 0000:07:00.0: pci config info:
[    3.968812] NVRM: GPU 0000:07:00.0:    registers look  like: 00000000aff3b263 0000000003909f34NVRM: GPU 0000:07:00.0:    fb        looks like: 000000004d91c650 000000004ddd7894NVRM: GPU 0000:07:00.0: Successfully mapped framebuffer and registers
[    3.968857] NVRM: GPU 0000:07:00.0: final mappings:
[    3.968857] NVRM: GPU 0000:07:00.0:     regs: 00000000aff3b263 0000000003909f34 0x00000000c6008869
[    3.970540] NVRM: GPU 0000:07:00.0: Tearing down registers
[    3.970548] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:624)
[    3.971116] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[    4.087644] NVRM: PBI is not supported for GPU 0000:07:00.0
[    4.087752] NVRM: nvidia_open...
[    4.087757] NVRM: GPU 0000:07:00.0: Opening GPU with minor number 0
[    4.109641] NVRM: GPU 0000:07:00.0: RmInitAdapter
[    4.109642] NVRM: GPU 0000:07:00.0: RmSetupRegisters for 0x10de:0x1e04
[    4.109643] NVRM: GPU 0000:07:00.0: pci config info:
[    4.109644] NVRM: GPU 0000:07:00.0:    registers look  like: 00000000aff3b263 0000000003909f34NVRM: GPU 0000:07:00.0:    fb        looks like: 000000004d91c650 000000004ddd7894NVRM: GPU 0000:07:00.0: Successfully mapped framebuffer and registers
[    4.109657] NVRM: GPU 0000:07:00.0: final mappings:
[    4.109658] NVRM: GPU 0000:07:00.0:     regs: 00000000aff3b263 0000000003909f34 0x00000000c6008869
[    4.110461] NVRM: GPU 0000:07:00.0: Tearing down registers
[    4.110464] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:624)
[    4.110557] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[    4.227079] NVRM: PBI is not supported for GPU 0000:07:00.0
[    4.227302] NVRM: ioctl(0xd1, 0xe17e99b4, 0xc)
[    4.227315] NVRM: ioctl(0x29, 0xe17e9ac0, 0x10)
[    4.227338] NVRM: GPU 0000:00:00.0: nvidia_close on GPU with minor number 255
[    4.227339] NVRM: nvidia_ctl_close
[    4.721730] NVRM: nvidia_open...
[    4.721745] NVRM: nvidia_ctl_open
[    4.721787] NVRM: ioctl(0xd2, 0x86e877f0, 0x48)
[    4.721802] NVRM: ioctl(0xd6, 0x86e87880, 0x8)
[    4.721803] NVRM: ioctl(0xc8, 0x10aa4f00, 0x900)
[    4.721806] NVRM: ioctl(0x2b, 0x86e87950, 0x20)
[    5.007261] NVRM: ioctl(0x2a, 0x86e878f0, 0x20)
[    5.007315] NVRM: ioctl(0x2a, 0x86e878f0, 0x20)
[    5.007335] NVRM: ioctl(0x2a, 0x86e87710, 0x20)
[    5.007509] NVRM: nvidia_open...
[    5.007528] NVRM: GPU 0000:07:00.0: Opening GPU with minor number 0
[    5.026080] NVRM: GPU 0000:07:00.0: RmInitAdapter
[    5.026086] NVRM: GPU 0000:07:00.0: RmSetupRegisters for 0x10de:0x1e04
[    5.026089] NVRM: GPU 0000:07:00.0: pci config info:
[    5.026094] NVRM: GPU 0000:07:00.0:    registers look  like: 00000000aff3b263 0000000003909f34NVRM: GPU 0000:07:00.0:    fb        looks like: 000000004d91c650 000000004ddd7894NVRM: GPU 0000:07:00.0: Successfully mapped framebuffer and registers
[    5.026249] NVRM: GPU 0000:07:00.0: final mappings:
[    5.026254] NVRM: GPU 0000:07:00.0:     regs: 00000000aff3b263 0000000003909f34 0x00000000c6008869
[    5.030759] NVRM: GPU 0000:07:00.0: Tearing down registers
[    5.030772] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:624)
[    5.031442] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[    5.149917] NVRM: PBI is not supported for GPU 0000:07:00.0
[    5.149937] NVRM: nvidia_open...
[    5.149939] NVRM: GPU 0000:07:00.0: Opening GPU with minor number 0
[    5.164147] NVRM: GPU 0000:07:00.0: RmInitAdapter
[    5.164148] NVRM: GPU 0000:07:00.0: RmSetupRegisters for 0x10de:0x1e04
[    5.164148] NVRM: GPU 0000:07:00.0: pci config info:
[    5.164150] NVRM: GPU 0000:07:00.0:    registers look  like: 00000000aff3b263 0000000003909f34NVRM: GPU 0000:07:00.0:    fb        looks like: 000000004d91c650 000000004ddd7894NVRM: GPU 0000:07:00.0: Successfully mapped framebuffer and registers
[    5.164163] NVRM: GPU 0000:07:00.0: final mappings:
[    5.164164] NVRM: GPU 0000:07:00.0:     regs: 00000000aff3b263 0000000003909f34 0x00000000a208264a
[    5.164967] NVRM: GPU 0000:07:00.0: Tearing down registers
[    5.164971] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:624)
[    5.165096] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[    5.281495] NVRM: PBI is not supported for GPU 0000:07:00.0
[    5.281512] NVRM: ioctl(0xd1, 0x86e87614, 0xc)
[   46.949744] NVRM: nvidia_open...
[   46.949784] NVRM: nvidia_ctl_open
[   46.949804] NVRM: ioctl(0xd2, 0x9fee5440, 0x48)
[   46.949847] NVRM: ioctl(0xd6, 0x9fee54d0, 0x8)
[   46.949852] NVRM: ioctl(0xc8, 0xe407ae40, 0x900)
[   46.949860] NVRM: ioctl(0x2b, 0x9fee55a0, 0x20)
[   46.950845] NVRM: ioctl(0x2b, 0x9fee5510, 0x28)
[   46.951160] NVRM: ioctl(0x2b, 0x9fee5510, 0x28)
[   46.951195] NVRM: ioctl(0x2a, 0x9fee5510, 0x20)
[   46.951220] NVRM: ioctl(0x2a, 0x9fee5510, 0x20)
[   46.951237] NVRM: ioctl(0x2a, 0x9fee5510, 0x20)
[   46.951251] NVRM: ioctl(0x2a, 0x9fee5510, 0x20)
[   46.951364] NVRM: nvidia_open...
[   46.951418] NVRM: GPU 0000:07:00.0: Opening GPU with minor number 0
[   46.969096] NVRM: GPU 0000:07:00.0: RmInitAdapter
[   46.969097] NVRM: GPU 0000:07:00.0: RmSetupRegisters for 0x10de:0x1e04
[   46.969098] NVRM: GPU 0000:07:00.0: pci config info:
[   46.969099] NVRM: GPU 0000:07:00.0:    registers look  like: 00000000aff3b263 0000000003909f34NVRM: GPU 0000:07:00.0:    fb        looks like: 000000004d91c650 000000004ddd7894NVRM: GPU 0000:07:00.0: Successfully mapped framebuffer and registers
[   46.969115] NVRM: GPU 0000:07:00.0: final mappings:
[   46.969116] NVRM: GPU 0000:07:00.0:     regs: 00000000aff3b263 0000000003909f34 0x00000000c6008869
[   46.969918] NVRM: GPU 0000:07:00.0: Tearing down registers
[   46.969920] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:624)
[   46.970007] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[   47.100295] NVRM: PBI is not supported for GPU 0000:07:00.0
[   47.100364] NVRM: nvidia_open...
[   47.100379] NVRM: GPU 0000:07:00.0: Opening GPU with minor number 0
[   47.120081] NVRM: GPU 0000:07:00.0: RmInitAdapter
[   47.120082] NVRM: GPU 0000:07:00.0: RmSetupRegisters for 0x10de:0x1e04
[   47.120082] NVRM: GPU 0000:07:00.0: pci config info:
[   47.120084] NVRM: GPU 0000:07:00.0:    registers look  like: 00000000aff3b263 0000000003909f34NVRM: GPU 0000:07:00.0:    fb        looks like: 000000004d91c650 000000004ddd7894NVRM: GPU 0000:07:00.0: Successfully mapped framebuffer and registers
[   47.120098] NVRM: GPU 0000:07:00.0: final mappings:
[   47.120099] NVRM: GPU 0000:07:00.0:     regs: 00000000aff3b263 0000000003909f34 0x00000000a208264a
[   47.120924] NVRM: GPU 0000:07:00.0: Tearing down registers
[   47.120927] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:624)
[   47.121053] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[   47.237507] NVRM: PBI is not supported for GPU 0000:07:00.0
[   47.237522] NVRM: ioctl(0xd1, 0x9fee5414, 0xc)
[   47.237532] NVRM: ioctl(0x2a, 0x9fee30b0, 0x20)
[   47.237565] NVRM: ioctl(0x2a, 0x9fee5790, 0x20)
[   47.237594] NVRM: nvidia_open...
[   47.237597] NVRM: GPU 0000:07:00.0: Opening GPU with minor number 0
[   47.256044] NVRM: GPU 0000:07:00.0: RmInitAdapter
[   47.256045] NVRM: GPU 0000:07:00.0: RmSetupRegisters for 0x10de:0x1e04
[   47.256046] NVRM: GPU 0000:07:00.0: pci config info:
[   47.256047] NVRM: GPU 0000:07:00.0:    registers look  like: 00000000aff3b263 0000000003909f34NVRM: GPU 0000:07:00.0:    fb        looks like: 000000004d91c650 000000004ddd7894NVRM: GPU 0000:07:00.0: Successfully mapped framebuffer and registers
[   47.256060] NVRM: GPU 0000:07:00.0: final mappings:
[   47.256060] NVRM: GPU 0000:07:00.0:     regs: 00000000aff3b263 0000000003909f34 0x000000008a8bc16c
[   47.256846] NVRM: GPU 0000:07:00.0: Tearing down registers
[   47.256849] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:624)
[   47.256977] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[   47.373394] NVRM: PBI is not supported for GPU 0000:07:00.0
[   47.373455] NVRM: nvidia_open...
[   47.373459] NVRM: GPU 0000:07:00.0: Opening GPU with minor number 0
[   47.388295] NVRM: GPU 0000:07:00.0: RmInitAdapter
[   47.388296] NVRM: GPU 0000:07:00.0: RmSetupRegisters for 0x10de:0x1e04
[   47.388297] NVRM: GPU 0000:07:00.0: pci config info:
[   47.388298] NVRM: GPU 0000:07:00.0:    registers look  like: 00000000aff3b263 0000000003909f34NVRM: GPU 0000:07:00.0:    fb        looks like: 000000004d91c650 000000004ddd7894NVRM: GPU 0000:07:00.0: Successfully mapped framebuffer and registers
[   47.388314] NVRM: GPU 0000:07:00.0: final mappings:
[   47.388314] NVRM: GPU 0000:07:00.0:     regs: 00000000aff3b263 0000000003909f34 0x0000000050932ed6
[   47.389155] NVRM: GPU 0000:07:00.0: Tearing down registers
[   47.389158] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x23:0x56:624)
[   47.389254] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 0
[   47.505811] NVRM: PBI is not supported for GPU 0000:07:00.0
[   47.505827] NVRM: ioctl(0xd1, 0x9fee5694, 0xc)
[   47.505831] NVRM: ioctl(0x2a, 0x9fee5790, 0x20)
[   47.505868] NVRM: ioctl(0x2a, 0x9fee5700, 0x20)
[   47.505872] NVRM: ioctl(0x29, 0x9fee5770, 0x10)
[   47.505892] NVRM: GPU 0000:00:00.0: nvidia_close on GPU with minor number 255
[   47.505893] NVRM: nvidia_ctl_close