NVIDIA related warning on booting linux 4.14

I am seeing a warning while booting linux 4.14.2

[   43.438583] resource sanity check: requesting [mem 0x000a0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000a0000-0x000bffff window]
[   43.438698] caller _nv001193rm+0x166/0x1c0 [nvidia] mapping multiple BARs
[   43.802318] nvidia-modeset: Allocated GPU:0 (GPU-a432c7bb-af50-7966-be96-385d4d9c5c97) @ PCI:0000:01:00.0

This warning wasn’t there under the previous kernel I was using (4.9 series)

They just changed the message in newer kernels. It has probably always been there due to a buggy bios. The previous message was
Info: mapping multiple BARs. Your kernel is fine.

GPU bios?

Motherboard BIOS.

You may disregard these messages - they are harmless.

I did journalctl | grep BARs. The journal goes back to june. There was no mention of the term ‘BARs’ in the journal till now.

I can confirm this message just occured with kernel 4.14.x, never had this with older versions.

According to this: https://lwn.net/Articles/300667/

(patch) ioremap sanity check to catch mapping requests exceeding the BAR sizes

Go through the iomem resource tree to check if any of the ioremap() requests
span more than any slot in the iomem resource tree and do a WARN_ON() if we hit
this check.

This will raise a red-flag, if some driver is mapping more than what
is needed. And hopefully identify possible corruptions much earlier.

Can someone confirm, this won’t lead to “possible corruptions”?

Thanks in advance
Marcus

I’m a bit puzzled why this message is displayed just now for some people. The change of the message was already in kernel 4.8. Since then not much has changed in regard to ioremap. Just some changes in the e820 code.
Can this cause corruption? Possibly but unlikely and if so, you can’t do anything about it anyway other than wait for a fixed bios from your manufacturer.
Why is this displayed now? Mystery, better asked that on the kernel mailing list.

I filed bug 2017354 about this. From the investigation, I’m pretty sure this message is harmless but we’re fixing it for a future driver release nonetheless.

Is this fixed in 390.12? I’m on 4.9.76 at the moment before the warning started (in 4.14.xx).

No, is not fixed in 390.12.

[Fr Jan 12 10:01:02 2018] resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000d3fff window]
[Fr Jan 12 10:01:02 2018] caller _nv001171rm+0xe3/0x1d0 [nvidia] mapping multiple BARs
[Fr Jan 12 10:01:02 2018] resource sanity check: requesting [mem 0x000e0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000e0000-0x000e3fff window]
[Fr Jan 12 10:01:02 2018] caller _nv029907rm+0x57/0x90 [nvidia] mapping multiple BARs

It’s there on 4.15-rc7 and 390.12.

~
towo:Defiant> inxi -SG
System:    Host: Defiant Kernel: 4.15.0-rc7-siduction-amd64 x86_64 bits: 64 Desktop: Cinnamon 3.4.6
           Distro: siduction 13.2.0 December - xfce - (201312310323)
Graphics:  Card: NVIDIA GP107 [GeForce GTX 1050 Ti]
           Display Server: x11 (X.Org 1.19.5 ) driver: nvidia Resolution: 1920x1080@60.00hz
           OpenGL: renderer: GeForce GTX 1050 Ti/PCIe/SSE2 version: 4.5.0 NVIDIA 390.12

I’m not seeing this message anymore on Linux kernel 4.14.16 and nvidia driver 390.25 which I verified by doing “dmesg | grep BAR”.

Hi Guys

Problem still present with 4.17.4 / 340.107
NVIDIA Corporation GT218 [GeForce 210] (rev a2)

If there is a patch.

[ 9.427688] resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000d3fff window
]
[ 9.427781] caller _nv000788rm+0xe4/0x1c0 [nvidia] mapping multiple BARs
[ 9.437157] resource sanity check: requesting [mem 0x000e0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000e0000-0x000e3fff window
]
[ 9.437246] caller _nv013519rm+0x57/0xa0 [nvidia] mapping multiple BARs

[ 17.483331] ------------[ cut here ]------------
[ 17.483334] Bad or missing usercopy whitelist? Kernel memory exposure attempt detected from SLAB object ‘nvidia_stack_t’ (offset 11864, size 3)!
[ 17.483342] WARNING: CPU: 1 PID: 456 at mm/usercopy.c:81 usercopy_warn+0x7e/0xa0
[ 17.483343] Modules linked in: uvcvideo aes_x86_64 videobuf2_vmalloc videobuf2_memops algif_skcipher videobuf2_v4l2 joydev intel_rapl x86_pkg_temp_thermal usbmouse videobuf2_common usbkbd videodev intel_powerclamp media usbhid coretemp kvm_intel kvm snd_hda_codec_hdmi irqbypass crct10dif_pclmul crc
32_pclmul crc32c_intel ghash_clmulni_intel cryptd intel_cstate nvidia(PO) wmi_bmof evdev intel_uncore drm ehci_pci snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core snd_pcm snd_timer snd ehci_hcd intel_rapl_perf soundcore lpc_ich mei_me mei xhci_pci xhci_hc
d mfd_core usbcore e1000e ptp pps_core video i2c_i801 thermal ie31200_edac wmi shpchp bridge button fan stp llc sch_fq_codel w83627ehf hwmon_vid
[ 17.483374] CPU: 1 PID: 456 Comm: X Tainted: P O 4.17.4-aufs #1
[ 17.483374] Hardware name: /DH77EB, BIOS EBH7710H.86A.0103.2014.1015.1532 10/15/2014
[ 17.483376] RIP: 0010:usercopy_warn+0x7e/0xa0
[ 17.483377] RSP: 0018:ffffb5dd01fa7bb0 EFLAGS: 00010286
[ 17.483378] RAX: 0000000000000000 RBX: ffff94d987202f98 RCX: 0000000000000006
[ 17.483378] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff94d99ec96730
[ 17.483379] RBP: 0000000000000003 R08: 00000000000002e1 R09: 0000000000000004
[ 17.483380] R10: ffffffff88eb3940 R11: 0000000000000001 R12: 0000000000000001
[ 17.483380] R13: ffff94d987202f9b R14: ffff94d987202f98 R15: ffff94d987202fe0
[ 17.483381] FS: 00007f4c677b5680(0000) GS:ffff94d99ec80000(0000) knlGS:0000000000000000
[ 17.483382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 17.483383] CR2: 00007f4c5e59f160 CR3: 0000000300dca004 CR4: 00000000001606e0
[ 17.483384] Call Trace:
[ 17.483388] __check_object_size+0x9c/0x1a0
[ 17.483463] os_memcpy_to_user+0x21/0x40 [nvidia]
[ 17.483525] _nv001372rm+0xa5/0x260 [nvidia]
[ 17.483588] ? _nv004784rm+0x4eba/0x5500 [nvidia]
[ 17.483648] ? _nv004331rm+0xec/0xf0 [nvidia]
[ 17.483707] ? _nv004326rm+0xca/0x650 [nvidia]
[ 17.483767] ? _nv015126rm+0x576/0x5c0 [nvidia]
[ 17.483827] ? _nv000694rm+0x2e/0x60 [nvidia]
[ 17.483886] ? _nv000789rm+0x5f5/0x8b0 [nvidia]
[ 17.483944] ? rm_ioctl+0x73/0x100 [nvidia]
[ 17.484004] ? nv_check_pci_config_space+0x1691/0x1950 [nvidia]
[ 17.484063] ? nvidia_frontend_ioctl+0x2d/0x60 [nvidia]
[ 17.484122] ? nvidia_frontend_unlocked_ioctl+0x19/0x20 [nvidia]
[ 17.484124] ? do_vfs_ioctl+0xa4/0x630
[ 17.484125] ? ksys_ioctl+0x70/0x80
[ 17.484127] ? __x64_sys_ioctl+0x16/0x20
[ 17.484129] ? do_syscall_64+0x55/0x110
[ 17.484131] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 17.484132] Code: 48 c7 c0 2a 25 e7 88 48 0f 44 c2 41 50 51 41 51 48 89 f9 49 89 f1 4d 89 d8 4c 89 d2 48 89 c6 48 c7 c7 80 25 e7 88 e8 f2 b4 e3 ff <
0f> 0b 48 83 c4 18 c3 48 c7 c6 6c 8b e8 88 49 89 f1 49 89 f3 eb
[ 17.484153] —[ end trace d1adf9f719aa56fd ]—

The message is still there:

4.14.52-gentoo #1 SMP PREEMPT x86_64 Intel® Xeon® CPU E31270 @ 3.40GHz
nvidia 390.87
GT 710

[ 56.934751] resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000d3fff window]
[ 56.934855] caller _nv001169rm+0xe3/0x1d0 [nvidia] mapping multiple BARs

18.04.4 here with 440 drivers…still having it. resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]

Running 5.4.0-42-lowlatency with a GTX 1060 and GTX 1660.

This happens far far too often and kills my session…drops me back to lightdm/restarts X.

I can confirm issue still present with:
Ubuntu 18.04.5 LTS
CUDA Version: 11.0
Driver Version: 450.51.06
GPU GeForce RTX 2080 ti

When I hit this issue, nvidia-smi stops working, that message comes continuously in kernel message logs (dmesg) and can’t access GPU after that.

Once I unplug the GPU (from motherboard) and plug it back and reboot the machine, it starts working and after some time issue is reproduced automatically. If I do only reboot, it doesn’t work.

Is this issue of hardware or the driver/kernel?

Message:
[ 3.588588] resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000d3fff window]
[ 3.588699] caller os_map_kernel_space.part.10+0x6d/0x80 [nvidia] mapping multiple BARs

This error came back (Aug. 20 2020).

My system:

CentOS 8 x64 without GNOME (Just command line)
GeForce GTX 1660 SUPER (6GB)
Driver: NVIDIA-Linux-x86_64-450.66.run
Mother board: TUF B450M-PLUS GAMING

In my case the error is shown when I type command:

nvidia-smi

Error:

[ 59.970061] resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
[ 59.971671] caller _nv000745rm+0x1af/0x200 [nvidia] mapping multiple BARs

Hardware:

CPU: Intel® Core™ i7-9800X CPU @ 3.80GHz
RAM: DDR4 4 x 16 GB
Motherboard: ASUSTeK COMPUTER INC. WS X299 SAGE (bios Version: 3203 06/10/2020)

 +-[0000:64]-+-00.0-[65-68]----00.0-[66-68]--+-08.0-[67]--+-00.0  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti]
 |           |                               |            +-00.1  NVIDIA Corporation TU102 High Definition Audio Controller
 |           |                               |            +-00.2  NVIDIA Corporation TU102 USB 3.1 Host Controller
 |           |                               |            \-00.3  NVIDIA Corporation TU102 USB Type-C UCSI Controller
 |           |                               \-10.0-[68]--+-00.0  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti]
 |           |                                            +-00.1  NVIDIA Corporation TU102 High Definition Audio Controller
 |           |                                            +-00.2  NVIDIA Corporation TU102 USB 3.1 Host Controller
 |           |                                            \-00.3  NVIDIA Corporation TU102 USB Type-C UCSI Controller


 +-[0000:16]-+-00.0-[17-1a]----00.0-[18-1a]--+-08.0-[19]----00.0  Intel Corporation 82599 10 Gigabit Network Connection
 |           |                               \-10.0-[1a]--+-00.0  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A]
 |           |                                            +-00.1  NVIDIA Corporation TU102 High Definition Audio Controller
 |           |                                            +-00.2  NVIDIA Corporation TU102 USB 3.1 Host Controller
 |           |                                            \-00.3  NVIDIA Corporation TU102 USB Type-C UCSI Controller

cat /etc/redhat-release
CentOS Linux release 8.2.2004 (Core)

uname -r
4.18.0-193.19.1.el8_2.x86_64

Driver: NVIDIA-Linux-x86_64-455.23.04.run

All GPU work on this OS, but in journal Error, example:

Nov 25 12:26:21 nvtest.localdomain kernel: caller _nv000708rm+0x1af/0x200 [nvidia] mapping multiple BARs
Nov 25 12:26:24 nvtest.localdomain kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c4000-0x000c7fff window]

cat /etc/os-release
Ubuntu 20.04.1 LTS

uname -r
5.4.0-53-generic

Driver: NVIDIA-Linux-x86_64-455.45.01.run

GPU 67, 68 work on this OS, 1a not working. In log Error (pci 19, 1a), example: BAR 1: failed to assign [mem size 0x10000000 64bit pref]

nvidia-smi

Wed Nov 25 16:12:11 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01    Driver Version: 455.45.01    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:67:00.0 Off |                  N/A |
| 29%   39C    P8    20W / 250W |      5MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:68:00.0 Off |                  N/A |
| 28%   37C    P8    21W / 250W |     68MiB / 11016MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1038      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A      1280      G   /usr/bin/gnome-shell                0MiB |
|    1   N/A  N/A      1038      G   /usr/lib/xorg/Xorg                 56MiB |
|    1   N/A  N/A      1280      G   /usr/bin/gnome-shell               10MiB |
+-----------------------------------------------------------------------------+

cat /var/log/syslog | grep 1a:

Nov 25 16:02:57 arm1 kernel: [    1.454939] pci 0000:1a:00.0: [10de:1e07] type 00 class 0x030000
Nov 25 16:02:57 arm1 kernel: [    1.454962] pci 0000:1a:00.0: reg 0x10: [mem 0x95000000-0x95ffffff]
Nov 25 16:02:57 arm1 kernel: [    1.454971] pci 0000:1a:00.0: reg 0x14: [mem 0xdfe0000000-0xdfefffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454980] pci 0000:1a:00.0: reg 0x1c: [mem 0xdff0000000-0xdff1ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454986] pci 0000:1a:00.0: reg 0x24: [io  0x6000-0x607f]
Nov 25 16:02:57 arm1 kernel: [    1.454991] pci 0000:1a:00.0: reg 0x30: [mem 0x96000000-0x9607ffff pref]
Nov 25 16:02:57 arm1 kernel: [    1.454997] pci 0000:1a:00.0: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.455061] pci 0000:1a:00.0: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.455140] pci 0000:1a:00.1: [10de:10f7] type 00 class 0x040300
Nov 25 16:02:57 arm1 kernel: [    1.455157] pci 0000:1a:00.1: reg 0x10: [mem 0x96080000-0x96083fff]
Nov 25 16:02:57 arm1 kernel: [    1.455204] pci 0000:1a:00.1: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.455289] pci 0000:1a:00.2: [10de:1ad6] type 00 class 0x0c0330
Nov 25 16:02:57 arm1 kernel: [    1.455310] pci 0000:1a:00.2: reg 0x10: [mem 0xdff2000000-0xdff203ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.455328] pci 0000:1a:00.2: reg 0x1c: [mem 0xdff2040000-0xdff204ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.455348] pci 0000:1a:00.2: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.455394] pci 0000:1a:00.2: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.455438] pci 0000:1a:00.3: [10de:1ad7] type 00 class 0x0c8000
Nov 25 16:02:57 arm1 kernel: [    1.455455] pci 0000:1a:00.3: reg 0x10: [mem 0x96084000-0x96084fff]
Nov 25 16:02:57 arm1 kernel: [    1.455502] pci 0000:1a:00.3: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.455548] pci 0000:1a:00.3: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:1a:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:1a:00.0: vgaarb: bridge control possible
Nov 25 16:02:57 arm1 kernel: [    2.500848] pci 0000:1a:00.0: BAR 1: no space for [mem size 0x10000000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500849] pci 0000:1a:00.0: BAR 1: failed to assign [mem size 0x10000000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500850] pci 0000:1a:00.0: BAR 3: no space for [mem size 0x02000000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500850] pci 0000:1a:00.0: BAR 3: failed to assign [mem size 0x02000000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500851] pci 0000:1a:00.2: BAR 0: assigned [mem 0x960c0000-0x960fffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500859] pci 0000:1a:00.2: BAR 3: assigned [mem 0x96090000-0x9609ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500952] pci 0000:1a:00.0: BAR 1: assigned [mem 0xd000000000-0xd00fffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500959] pci 0000:1a:00.0: BAR 3: assigned [mem 0xd010000000-0xd011ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500966] pci 0000:1a:00.0: BAR 0: no space for [mem size 0x01000000]
Nov 25 16:02:57 arm1 kernel: [    2.500967] pci 0000:1a:00.0: BAR 0: failed to assign [mem size 0x01000000]
Nov 25 16:02:57 arm1 kernel: [    2.500968] pci 0000:1a:00.0: BAR 6: no space for [mem size 0x00080000 pref]
Nov 25 16:02:57 arm1 kernel: [    2.500968] pci 0000:1a:00.0: BAR 6: failed to assign [mem size 0x00080000 pref]
Nov 25 16:02:57 arm1 kernel: [    2.500969] pci 0000:1a:00.2: BAR 0: assigned [mem 0xd012000000-0xd01203ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500976] pci 0000:1a:00.2: BAR 3: assigned [mem 0xd012040000-0xd01204ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500983] pci 0000:1a:00.1: BAR 0: no space for [mem size 0x00004000]
Nov 25 16:02:57 arm1 kernel: [    2.500984] pci 0000:1a:00.1: BAR 0: failed to assign [mem size 0x00004000]
Nov 25 16:02:57 arm1 kernel: [    2.500985] pci 0000:1a:00.3: BAR 0: no space for [mem size 0x00001000]
Nov 25 16:02:57 arm1 kernel: [    2.500985] pci 0000:1a:00.3: BAR 0: failed to assign [mem size 0x00001000]
Nov 25 16:02:57 arm1 kernel: [    2.501016] pci_bus 0000:1a: resource 0 [io  0x6000-0x6fff]
Nov 25 16:02:57 arm1 kernel: [    2.501016] pci_bus 0000:1a: resource 2 [mem 0xd000000000-0xd017ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.503521] pci 0000:1a:00.1: D0 power state depends on 0000:1a:00.0
Nov 25 16:02:57 arm1 kernel: [    2.503570] pci 0000:1a:00.2: D0 power state depends on 0000:1a:00.0
Nov 25 16:02:57 arm1 kernel: [    2.503812] pci 0000:1a:00.3: D0 power state depends on 0000:1a:00.0
Nov 25 16:02:57 arm1 kernel: [    2.754493] xhci_hcd 0000:1a:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.754495] xhci_hcd 0000:1a:00.2: new USB bus registered, assigned bus number 7
Nov 25 16:02:57 arm1 kernel: [    2.755120] xhci_hcd 0000:1a:00.2: hcc params 0x0180ff05 hci version 0x110 quirks 0x0000000000000010
Nov 25 16:02:57 arm1 kernel: [    2.755218] usb usb7: SerialNumber: 0000:1a:00.2
Nov 25 16:02:57 arm1 kernel: [    2.755432] xhci_hcd 0000:1a:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.755434] xhci_hcd 0000:1a:00.2: new USB bus registered, assigned bus number 8
Nov 25 16:02:57 arm1 kernel: [    2.755435] xhci_hcd 0000:1a:00.2: Host supports USB 3.1 Enhanced SuperSpeed
Nov 25 16:02:57 arm1 kernel: [    2.755461] usb usb8: SerialNumber: 0000:1a:00.2
Nov 25 16:02:57 arm1 kernel: [    2.964130] nvidia-gpu 0000:1a:00.3: pcim_iomap failed
Nov 25 16:02:57 arm1 kernel: [    2.964201] nvidia-gpu: probe of 0000:1a:00.3 failed with error -12
Nov 25 16:02:57 arm1 kernel: [   11.639229] snd_hda_intel 0000:1a:00.1: Disabling MSI
Nov 25 16:02:57 arm1 kernel: [   11.639251] snd_hda_intel 0000:1a:00.1: Handle vga_switcheroo audio client
Nov 25 16:02:57 arm1 kernel: [   11.639268] snd_hda_intel 0000:1a:00.1: can't ioremap BAR 0: [??? 0x00000000 flags 0x0]
Nov 25 16:02:57 arm1 kernel: [   11.639269] snd_hda_intel 0000:1a:00.1: ioremap error
Nov 25 16:02:57 arm1 kernel: [   11.987947] nvidia 0000:1a:00.0: enabling device (0100 -> 0103)
Nov 25 16:02:57 arm1 kernel: [   11.988020] NVRM: BAR0 is 0M @ 0x0 (PCI:0000:1a:00.0)
Nov 25 16:02:57 arm1 kernel: [   11.988027] nvidia: probe of 0000:1a:00.0 failed with error -1

cat /var/log/syslog | grep 19:

Nov 25 16:02:57 arm1 kernel: [    1.454558] pci 0000:19:00.0: [8086:1557] type 00 class 0x020000
Nov 25 16:02:57 arm1 kernel: [    1.454579] pci 0000:19:00.0: reg 0x10: [mem 0xdff2200000-0xdff227ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454586] pci 0000:19:00.0: reg 0x18: [io  0x7000-0x701f]
Nov 25 16:02:57 arm1 kernel: [    1.454602] pci 0000:19:00.0: reg 0x20: [mem 0xdff2280000-0xdff2283fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454609] pci 0000:19:00.0: reg 0x30: [mem 0x96200000-0x9627ffff pref]
Nov 25 16:02:57 arm1 kernel: [    1.454667] pci 0000:19:00.0: PME# supported from D0 D3hot D3cold
Nov 25 16:02:57 arm1 kernel: [    1.454696] pci 0000:19:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454697] pci 0000:19:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
Nov 25 16:02:57 arm1 kernel: [    1.454712] pci 0000:19:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454713] pci 0000:19:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
Nov 25 16:02:57 arm1 kernel: [    2.500711] pci 0000:19:00.0: BAR 7: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500712] pci 0000:19:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500715] pci 0000:19:00.0: BAR 10: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500715] pci 0000:19:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500762] pci 0000:19:00.0: BAR 0: assigned [mem 0x96280000-0x962fffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500769] pci 0000:19:00.0: BAR 4: no space for [mem size 0x00004000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500770] pci 0000:19:00.0: BAR 4: failed to assign [mem size 0x00004000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500771] pci 0000:19:00.0: BAR 7: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500771] pci 0000:19:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500772] pci 0000:19:00.0: BAR 10: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500772] pci 0000:19:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500829] pci 0000:19:00.0: BAR 0: assigned [mem 0x96100000-0x9617ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500835] pci 0000:19:00.0: BAR 6: assigned [mem 0x96180000-0x961fffff pref]
Nov 25 16:02:57 arm1 kernel: [    2.500836] pci 0000:19:00.0: BAR 4: no space for [mem size 0x00004000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500837] pci 0000:19:00.0: BAR 4: failed to assign [mem size 0x00004000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500837] pci 0000:19:00.0: BAR 7: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500838] pci 0000:19:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500839] pci 0000:19:00.0: BAR 10: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500839] pci 0000:19:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500922] pci 0000:19:00.0: BAR 0: assigned [mem 0xd018000000-0xd01807ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500929] pci 0000:19:00.0: BAR 6: no space for [mem size 0x00080000 pref]
Nov 25 16:02:57 arm1 kernel: [    2.500930] pci 0000:19:00.0: BAR 6: failed to assign [mem size 0x00080000 pref]
Nov 25 16:02:57 arm1 kernel: [    2.500930] pci 0000:19:00.0: BAR 4: assigned [mem 0xd018080000-0xd018083fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500937] pci 0000:19:00.0: BAR 7: assigned [mem 0xd018084000-0xd018183fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500940] pci 0000:19:00.0: BAR 10: assigned [mem 0xd018184000-0xd018283fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.501015] pci_bus 0000:19: resource 0 [io  0x7000-0x7fff]
Nov 25 16:02:57 arm1 kernel: [    2.501015] pci_bus 0000:19: resource 2 [mem 0xd018000000-0xd0184fffff 64bit pref]

cat /var/log/syslog | grep 67:

Nov 25 16:02:57 arm1 nvidia-persistenced: device 0000:67:00.0 - registered
Nov 25 16:02:57 arm1 kernel: [    1.965918] pci 0000:67:00.0: [10de:1e04] type 00 class 0x030000
Nov 25 16:02:57 arm1 kernel: [    1.965941] pci 0000:67:00.0: reg 0x10: [mem 0xd7000000-0xd7ffffff]
Nov 25 16:02:57 arm1 kernel: [    1.965950] pci 0000:67:00.0: reg 0x14: [mem 0xc0000000-0xcfffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.965960] pci 0000:67:00.0: reg 0x1c: [mem 0xd0000000-0xd1ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.965965] pci 0000:67:00.0: reg 0x24: [io  0xb000-0xb07f]
Nov 25 16:02:57 arm1 kernel: [    1.965971] pci 0000:67:00.0: reg 0x30: [mem 0xd8000000-0xd807ffff pref]
Nov 25 16:02:57 arm1 kernel: [    1.965977] pci 0000:67:00.0: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966040] pci 0000:67:00.0: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.966115] pci 0000:67:00.1: [10de:10f7] type 00 class 0x040300
Nov 25 16:02:57 arm1 kernel: [    1.966132] pci 0000:67:00.1: reg 0x10: [mem 0xd8080000-0xd8083fff]
Nov 25 16:02:57 arm1 kernel: [    1.966178] pci 0000:67:00.1: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966270] pci 0000:67:00.2: [10de:1ad6] type 00 class 0x0c0330
Nov 25 16:02:57 arm1 kernel: [    1.966290] pci 0000:67:00.2: reg 0x10: [mem 0xd2000000-0xd203ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966308] pci 0000:67:00.2: reg 0x1c: [mem 0xd2040000-0xd204ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966327] pci 0000:67:00.2: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966373] pci 0000:67:00.2: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.966417] pci 0000:67:00.3: [10de:1ad7] type 00 class 0x0c8000
Nov 25 16:02:57 arm1 kernel: [    1.966434] pci 0000:67:00.3: reg 0x10: [mem 0xd8084000-0xd8084fff]
Nov 25 16:02:57 arm1 kernel: [    1.966480] pci 0000:67:00.3: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966526] pci 0000:67:00.3: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:67:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:67:00.0: vgaarb: bridge control possible
Nov 25 16:02:57 arm1 kernel: [    2.501078] pci_bus 0000:67: resource 0 [io  0xb000-0xbfff]
Nov 25 16:02:57 arm1 kernel: [    2.501078] pci_bus 0000:67: resource 1 [mem 0xd7000000-0xd80fffff]
Nov 25 16:02:57 arm1 kernel: [    2.501079] pci_bus 0000:67: resource 2 [mem 0xc0000000-0xd20fffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.503885] pci 0000:67:00.1: D0 power state depends on 0000:67:00.0
Nov 25 16:02:57 arm1 kernel: [    2.503929] pci 0000:67:00.2: D0 power state depends on 0000:67:00.0
Nov 25 16:02:57 arm1 kernel: [    2.504135] pci 0000:67:00.3: D0 power state depends on 0000:67:00.0
Nov 25 16:02:57 arm1 kernel: [    2.755667] xhci_hcd 0000:67:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.755669] xhci_hcd 0000:67:00.2: new USB bus registered, assigned bus number 9
Nov 25 16:02:57 arm1 kernel: [    2.756280] xhci_hcd 0000:67:00.2: hcc params 0x0180ff05 hci version 0x110 quirks 0x0000000000000010
Nov 25 16:02:57 arm1 kernel: [    2.756376] usb usb9: SerialNumber: 0000:67:00.2
Nov 25 16:02:57 arm1 kernel: [    2.756497] xhci_hcd 0000:67:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.756498] xhci_hcd 0000:67:00.2: new USB bus registered, assigned bus number 10
Nov 25 16:02:57 arm1 kernel: [    2.756500] xhci_hcd 0000:67:00.2: Host supports USB 3.1 Enhanced SuperSpeed
Nov 25 16:02:57 arm1 kernel: [    2.756525] usb usb10: SerialNumber: 0000:67:00.2
Nov 25 16:02:57 arm1 kernel: [   11.639400] snd_hda_intel 0000:67:00.1: Disabling MSI
Nov 25 16:02:57 arm1 kernel: [   11.639417] snd_hda_intel 0000:67:00.1: Handle vga_switcheroo audio client
Nov 25 16:02:57 arm1 kernel: [   11.988042] nvidia 0000:67:00.0: enabling device (0100 -> 0103)
Nov 25 16:02:57 arm1 kernel: [   11.988086] nvidia 0000:67:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Nov 25 16:02:57 arm1 kernel: [   12.423398] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:67:00.0 on minor 0
Nov 25 16:02:57 arm1 kernel: [   13.555268] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input15
Nov 25 16:02:57 arm1 kernel: [   13.555302] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input16
Nov 25 16:02:57 arm1 kernel: [   13.555383] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input17
Nov 25 16:02:57 arm1 kernel: [   13.555430] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input18
Nov 25 16:02:57 arm1 kernel: [   13.555491] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input19
Nov 25 16:02:57 arm1 kernel: [   13.555538] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input20
Nov 25 16:02:57 arm1 kernel: [   13.555570] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input21

cat /var/log/syslog | grep 68:

Nov 25 16:02:57 arm1 nvidia-persistenced: device 0000:68:00.0 - registered
Nov 25 16:02:57 arm1 kernel: [    1.966604] pci 0000:68:00.0: [10de:1e04] type 00 class 0x030000
Nov 25 16:02:57 arm1 kernel: [    1.966628] pci 0000:68:00.0: reg 0x10: [mem 0xd5000000-0xd5ffffff]
Nov 25 16:02:57 arm1 kernel: [    1.966639] pci 0000:68:00.0: reg 0x14: [mem 0xa0000000-0xafffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966651] pci 0000:68:00.0: reg 0x1c: [mem 0xb0000000-0xb1ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966658] pci 0000:68:00.0: reg 0x24: [io  0xa000-0xa07f]
Nov 25 16:02:57 arm1 kernel: [    1.966665] pci 0000:68:00.0: reg 0x30: [mem 0xd6000000-0xd607ffff pref]
Nov 25 16:02:57 arm1 kernel: [    1.966671] pci 0000:68:00.0: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966687] pci 0000:68:00.0: BAR 3: assigned to efifb
Nov 25 16:02:57 arm1 kernel: [    1.966735] pci 0000:68:00.0: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.966785] pci 0000:68:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x16 link at 0000:66:10.0 (capable of 126.016 Gb/s with 8 GT/s x16 link)
Nov 25 16:02:57 arm1 kernel: [    1.966812] pci 0000:68:00.1: [10de:10f7] type 00 class 0x040300
Nov 25 16:02:57 arm1 kernel: [    1.966829] pci 0000:68:00.1: reg 0x10: [mem 0xd6080000-0xd6083fff]
Nov 25 16:02:57 arm1 kernel: [    1.966875] pci 0000:68:00.1: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966960] pci 0000:68:00.2: [10de:1ad6] type 00 class 0x0c0330
Nov 25 16:02:57 arm1 kernel: [    1.966981] pci 0000:68:00.2: reg 0x10: [mem 0xb2000000-0xb203ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966999] pci 0000:68:00.2: reg 0x1c: [mem 0xb2040000-0xb204ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.967018] pci 0000:68:00.2: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.967064] pci 0000:68:00.2: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.967113] pci 0000:68:00.3: [10de:1ad7] type 00 class 0x0c8000
Nov 25 16:02:57 arm1 kernel: [    1.967130] pci 0000:68:00.3: reg 0x10: [mem 0xd6084000-0xd6084fff]
Nov 25 16:02:57 arm1 kernel: [    1.967176] pci 0000:68:00.3: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.967221] pci 0000:68:00.3: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:68:00.0: vgaarb: setting as boot VGA device
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:68:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:68:00.0: vgaarb: bridge control possible
Nov 25 16:02:57 arm1 kernel: [    2.501079] pci_bus 0000:68: resource 0 [io  0xa000-0xafff]
Nov 25 16:02:57 arm1 kernel: [    2.501080] pci_bus 0000:68: resource 1 [mem 0xd5000000-0xd60fffff]
Nov 25 16:02:57 arm1 kernel: [    2.501081] pci_bus 0000:68: resource 2 [mem 0xa0000000-0xb20fffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.504147] pci 0000:68:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
Nov 25 16:02:57 arm1 kernel: [    2.504192] pci 0000:68:00.1: D0 power state depends on 0000:68:00.0
Nov 25 16:02:57 arm1 kernel: [    2.504237] pci 0000:68:00.2: D0 power state depends on 0000:68:00.0
Nov 25 16:02:57 arm1 kernel: [    2.504361] pci 0000:68:00.3: D0 power state depends on 0000:68:00.0
Nov 25 16:02:57 arm1 kernel: [    2.756714] xhci_hcd 0000:68:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.756716] xhci_hcd 0000:68:00.2: new USB bus registered, assigned bus number 11
Nov 25 16:02:57 arm1 kernel: [    2.757305] xhci_hcd 0000:68:00.2: hcc params 0x0180ff05 hci version 0x110 quirks 0x0000000000000010
Nov 25 16:02:57 arm1 kernel: [    2.757390] usb usb11: SerialNumber: 0000:68:00.2
Nov 25 16:02:57 arm1 kernel: [    2.757499] xhci_hcd 0000:68:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.757500] xhci_hcd 0000:68:00.2: new USB bus registered, assigned bus number 12
Nov 25 16:02:57 arm1 kernel: [    2.757501] xhci_hcd 0000:68:00.2: Host supports USB 3.1 Enhanced SuperSpeed
Nov 25 16:02:57 arm1 kernel: [    2.757526] usb usb12: SerialNumber: 0000:68:00.2
Nov 25 16:02:57 arm1 kernel: [   11.639545] snd_hda_intel 0000:68:00.1: Disabling MSI
Nov 25 16:02:57 arm1 kernel: [   11.639559] snd_hda_intel 0000:68:00.1: Handle vga_switcheroo audio client
Nov 25 16:02:57 arm1 kernel: [   12.104147] nvidia 0000:68:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
Nov 25 16:02:57 arm1 kernel: [   12.423453] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:68:00.0 on minor 1
Nov 25 16:02:57 arm1 kernel: [   13.554660] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input22
Nov 25 16:02:57 arm1 kernel: [   13.554781] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input23
Nov 25 16:02:57 arm1 kernel: [   13.554865] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input24
Nov 25 16:02:57 arm1 kernel: [   13.554921] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input25
Nov 25 16:02:57 arm1 kernel: [   13.554996] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input26
Nov 25 16:02:57 arm1 kernel: [   13.555088] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input27
Nov 25 16:02:57 arm1 kernel: [   13.555192] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input28

root@arm1:~# lspci -s 1a:00.0 -vv

1a:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. TU102 [GeForce RTX 2080 Ti Rev. A]
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 25
	NUMA node: 0
	Region 0: Memory at <ignored> (32-bit, non-prefetchable)
	Region 1: Memory at d000000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at d010000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at 6000 [size=128]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #16, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [250 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [258 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [bb0 v1] Resizable BAR <?>
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

root@arm1:~# lspci -s 67:00.0 -vv

67:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. TU102 [GeForce RTX 2080 Ti]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 105
	NUMA node: 0
	Region 0: Memory at d7000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at d0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at b000 [size=128]
	Expansion ROM at d8000000 [virtual] [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee001b8  Data: 0000
	Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #8, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x16 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, NROPrPrP-, LTR-
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [258 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [bb0 v1] Resizable BAR <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

root@arm1:~# lspci -s 68:00.0 -vv

68:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. TU102 [GeForce RTX 2080 Ti]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 104
	NUMA node: 0
	Region 0: Memory at d5000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at a0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at b0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at a000 [size=128]
	Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00198  Data: 0000
	Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #16, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x16 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, NROPrPrP-, LTR-
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [258 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [bb0 v1] Resizable BAR <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

root@arm1:~# lspci -s 19:00.0 -vv

19:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation 82599 10 Gigabit Network Connection
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 25
	NUMA node: 0
	Region 0: Memory at d018000000 (64-bit, prefetchable) [size=512K]
	Region 2: I/O ports at 7000 [size=32]
	Region 4: Memory at d018080000 (64-bit, prefetchable) [size=16K]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #8, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <2us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (ok), Width x8 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR-
			 10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, TPHComp-, ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
		Not readable
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [140 v1] Device Serial Number f0-b4-d2-ff-ff-4b-90-a0
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 128, stride: 2, Device ID: 10ed
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 000000d018084000 (64-bit, prefetchable)
		Region 3: Memory at 000000d018184000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: ixgbe
	Kernel modules: ixgbe