PLX/PCI Error. MultiGPU. BAR: failed to assign

Mordok · November 27, 2020, 8:46am

Hardware:

CPU: Intel(R) Core™ i7-9800X CPU @ 3.80GHz
RAM: DDR4 4 x 16 GB
Motherboard: ASUSTeK COMPUTER INC. WS X299 SAGE (bios Version: 3203 06/10/2020)

 +-[0000:64]-+-00.0-[65-68]----00.0-[66-68]--+-08.0-[67]--+-00.0  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti]
 |           |                               |            +-00.1  NVIDIA Corporation TU102 High Definition Audio Controller
 |           |                               |            +-00.2  NVIDIA Corporation TU102 USB 3.1 Host Controller
 |           |                               |            \-00.3  NVIDIA Corporation TU102 USB Type-C UCSI Controller
 |           |                               \-10.0-[68]--+-00.0  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti]
 |           |                                            +-00.1  NVIDIA Corporation TU102 High Definition Audio Controller
 |           |                                            +-00.2  NVIDIA Corporation TU102 USB 3.1 Host Controller
 |           |                                            \-00.3  NVIDIA Corporation TU102 USB Type-C UCSI Controller


 +-[0000:16]-+-00.0-[17-1a]----00.0-[18-1a]--+-08.0-[19]----00.0  Intel Corporation 82599 10 Gigabit Network Connection
 |           |                               \-10.0-[1a]--+-00.0  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A]
 |           |                                            +-00.1  NVIDIA Corporation TU102 High Definition Audio Controller
 |           |                                            +-00.2  NVIDIA Corporation TU102 USB 3.1 Host Controller
 |           |                                            \-00.3  NVIDIA Corporation TU102 USB Type-C UCSI Controller

cat /etc/redhat-release
CentOS Linux release 8.2.2004 (Core)

uname -r
4.18.0-193.19.1.el8_2.x86_64

Driver: NVIDIA-Linux-x86_64-455.23.04.run

All GPU work on this OS, but in journal Error, example:

Nov 25 12:26:21 nvtest.localdomain kernel: caller _nv000708rm+0x1af/0x200 [nvidia] mapping multiple BARs
Nov 25 12:26:24 nvtest.localdomain kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c4000-0x000c7fff window]

cat /etc/os-release
Ubuntu 20.04.1 LTS

uname -r
5.4.0-53-generic

Driver: NVIDIA-Linux-x86_64-455.45.01.run

GPU 67, 68 work on this OS, 1a not working. In log Error (pci 19, 1a), example: BAR 1: failed to assign [mem size 0x10000000 64bit pref]

nvidia-smi

Wed Nov 25 16:12:11 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01    Driver Version: 455.45.01    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:67:00.0 Off |                  N/A |
| 29%   39C    P8    20W / 250W |      5MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:68:00.0 Off |                  N/A |
| 28%   37C    P8    21W / 250W |     68MiB / 11016MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1038      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A      1280      G   /usr/bin/gnome-shell                0MiB |
|    1   N/A  N/A      1038      G   /usr/lib/xorg/Xorg                 56MiB |
|    1   N/A  N/A      1280      G   /usr/bin/gnome-shell               10MiB |
+-----------------------------------------------------------------------------+

cat /var/log/syslog | grep 1a:

Nov 25 16:02:57 arm1 kernel: [    1.454939] pci 0000:1a:00.0: [10de:1e07] type 00 class 0x030000
Nov 25 16:02:57 arm1 kernel: [    1.454962] pci 0000:1a:00.0: reg 0x10: [mem 0x95000000-0x95ffffff]
Nov 25 16:02:57 arm1 kernel: [    1.454971] pci 0000:1a:00.0: reg 0x14: [mem 0xdfe0000000-0xdfefffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454980] pci 0000:1a:00.0: reg 0x1c: [mem 0xdff0000000-0xdff1ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454986] pci 0000:1a:00.0: reg 0x24: [io  0x6000-0x607f]
Nov 25 16:02:57 arm1 kernel: [    1.454991] pci 0000:1a:00.0: reg 0x30: [mem 0x96000000-0x9607ffff pref]
Nov 25 16:02:57 arm1 kernel: [    1.454997] pci 0000:1a:00.0: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.455061] pci 0000:1a:00.0: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.455140] pci 0000:1a:00.1: [10de:10f7] type 00 class 0x040300
Nov 25 16:02:57 arm1 kernel: [    1.455157] pci 0000:1a:00.1: reg 0x10: [mem 0x96080000-0x96083fff]
Nov 25 16:02:57 arm1 kernel: [    1.455204] pci 0000:1a:00.1: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.455289] pci 0000:1a:00.2: [10de:1ad6] type 00 class 0x0c0330
Nov 25 16:02:57 arm1 kernel: [    1.455310] pci 0000:1a:00.2: reg 0x10: [mem 0xdff2000000-0xdff203ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.455328] pci 0000:1a:00.2: reg 0x1c: [mem 0xdff2040000-0xdff204ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.455348] pci 0000:1a:00.2: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.455394] pci 0000:1a:00.2: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.455438] pci 0000:1a:00.3: [10de:1ad7] type 00 class 0x0c8000
Nov 25 16:02:57 arm1 kernel: [    1.455455] pci 0000:1a:00.3: reg 0x10: [mem 0x96084000-0x96084fff]
Nov 25 16:02:57 arm1 kernel: [    1.455502] pci 0000:1a:00.3: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.455548] pci 0000:1a:00.3: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:1a:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:1a:00.0: vgaarb: bridge control possible
Nov 25 16:02:57 arm1 kernel: [    2.500848] pci 0000:1a:00.0: BAR 1: no space for [mem size 0x10000000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500849] pci 0000:1a:00.0: BAR 1: failed to assign [mem size 0x10000000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500850] pci 0000:1a:00.0: BAR 3: no space for [mem size 0x02000000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500850] pci 0000:1a:00.0: BAR 3: failed to assign [mem size 0x02000000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500851] pci 0000:1a:00.2: BAR 0: assigned [mem 0x960c0000-0x960fffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500859] pci 0000:1a:00.2: BAR 3: assigned [mem 0x96090000-0x9609ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500952] pci 0000:1a:00.0: BAR 1: assigned [mem 0xd000000000-0xd00fffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500959] pci 0000:1a:00.0: BAR 3: assigned [mem 0xd010000000-0xd011ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500966] pci 0000:1a:00.0: BAR 0: no space for [mem size 0x01000000]
Nov 25 16:02:57 arm1 kernel: [    2.500967] pci 0000:1a:00.0: BAR 0: failed to assign [mem size 0x01000000]
Nov 25 16:02:57 arm1 kernel: [    2.500968] pci 0000:1a:00.0: BAR 6: no space for [mem size 0x00080000 pref]
Nov 25 16:02:57 arm1 kernel: [    2.500968] pci 0000:1a:00.0: BAR 6: failed to assign [mem size 0x00080000 pref]
Nov 25 16:02:57 arm1 kernel: [    2.500969] pci 0000:1a:00.2: BAR 0: assigned [mem 0xd012000000-0xd01203ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500976] pci 0000:1a:00.2: BAR 3: assigned [mem 0xd012040000-0xd01204ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500983] pci 0000:1a:00.1: BAR 0: no space for [mem size 0x00004000]
Nov 25 16:02:57 arm1 kernel: [    2.500984] pci 0000:1a:00.1: BAR 0: failed to assign [mem size 0x00004000]
Nov 25 16:02:57 arm1 kernel: [    2.500985] pci 0000:1a:00.3: BAR 0: no space for [mem size 0x00001000]
Nov 25 16:02:57 arm1 kernel: [    2.500985] pci 0000:1a:00.3: BAR 0: failed to assign [mem size 0x00001000]
Nov 25 16:02:57 arm1 kernel: [    2.501016] pci_bus 0000:1a: resource 0 [io  0x6000-0x6fff]
Nov 25 16:02:57 arm1 kernel: [    2.501016] pci_bus 0000:1a: resource 2 [mem 0xd000000000-0xd017ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.503521] pci 0000:1a:00.1: D0 power state depends on 0000:1a:00.0
Nov 25 16:02:57 arm1 kernel: [    2.503570] pci 0000:1a:00.2: D0 power state depends on 0000:1a:00.0
Nov 25 16:02:57 arm1 kernel: [    2.503812] pci 0000:1a:00.3: D0 power state depends on 0000:1a:00.0
Nov 25 16:02:57 arm1 kernel: [    2.754493] xhci_hcd 0000:1a:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.754495] xhci_hcd 0000:1a:00.2: new USB bus registered, assigned bus number 7
Nov 25 16:02:57 arm1 kernel: [    2.755120] xhci_hcd 0000:1a:00.2: hcc params 0x0180ff05 hci version 0x110 quirks 0x0000000000000010
Nov 25 16:02:57 arm1 kernel: [    2.755218] usb usb7: SerialNumber: 0000:1a:00.2
Nov 25 16:02:57 arm1 kernel: [    2.755432] xhci_hcd 0000:1a:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.755434] xhci_hcd 0000:1a:00.2: new USB bus registered, assigned bus number 8
Nov 25 16:02:57 arm1 kernel: [    2.755435] xhci_hcd 0000:1a:00.2: Host supports USB 3.1 Enhanced SuperSpeed
Nov 25 16:02:57 arm1 kernel: [    2.755461] usb usb8: SerialNumber: 0000:1a:00.2
Nov 25 16:02:57 arm1 kernel: [    2.964130] nvidia-gpu 0000:1a:00.3: pcim_iomap failed
Nov 25 16:02:57 arm1 kernel: [    2.964201] nvidia-gpu: probe of 0000:1a:00.3 failed with error -12
Nov 25 16:02:57 arm1 kernel: [   11.639229] snd_hda_intel 0000:1a:00.1: Disabling MSI
Nov 25 16:02:57 arm1 kernel: [   11.639251] snd_hda_intel 0000:1a:00.1: Handle vga_switcheroo audio client
Nov 25 16:02:57 arm1 kernel: [   11.639268] snd_hda_intel 0000:1a:00.1: can't ioremap BAR 0: [??? 0x00000000 flags 0x0]
Nov 25 16:02:57 arm1 kernel: [   11.639269] snd_hda_intel 0000:1a:00.1: ioremap error
Nov 25 16:02:57 arm1 kernel: [   11.987947] nvidia 0000:1a:00.0: enabling device (0100 -> 0103)
Nov 25 16:02:57 arm1 kernel: [   11.988020] NVRM: BAR0 is 0M @ 0x0 (PCI:0000:1a:00.0)
Nov 25 16:02:57 arm1 kernel: [   11.988027] nvidia: probe of 0000:1a:00.0 failed with error -1

cat /var/log/syslog | grep 19:

Nov 25 16:02:57 arm1 kernel: [    1.454558] pci 0000:19:00.0: [8086:1557] type 00 class 0x020000
Nov 25 16:02:57 arm1 kernel: [    1.454579] pci 0000:19:00.0: reg 0x10: [mem 0xdff2200000-0xdff227ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454586] pci 0000:19:00.0: reg 0x18: [io  0x7000-0x701f]
Nov 25 16:02:57 arm1 kernel: [    1.454602] pci 0000:19:00.0: reg 0x20: [mem 0xdff2280000-0xdff2283fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454609] pci 0000:19:00.0: reg 0x30: [mem 0x96200000-0x9627ffff pref]
Nov 25 16:02:57 arm1 kernel: [    1.454667] pci 0000:19:00.0: PME# supported from D0 D3hot D3cold
Nov 25 16:02:57 arm1 kernel: [    1.454696] pci 0000:19:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454697] pci 0000:19:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
Nov 25 16:02:57 arm1 kernel: [    1.454712] pci 0000:19:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.454713] pci 0000:19:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
Nov 25 16:02:57 arm1 kernel: [    2.500711] pci 0000:19:00.0: BAR 7: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500712] pci 0000:19:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500715] pci 0000:19:00.0: BAR 10: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500715] pci 0000:19:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500762] pci 0000:19:00.0: BAR 0: assigned [mem 0x96280000-0x962fffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500769] pci 0000:19:00.0: BAR 4: no space for [mem size 0x00004000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500770] pci 0000:19:00.0: BAR 4: failed to assign [mem size 0x00004000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500771] pci 0000:19:00.0: BAR 7: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500771] pci 0000:19:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500772] pci 0000:19:00.0: BAR 10: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500772] pci 0000:19:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500829] pci 0000:19:00.0: BAR 0: assigned [mem 0x96100000-0x9617ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500835] pci 0000:19:00.0: BAR 6: assigned [mem 0x96180000-0x961fffff pref]
Nov 25 16:02:57 arm1 kernel: [    2.500836] pci 0000:19:00.0: BAR 4: no space for [mem size 0x00004000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500837] pci 0000:19:00.0: BAR 4: failed to assign [mem size 0x00004000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500837] pci 0000:19:00.0: BAR 7: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500838] pci 0000:19:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500839] pci 0000:19:00.0: BAR 10: no space for [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500839] pci 0000:19:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500922] pci 0000:19:00.0: BAR 0: assigned [mem 0xd018000000-0xd01807ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500929] pci 0000:19:00.0: BAR 6: no space for [mem size 0x00080000 pref]
Nov 25 16:02:57 arm1 kernel: [    2.500930] pci 0000:19:00.0: BAR 6: failed to assign [mem size 0x00080000 pref]
Nov 25 16:02:57 arm1 kernel: [    2.500930] pci 0000:19:00.0: BAR 4: assigned [mem 0xd018080000-0xd018083fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500937] pci 0000:19:00.0: BAR 7: assigned [mem 0xd018084000-0xd018183fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.500940] pci 0000:19:00.0: BAR 10: assigned [mem 0xd018184000-0xd018283fff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.501015] pci_bus 0000:19: resource 0 [io  0x7000-0x7fff]
Nov 25 16:02:57 arm1 kernel: [    2.501015] pci_bus 0000:19: resource 2 [mem 0xd018000000-0xd0184fffff 64bit pref]

cat /var/log/syslog | grep 67:

Nov 25 16:02:57 arm1 nvidia-persistenced: device 0000:67:00.0 - registered
Nov 25 16:02:57 arm1 kernel: [    1.965918] pci 0000:67:00.0: [10de:1e04] type 00 class 0x030000
Nov 25 16:02:57 arm1 kernel: [    1.965941] pci 0000:67:00.0: reg 0x10: [mem 0xd7000000-0xd7ffffff]
Nov 25 16:02:57 arm1 kernel: [    1.965950] pci 0000:67:00.0: reg 0x14: [mem 0xc0000000-0xcfffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.965960] pci 0000:67:00.0: reg 0x1c: [mem 0xd0000000-0xd1ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.965965] pci 0000:67:00.0: reg 0x24: [io  0xb000-0xb07f]
Nov 25 16:02:57 arm1 kernel: [    1.965971] pci 0000:67:00.0: reg 0x30: [mem 0xd8000000-0xd807ffff pref]
Nov 25 16:02:57 arm1 kernel: [    1.965977] pci 0000:67:00.0: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966040] pci 0000:67:00.0: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.966115] pci 0000:67:00.1: [10de:10f7] type 00 class 0x040300
Nov 25 16:02:57 arm1 kernel: [    1.966132] pci 0000:67:00.1: reg 0x10: [mem 0xd8080000-0xd8083fff]
Nov 25 16:02:57 arm1 kernel: [    1.966178] pci 0000:67:00.1: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966270] pci 0000:67:00.2: [10de:1ad6] type 00 class 0x0c0330
Nov 25 16:02:57 arm1 kernel: [    1.966290] pci 0000:67:00.2: reg 0x10: [mem 0xd2000000-0xd203ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966308] pci 0000:67:00.2: reg 0x1c: [mem 0xd2040000-0xd204ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966327] pci 0000:67:00.2: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966373] pci 0000:67:00.2: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.966417] pci 0000:67:00.3: [10de:1ad7] type 00 class 0x0c8000
Nov 25 16:02:57 arm1 kernel: [    1.966434] pci 0000:67:00.3: reg 0x10: [mem 0xd8084000-0xd8084fff]
Nov 25 16:02:57 arm1 kernel: [    1.966480] pci 0000:67:00.3: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966526] pci 0000:67:00.3: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:67:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:67:00.0: vgaarb: bridge control possible
Nov 25 16:02:57 arm1 kernel: [    2.501078] pci_bus 0000:67: resource 0 [io  0xb000-0xbfff]
Nov 25 16:02:57 arm1 kernel: [    2.501078] pci_bus 0000:67: resource 1 [mem 0xd7000000-0xd80fffff]
Nov 25 16:02:57 arm1 kernel: [    2.501079] pci_bus 0000:67: resource 2 [mem 0xc0000000-0xd20fffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.503885] pci 0000:67:00.1: D0 power state depends on 0000:67:00.0
Nov 25 16:02:57 arm1 kernel: [    2.503929] pci 0000:67:00.2: D0 power state depends on 0000:67:00.0
Nov 25 16:02:57 arm1 kernel: [    2.504135] pci 0000:67:00.3: D0 power state depends on 0000:67:00.0
Nov 25 16:02:57 arm1 kernel: [    2.755667] xhci_hcd 0000:67:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.755669] xhci_hcd 0000:67:00.2: new USB bus registered, assigned bus number 9
Nov 25 16:02:57 arm1 kernel: [    2.756280] xhci_hcd 0000:67:00.2: hcc params 0x0180ff05 hci version 0x110 quirks 0x0000000000000010
Nov 25 16:02:57 arm1 kernel: [    2.756376] usb usb9: SerialNumber: 0000:67:00.2
Nov 25 16:02:57 arm1 kernel: [    2.756497] xhci_hcd 0000:67:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.756498] xhci_hcd 0000:67:00.2: new USB bus registered, assigned bus number 10
Nov 25 16:02:57 arm1 kernel: [    2.756500] xhci_hcd 0000:67:00.2: Host supports USB 3.1 Enhanced SuperSpeed
Nov 25 16:02:57 arm1 kernel: [    2.756525] usb usb10: SerialNumber: 0000:67:00.2
Nov 25 16:02:57 arm1 kernel: [   11.639400] snd_hda_intel 0000:67:00.1: Disabling MSI
Nov 25 16:02:57 arm1 kernel: [   11.639417] snd_hda_intel 0000:67:00.1: Handle vga_switcheroo audio client
Nov 25 16:02:57 arm1 kernel: [   11.988042] nvidia 0000:67:00.0: enabling device (0100 -> 0103)
Nov 25 16:02:57 arm1 kernel: [   11.988086] nvidia 0000:67:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Nov 25 16:02:57 arm1 kernel: [   12.423398] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:67:00.0 on minor 0
Nov 25 16:02:57 arm1 kernel: [   13.555268] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input15
Nov 25 16:02:57 arm1 kernel: [   13.555302] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input16
Nov 25 16:02:57 arm1 kernel: [   13.555383] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input17
Nov 25 16:02:57 arm1 kernel: [   13.555430] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input18
Nov 25 16:02:57 arm1 kernel: [   13.555491] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input19
Nov 25 16:02:57 arm1 kernel: [   13.555538] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input20
Nov 25 16:02:57 arm1 kernel: [   13.555570] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:08.0/0000:67:00.1/sound/card2/input21

cat /var/log/syslog | grep 68:

Nov 25 16:02:57 arm1 nvidia-persistenced: device 0000:68:00.0 - registered
Nov 25 16:02:57 arm1 kernel: [    1.966604] pci 0000:68:00.0: [10de:1e04] type 00 class 0x030000
Nov 25 16:02:57 arm1 kernel: [    1.966628] pci 0000:68:00.0: reg 0x10: [mem 0xd5000000-0xd5ffffff]
Nov 25 16:02:57 arm1 kernel: [    1.966639] pci 0000:68:00.0: reg 0x14: [mem 0xa0000000-0xafffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966651] pci 0000:68:00.0: reg 0x1c: [mem 0xb0000000-0xb1ffffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966658] pci 0000:68:00.0: reg 0x24: [io  0xa000-0xa07f]
Nov 25 16:02:57 arm1 kernel: [    1.966665] pci 0000:68:00.0: reg 0x30: [mem 0xd6000000-0xd607ffff pref]
Nov 25 16:02:57 arm1 kernel: [    1.966671] pci 0000:68:00.0: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966687] pci 0000:68:00.0: BAR 3: assigned to efifb
Nov 25 16:02:57 arm1 kernel: [    1.966735] pci 0000:68:00.0: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.966785] pci 0000:68:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x16 link at 0000:66:10.0 (capable of 126.016 Gb/s with 8 GT/s x16 link)
Nov 25 16:02:57 arm1 kernel: [    1.966812] pci 0000:68:00.1: [10de:10f7] type 00 class 0x040300
Nov 25 16:02:57 arm1 kernel: [    1.966829] pci 0000:68:00.1: reg 0x10: [mem 0xd6080000-0xd6083fff]
Nov 25 16:02:57 arm1 kernel: [    1.966875] pci 0000:68:00.1: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.966960] pci 0000:68:00.2: [10de:1ad6] type 00 class 0x0c0330
Nov 25 16:02:57 arm1 kernel: [    1.966981] pci 0000:68:00.2: reg 0x10: [mem 0xb2000000-0xb203ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.966999] pci 0000:68:00.2: reg 0x1c: [mem 0xb2040000-0xb204ffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    1.967018] pci 0000:68:00.2: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.967064] pci 0000:68:00.2: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    1.967113] pci 0000:68:00.3: [10de:1ad7] type 00 class 0x0c8000
Nov 25 16:02:57 arm1 kernel: [    1.967130] pci 0000:68:00.3: reg 0x10: [mem 0xd6084000-0xd6084fff]
Nov 25 16:02:57 arm1 kernel: [    1.967176] pci 0000:68:00.3: enabling Extended Tags
Nov 25 16:02:57 arm1 kernel: [    1.967221] pci 0000:68:00.3: PME# supported from D0 D3hot
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:68:00.0: vgaarb: setting as boot VGA device
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:68:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
Nov 25 16:02:57 arm1 kernel: [    2.476438] pci 0000:68:00.0: vgaarb: bridge control possible
Nov 25 16:02:57 arm1 kernel: [    2.501079] pci_bus 0000:68: resource 0 [io  0xa000-0xafff]
Nov 25 16:02:57 arm1 kernel: [    2.501080] pci_bus 0000:68: resource 1 [mem 0xd5000000-0xd60fffff]
Nov 25 16:02:57 arm1 kernel: [    2.501081] pci_bus 0000:68: resource 2 [mem 0xa0000000-0xb20fffff 64bit pref]
Nov 25 16:02:57 arm1 kernel: [    2.504147] pci 0000:68:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
Nov 25 16:02:57 arm1 kernel: [    2.504192] pci 0000:68:00.1: D0 power state depends on 0000:68:00.0
Nov 25 16:02:57 arm1 kernel: [    2.504237] pci 0000:68:00.2: D0 power state depends on 0000:68:00.0
Nov 25 16:02:57 arm1 kernel: [    2.504361] pci 0000:68:00.3: D0 power state depends on 0000:68:00.0
Nov 25 16:02:57 arm1 kernel: [    2.756714] xhci_hcd 0000:68:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.756716] xhci_hcd 0000:68:00.2: new USB bus registered, assigned bus number 11
Nov 25 16:02:57 arm1 kernel: [    2.757305] xhci_hcd 0000:68:00.2: hcc params 0x0180ff05 hci version 0x110 quirks 0x0000000000000010
Nov 25 16:02:57 arm1 kernel: [    2.757390] usb usb11: SerialNumber: 0000:68:00.2
Nov 25 16:02:57 arm1 kernel: [    2.757499] xhci_hcd 0000:68:00.2: xHCI Host Controller
Nov 25 16:02:57 arm1 kernel: [    2.757500] xhci_hcd 0000:68:00.2: new USB bus registered, assigned bus number 12
Nov 25 16:02:57 arm1 kernel: [    2.757501] xhci_hcd 0000:68:00.2: Host supports USB 3.1 Enhanced SuperSpeed
Nov 25 16:02:57 arm1 kernel: [    2.757526] usb usb12: SerialNumber: 0000:68:00.2
Nov 25 16:02:57 arm1 kernel: [   11.639545] snd_hda_intel 0000:68:00.1: Disabling MSI
Nov 25 16:02:57 arm1 kernel: [   11.639559] snd_hda_intel 0000:68:00.1: Handle vga_switcheroo audio client
Nov 25 16:02:57 arm1 kernel: [   12.104147] nvidia 0000:68:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
Nov 25 16:02:57 arm1 kernel: [   12.423453] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:68:00.0 on minor 1
Nov 25 16:02:57 arm1 kernel: [   13.554660] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input22
Nov 25 16:02:57 arm1 kernel: [   13.554781] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input23
Nov 25 16:02:57 arm1 kernel: [   13.554865] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input24
Nov 25 16:02:57 arm1 kernel: [   13.554921] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input25
Nov 25 16:02:57 arm1 kernel: [   13.554996] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input26
Nov 25 16:02:57 arm1 kernel: [   13.555088] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input27
Nov 25 16:02:57 arm1 kernel: [   13.555192] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:10.0/0000:68:00.1/sound/card3/input28

root@arm1:~# lspci -s 1a:00.0 -vv

1a:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. TU102 [GeForce RTX 2080 Ti Rev. A]
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 25
	NUMA node: 0
	Region 0: Memory at <ignored> (32-bit, non-prefetchable)
	Region 1: Memory at d000000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at d010000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at 6000 [size=128]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #16, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [250 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [258 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [bb0 v1] Resizable BAR <?>
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

root@arm1:~# lspci -s 67:00.0 -vv

67:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. TU102 [GeForce RTX 2080 Ti]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 105
	NUMA node: 0
	Region 0: Memory at d7000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at d0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at b000 [size=128]
	Expansion ROM at d8000000 [virtual] [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee001b8  Data: 0000
	Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #8, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x16 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, NROPrPrP-, LTR-
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [258 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [bb0 v1] Resizable BAR <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

root@arm1:~# lspci -s 68:00.0 -vv

68:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. TU102 [GeForce RTX 2080 Ti]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 104
	NUMA node: 0
	Region 0: Memory at d5000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at a0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at b0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at a000 [size=128]
	Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00198  Data: 0000
	Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #16, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x16 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, NROPrPrP-, LTR-
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [258 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [bb0 v1] Resizable BAR <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

root@arm1:~# lspci -s 19:00.0 -vv

19:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation 82599 10 Gigabit Network Connection
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 25
	NUMA node: 0
	Region 0: Memory at d018000000 (64-bit, prefetchable) [size=512K]
	Region 2: I/O ports at 7000 [size=32]
	Region 4: Memory at d018080000 (64-bit, prefetchable) [size=16K]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #8, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <2us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (ok), Width x8 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR-
			 10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, TPHComp-, ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
		Not readable
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [140 v1] Device Serial Number f0-b4-d2-ff-ff-4b-90-a0
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 128, stride: 2, Device ID: 10ed
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 000000d018084000 (64-bit, prefetchable)
		Region 3: Memory at 000000d018184000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: ixgbe
	Kernel modules: ixgbe

brittany.posey · April 23, 2021, 6:27pm

I have a similar issues where the BAR memory isn’t assigned correctly. Has there been any update to this issue?

generix · April 23, 2021, 7:16pm

Please see this:
https://forums.developer.nvidia.com/t/ph402-dual-p100-64g-rminitadapter-failed-memory-mapping-issue/173877/2?u=generix

generix · April 23, 2021, 7:18pm

If that doesn’t apply, please see this:
https://forums.developer.nvidia.com/t/nvidia-kernel-driver-cannot-bind-to-rtx-3060-laptop-gpu/175990/2?u=generix

brittany.posey · April 23, 2021, 11:02pm

So both of these forums don’t help in my situation. We are using the AGX Xavier, so there isn’t a BIOS we can configure. We can configure the device tree though. I created a forum about the issues we are seeing with using the AGX Xavier Module and Eqip Radios: PCIE DMA Bar 0 not valid on all PCIE lanes (Jetson AGX Xavier). If you know of a way we can do something similar, please let me know.
Thanks

generix · April 24, 2021, 11:59am

This forum and the posts are about the x86 platform graphics drivers, so I know next to nothing about the Jetson platforms, please keep that in mind for the following.
The outputs in your post on the jetson forum are from an out-of-tree driver, so not very useful since only someone who knows that driver would know what the output means.
So you should rather take a look at the raw kernel pci messages, check
lspci -t
so you know which device is connected where, use
lspci -vxx -s 0001:01:00.0
to get the memory allocations of the device and

dmesg |grep "root bus resource"

to get what memory windows the jetson firmware provides.
Also,

dmesg |grep "0001:01:00.0"

should give you a hint if resorce allocation fails and why. Don’t know if pci resource reallocation is already enabled in the jetson kernel (pci=realloc).

Mordok · April 26, 2021, 9:00am

In my problem, this solution seems to have helped 2 cards issue - #5 by hetzr281j