PCI Express card not being detected on boot occasionally

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

We are using Jetson Xavier Developer Kit with PCI Express x4 USB card (Startech PEXUSB3S44V), and have an issue that the card is not being detected by the system occasionally.
Every time the PCI card is not detected, we need reboot the system.

All our 5 Xavier dev kits and PCIe x4 cards have the same issue.
We found several existing issues and are advised to open a new topic there.
(ref: PCIE adapters that require +3.3V not being detected by the system - #4 by vidyas)

Here is system information of our Xavier:

ubuntu@xavier:~/jetsonUtilities$ ./jetsonInfo.py 
NVIDIA Jetson AGX Xavier [16GB]
 L4T 32.3.1 [ JetPack 4.3 ]
   Ubuntu 18.04.5 LTS
   Kernel Version: 4.9.140-rt93
nvdc: nvdcEventThread select failed: Bad file descriptor
 CUDA 10.0.326
   CUDA Architecture: 7.2
 OpenCV version: 3.4.3
   OpenCV Cuda: YES
 CUDNN: 7.6.3.28
 TensorRT: 6.0.1.10
 Vision Works: 1.6.0.500n
 VPI: 0.1.0

Here is output of dmesg when PCI card is NOT detected:

[    4.365128] tegra-pcie-dw 14180000.pcie: Setting init speed to max speed
[    4.366521] OF: PCI: host bridge /pcie@14180000 ranges:
[    4.366544] OF: PCI:    IO 0x38100000..0x381fffff -> 0x38100000
[    4.366552] OF: PCI:   MEM 0x1b40000000..0x1bffffffff -> 0x40000000
[    4.366558] OF: PCI:   MEM 0x1800000000..0x1b3fffffff -> 0x1800000000
[    4.476650] tegra-pcie-dw 14180000.pcie: link is up
[    4.476866] tegra-pcie-dw 14180000.pcie: PCI host bridge to bus 0000:00
[    4.476873] pci_bus 0000:00: root bus resource [bus 00-ff]
[    4.476894] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[    4.476899] pci_bus 0000:00: root bus resource [mem 0x1b40000000-0x1bffffffff] (bus address [0x40000000-0xffffffff])
[    4.476903] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1b3fffffff pref]
[    4.476943] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[    4.477093] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    4.477271] iommu: Adding device 0000:00:00.0 to group 62
[    4.477493] pci 0000:01:00.0: [1987:5012] type 00 class 0x010802
[    4.477634] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[    4.478568] iommu: Adding device 0000:01:00.0 to group 63
[    4.483372] pci 0000:00:00.0: BAR 14: assigned [mem 0x1b40000000-0x1b400fffff]
[    4.483380] pci 0000:01:00.0: BAR 0: assigned [mem 0x1b40000000-0x1b40003fff 64bit]
[    4.483459] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    4.483467] pci 0000:00:00.0:   bridge window [mem 0x1b40000000-0x1b400fffff]
[    4.483496] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    4.483541] pci 0000:01:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  512
[    4.483857] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    4.483864] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    4.483873] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[    4.484063] aer 0000:00:00.0:pcie002: service driver aer loaded
[    4.484459] nvme nvme0: pci function 0000:01:00.0
[    4.484512] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[    4.484846] tegra-pcie-dw 14100000.pcie: Setting init speed to max speed
[    4.485970] OF: PCI: host bridge /pcie@14100000 ranges:
[    4.485989] OF: PCI:    IO 0x30100000..0x301fffff -> 0x30100000
[    4.486019] OF: PCI:   MEM 0x1230000000..0x123fffffff -> 0x40000000
[    4.486024] OF: PCI:   MEM 0x1200000000..0x122fffffff -> 0x1200000000
[    4.596638] tegra-pcie-dw 14100000.pcie: link is up
[    4.596814] tegra-pcie-dw 14100000.pcie: PCI host bridge to bus 0001:00
[    4.596822] pci_bus 0001:00: root bus resource [bus 00-ff]
[    4.596828] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x30100000-0x301fffff])
[    4.596833] pci_bus 0001:00: root bus resource [mem 0x1230000000-0x123fffffff] (bus address [0x40000000-0x4fffffff])
[    4.596850] pci_bus 0001:00: root bus resource [mem 0x1200000000-0x122fffffff pref]
[    4.596874] pci 0001:00:00.0: [10de:1ad2] type 01 class 0x060400
[    4.597003] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[    4.597168] iommu: Adding device 0001:00:00.0 to group 64
[    4.597426] pci 0001:01:00.0: [1b4b:9171] type 00 class 0x010601
[    4.597533] pci 0001:01:00.0: reg 0x10: [io  0x8000-0x8007]
[    4.597702] pci 0001:01:00.0: reg 0x14: [io  0x8040-0x8043]
[    4.597782] pci 0001:01:00.0: reg 0x18: [io  0x8100-0x8107]
[    4.597844] pci 0001:01:00.0: reg 0x1c: [io  0x8140-0x8143]
[    4.597921] pci 0001:01:00.0: reg 0x20: [io  0x800000-0x80000f]
[    4.597984] pci 0001:01:00.0: reg 0x24: [mem 0x00900000-0x009001ff]
[    4.598047] pci 0001:01:00.0: reg 0x30: [mem 0xd0000000-0xd000ffff pref]
[    4.598378] pci 0001:01:00.0: PME# supported from D3hot
[    4.598615] iommu: Adding device 0001:01:00.0 to group 65
[    4.602908] pci 0001:00:00.0: BAR 14: assigned [mem 0x1230000000-0x12300fffff]
[    4.602929] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    4.602940] pci 0001:01:00.0: BAR 6: assigned [mem 0x1230000000-0x123000ffff pref]
[    4.602945] pci 0001:01:00.0: BAR 5: assigned [mem 0x1230010000-0x12300101ff]
[    4.603007] pci 0001:01:00.0: BAR 4: assigned [io  0x100000-0x10000f]
[    4.603067] pci 0001:01:00.0: BAR 0: assigned [io  0x100010-0x100017]
[    4.603148] pci 0001:01:00.0: BAR 2: assigned [io  0x100018-0x10001f]
[    4.603211] pci 0001:01:00.0: BAR 1: assigned [io  0x100020-0x100023]
[    4.603286] pci 0001:01:00.0: BAR 3: assigned [io  0x100024-0x100027]
[    4.603351] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    4.603356] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    4.603375] pci 0001:00:00.0:   bridge window [mem 0x1230000000-0x12300fffff]
[    4.603394] pci 0001:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    4.603491] pci 0001:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  512
[    4.603970] pcieport 0001:00:00.0: Signaling PME through PCIe PME interrupt
[    4.603977] pci 0001:01:00.0: Signaling PME through PCIe PME interrupt
[    4.604003] pcie_pme 0001:00:00.0:pcie001: service driver pcie_pme loaded
[    4.604539] aer 0001:00:00.0:pcie002: service driver aer loaded
[    4.605301] ahci 0001:01:00.0: version 3.0
[    4.605859] ahci 0001:01:00.0: AHCI 0001.0000 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
[    4.605867] ahci 0001:01:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs 
[    4.606531] scsi host1: ahci
[    4.606799] ata1: SATA max UDMA/133 abar m512@0x1230010000 port 0x1230010100 irq 819
[    4.606919] tegra-pcie-dw 14140000.pcie: Setting init speed to max speed
[    4.607996] OF: PCI: host bridge /pcie@14140000 ranges:
[    4.608040] OF: PCI:    IO 0x34100000..0x341fffff -> 0x34100000
[    4.608057] OF: PCI:   MEM 0x12b0000000..0x12bfffffff -> 0x40000000
[    4.608063] OF: PCI:   MEM 0x1280000000..0x12afffffff -> 0x1280000000
[    4.914772] ata1: SATA link down (SStatus 0 SControl 300)
[    5.120862] tegra-pcie-dw 14140000.pcie: link is down
[    5.120977] tegra-pcie-dw 14140000.pcie: PCI host bridge to bus 0003:00
[    5.120983] pci_bus 0003:00: root bus resource [bus 00-ff]
[    5.121005] pci_bus 0003:00: root bus resource [io  0x200000-0x2fffff] (bus address [0x34100000-0x341fffff])
[    5.121012] pci_bus 0003:00: root bus resource [mem 0x12b0000000-0x12bfffffff] (bus address [0x40000000-0x4fffffff])
[    5.121016] pci_bus 0003:00: root bus resource [mem 0x1280000000-0x12afffffff pref]
[    5.121056] pci 0003:00:00.0: [10de:1ad2] type 01 class 0x060400
[    5.121209] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[    5.121383] iommu: Adding device 0003:00:00.0 to group 66
[    5.121726] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[    5.121748] pci 0003:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    5.122080] pcieport 0003:00:00.0: Signaling PME through PCIe PME interrupt
[    5.122092] pcie_pme 0003:00:00.0:pcie001: service driver pcie_pme loaded
[    5.122353] aer 0003:00:00.0:pcie002: service driver aer loaded
[    5.122528] pcie_pme 0003:00:00.0:pcie001: unloading service driver pcie_pme
[    5.122679] aer 0003:00:00.0:pcie002: unloading service driver aer
[    5.122905] pci_bus 0003:01: busn_res: [bus 01-ff] is released
[    5.123005] iommu: Removing device 0003:00:00.0 from group 66
[    5.123045] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[    5.124193] tegra-pcie-dw 14140000.pcie: PCIe link is not up...!
[    5.125118] tegra-pcie-dw 141a0000.pcie: Setting init speed to max speed
[    5.126439] OF: PCI: host bridge /pcie@141a0000 ranges:
[    5.126456] OF: PCI:    IO 0x3a100000..0x3a1fffff -> 0x3a100000
[    5.126463] OF: PCI:   MEM 0x1f40000000..0x1fffffffff -> 0x40000000
[    5.126481] OF: PCI:   MEM 0x1c00000000..0x1f3fffffff -> 0x1c00000000
[    5.236662] tegra-pcie-dw 141a0000.pcie: link is up
[    5.236913] tegra-pcie-dw 141a0000.pcie: PCI host bridge to bus 0005:00
[    5.236920] pci_bus 0005:00: root bus resource [bus 00-ff]
[    5.236927] pci_bus 0005:00: root bus resource [io  0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[    5.236943] pci_bus 0005:00: root bus resource [mem 0x1f40000000-0x1fffffffff] (bus address [0x40000000-0xffffffff])
[    5.236950] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1f3fffffff pref]
[    5.236987] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[    5.237126] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[    5.237275] iommu: Adding device 0005:00:00.0 to group 66
[    5.237632] pci 0005:01:00.0: [12d8:2608] type 01 class 0x060400
[    5.238365] pci 0005:01:00.0: supports D1 D2
[    5.238369] pci 0005:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.238642] iommu: Adding device 0005:01:00.0 to group 67
[    5.316679] pci 0005:01:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[    5.316734] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[    5.316755] pci 0005:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    5.316761] pci 0005:01:00.0: Failed attempting to set the MPS
[    5.316894] pci 0005:01:00.0: Max Payload Size set to  128/ 256 (was  128), Max Read Rq  128
[    5.317217] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[    5.317227] pci 0005:01:00.0: Signaling PME through PCIe PME interrupt
[    5.317236] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[    5.317541] aer 0005:00:00.0:pcie002: service driver aer loaded
[    5.317714] pcieport 0005:01:00.0: of_irq_parse_pci() failed with rc=134
[    5.329639] pcieport 0005:01:00.0: Refused to change power state, currently in D3
[    5.341723] pcie_pme 0005:00:00.0:pcie001: unloading service driver pcie_pme
[    5.341858] aer 0005:00:00.0:pcie002: unloading service driver aer
[    5.342057] iommu: Removing device 0005:01:00.0 from group 67
[    5.342142] pci_bus 0005:01: busn_res: [bus 01-ff] is released
[    5.342260] iommu: Removing device 0005:00:00.0 from group 66
[    5.342326] pci_bus 0005:00: busn_res: [bus 00-ff] is released

And the below is output when PCI card is successfully detected:

[    4.485140] tegra-pcie-dw 14180000.pcie: Setting init speed to max speed
[    4.486664] OF: PCI: host bridge /pcie@14180000 ranges:
[    4.486710] OF: PCI:    IO 0x38100000..0x381fffff -> 0x38100000
[    4.486719] OF: PCI:   MEM 0x1b40000000..0x1bffffffff -> 0x40000000
[    4.486724] OF: PCI:   MEM 0x1800000000..0x1b3fffffff -> 0x1800000000
[    4.595059] tegra-pcie-dw 14180000.pcie: link is up
[    4.595314] tegra-pcie-dw 14180000.pcie: PCI host bridge to bus 0000:00
[    4.595322] pci_bus 0000:00: root bus resource [bus 00-ff]
[    4.595329] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[    4.595346] pci_bus 0000:00: root bus resource [mem 0x1b40000000-0x1bffffffff] (bus address [0x40000000-0xffffffff])
[    4.595351] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1b3fffffff pref]
[    4.595392] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[    4.595530] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    4.595723] iommu: Adding device 0000:00:00.0 to group 62
[    4.596073] pci 0000:01:00.0: [1987:5012] type 00 class 0x010802
[    4.596194] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[    4.597185] iommu: Adding device 0000:01:00.0 to group 63
[    4.601925] pci 0000:00:00.0: BAR 14: assigned [mem 0x1b40000000-0x1b400fffff]
[    4.601933] pci 0000:01:00.0: BAR 0: assigned [mem 0x1b40000000-0x1b40003fff 64bit]
[    4.602049] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    4.602073] pci 0000:00:00.0:   bridge window [mem 0x1b40000000-0x1b400fffff]
[    4.602090] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    4.602152] pci 0000:01:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  512
[    4.602477] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    4.602484] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    4.602510] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[    4.602739] aer 0000:00:00.0:pcie002: service driver aer loaded
[    4.603220] nvme nvme0: pci function 0000:01:00.0
[    4.603308] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[    4.603501] tegra-pcie-dw 14100000.pcie: Setting init speed to max speed
[    4.605360] OF: PCI: host bridge /pcie@14100000 ranges:
[    4.605409] OF: PCI:    IO 0x30100000..0x301fffff -> 0x30100000
[    4.605419] OF: PCI:   MEM 0x1230000000..0x123fffffff -> 0x40000000
[    4.605428] OF: PCI:   MEM 0x1200000000..0x122fffffff -> 0x1200000000
[    4.715062] tegra-pcie-dw 14100000.pcie: link is up
[    4.715276] tegra-pcie-dw 14100000.pcie: PCI host bridge to bus 0001:00
[    4.715283] pci_bus 0001:00: root bus resource [bus 00-ff]
[    4.715290] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x30100000-0x301fffff])
[    4.715311] pci_bus 0001:00: root bus resource [mem 0x1230000000-0x123fffffff] (bus address [0x40000000-0x4fffffff])
[    4.715315] pci_bus 0001:00: root bus resource [mem 0x1200000000-0x122fffffff pref]
[    4.715354] pci 0001:00:00.0: [10de:1ad2] type 01 class 0x060400
[    4.715490] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[    4.715694] iommu: Adding device 0001:00:00.0 to group 64
[    4.716086] pci 0001:01:00.0: [1b4b:9171] type 00 class 0x010601
[    4.716193] pci 0001:01:00.0: reg 0x10: [io  0x8000-0x8007]
[    4.716256] pci 0001:01:00.0: reg 0x14: [io  0x8040-0x8043]
[    4.716319] pci 0001:01:00.0: reg 0x18: [io  0x8100-0x8107]
[    4.716382] pci 0001:01:00.0: reg 0x1c: [io  0x8140-0x8143]
[    4.716458] pci 0001:01:00.0: reg 0x20: [io  0x800000-0x80000f]
[    4.716521] pci 0001:01:00.0: reg 0x24: [mem 0x00900000-0x009001ff]
[    4.716596] pci 0001:01:00.0: reg 0x30: [mem 0xd0000000-0xd000ffff pref]
[    4.716937] pci 0001:01:00.0: PME# supported from D3hot
[    4.717211] iommu: Adding device 0001:01:00.0 to group 65
[    4.721759] pci 0001:00:00.0: BAR 14: assigned [mem 0x1230000000-0x12300fffff]
[    4.721766] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    4.721773] pci 0001:01:00.0: BAR 6: assigned [mem 0x1230000000-0x123000ffff pref]
[    4.721777] pci 0001:01:00.0: BAR 5: assigned [mem 0x1230010000-0x12300101ff]
[    4.721837] pci 0001:01:00.0: BAR 4: assigned [io  0x100000-0x10000f]
[    4.721897] pci 0001:01:00.0: BAR 0: assigned [io  0x100010-0x100017]
[    4.721959] pci 0001:01:00.0: BAR 2: assigned [io  0x100018-0x10001f]
[    4.722048] pci 0001:01:00.0: BAR 1: assigned [io  0x100020-0x100023]
[    4.722109] pci 0001:01:00.0: BAR 3: assigned [io  0x100024-0x100027]
[    4.722181] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    4.722186] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    4.722193] pci 0001:00:00.0:   bridge window [mem 0x1230000000-0x12300fffff]
[    4.722209] pci 0001:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    4.722342] pci 0001:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  512
[    4.722719] pcieport 0001:00:00.0: Signaling PME through PCIe PME interrupt
[    4.722727] pci 0001:01:00.0: Signaling PME through PCIe PME interrupt
[    4.722736] pcie_pme 0001:00:00.0:pcie001: service driver pcie_pme loaded
[    4.722953] aer 0001:00:00.0:pcie002: service driver aer loaded
[    4.723459] ahci 0001:01:00.0: version 3.0
[    4.723968] ahci 0001:01:00.0: AHCI 0001.0000 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
[    4.723976] ahci 0001:01:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs 
[    4.724851] scsi host1: ahci
[    4.725179] ata1: SATA max UDMA/133 abar m512@0x1230010000 port 0x1230010100 irq 819
[    4.725389] tegra-pcie-dw 14140000.pcie: Setting init speed to max speed
[    4.726516] OF: PCI: host bridge /pcie@14140000 ranges:
[    4.726550] OF: PCI:    IO 0x34100000..0x341fffff -> 0x34100000
[    4.726558] OF: PCI:   MEM 0x12b0000000..0x12bfffffff -> 0x40000000
[    4.726563] OF: PCI:   MEM 0x1280000000..0x12afffffff -> 0x1280000000
[    5.033433] ata1: SATA link down (SStatus 0 SControl 300)
[    5.221244] tegra-pcie-dw 14140000.pcie: link is down
[    5.221386] tegra-pcie-dw 14140000.pcie: PCI host bridge to bus 0003:00
[    5.221408] pci_bus 0003:00: root bus resource [bus 00-ff]
[    5.221415] pci_bus 0003:00: root bus resource [io  0x200000-0x2fffff] (bus address [0x34100000-0x341fffff])
[    5.221420] pci_bus 0003:00: root bus resource [mem 0x12b0000000-0x12bfffffff] (bus address [0x40000000-0x4fffffff])
[    5.221424] pci_bus 0003:00: root bus resource [mem 0x1280000000-0x12afffffff pref]
[    5.221462] pci 0003:00:00.0: [10de:1ad2] type 01 class 0x060400
[    5.221605] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[    5.221778] iommu: Adding device 0003:00:00.0 to group 66
[    5.221961] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[    5.221982] pci 0003:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    5.222407] pcieport 0003:00:00.0: Signaling PME through PCIe PME interrupt
[    5.222418] pcie_pme 0003:00:00.0:pcie001: service driver pcie_pme loaded
[    5.222771] aer 0003:00:00.0:pcie002: service driver aer loaded
[    5.222968] pcie_pme 0003:00:00.0:pcie001: unloading service driver pcie_pme
[    5.223095] aer 0003:00:00.0:pcie002: unloading service driver aer
[    5.223330] pci_bus 0003:01: busn_res: [bus 01-ff] is released
[    5.223405] iommu: Removing device 0003:00:00.0 from group 66
[    5.223431] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[    5.224626] tegra-pcie-dw 14140000.pcie: PCIe link is not up...!
[    5.225294] tegra-pcie-dw 141a0000.pcie: Setting init speed to max speed
[    5.226784] OF: PCI: host bridge /pcie@141a0000 ranges:
[    5.226801] OF: PCI:    IO 0x3a100000..0x3a1fffff -> 0x3a100000
[    5.226823] OF: PCI:   MEM 0x1f40000000..0x1fffffffff -> 0x40000000
[    5.226829] OF: PCI:   MEM 0x1c00000000..0x1f3fffffff -> 0x1c00000000
[    5.331073] tegra-pcie-dw 141a0000.pcie: link is up
[    5.331353] tegra-pcie-dw 141a0000.pcie: PCI host bridge to bus 0005:00
[    5.331360] pci_bus 0005:00: root bus resource [bus 00-ff]
[    5.331367] pci_bus 0005:00: root bus resource [io  0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[    5.331372] pci_bus 0005:00: root bus resource [mem 0x1f40000000-0x1fffffffff] (bus address [0x40000000-0xffffffff])
[    5.331376] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1f3fffffff pref]
[    5.331425] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[    5.331589] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[    5.331750] iommu: Adding device 0005:00:00.0 to group 66
[    5.332131] pci 0005:01:00.0: [12d8:2608] type 01 class 0x060400
[    5.332870] pci 0005:01:00.0: supports D1 D2
[    5.332874] pci 0005:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.333193] iommu: Adding device 0005:01:00.0 to group 67
[    5.336371] pci 0005:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    5.336890] pci 0005:02:01.0: [12d8:2608] type 01 class 0x060400
[    5.337673] pci 0005:02:01.0: supports D1 D2
[    5.337677] pci 0005:02:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.338025] iommu: Adding device 0005:02:01.0 to group 68
[    5.338238] pci 0005:02:02.0: [12d8:2608] type 01 class 0x060400
[    5.339017] pci 0005:02:02.0: supports D1 D2
[    5.339021] pci 0005:02:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.339312] iommu: Adding device 0005:02:02.0 to group 69
[    5.339573] pci 0005:02:03.0: [12d8:2608] type 01 class 0x060400
[    5.340434] pci 0005:02:03.0: supports D1 D2
[    5.340438] pci 0005:02:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.340714] iommu: Adding device 0005:02:03.0 to group 70
[    5.340933] pci 0005:02:04.0: [12d8:2608] type 01 class 0x060400
[    5.341698] pci 0005:02:04.0: supports D1 D2
[    5.341702] pci 0005:02:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.342034] iommu: Adding device 0005:02:04.0 to group 71
[    5.342379] pci 0005:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    5.342423] pci 0005:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    5.342467] pci 0005:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    5.342509] pci 0005:02:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    5.343025] pci 0005:03:00.0: [1912:0015] type 00 class 0x0c0330
[    5.343158] pci 0005:03:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[    5.343859] pci 0005:03:00.0: PME# supported from D0 D3hot
[    5.344175] iommu: Adding device 0005:03:00.0 to group 72
[    5.344636] pci_bus 0005:03: busn_res: [bus 03-ff] end is updated to 03
[    5.345094] pci 0005:04:00.0: [1912:0015] type 00 class 0x0c0330
[    5.345241] pci 0005:04:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[    5.345941] pci 0005:04:00.0: PME# supported from D0 D3hot
[    5.346257] iommu: Adding device 0005:04:00.0 to group 73
[    5.346710] pci_bus 0005:04: busn_res: [bus 04-ff] end is updated to 04
[    5.347163] pci 0005:05:00.0: [1912:0015] type 00 class 0x0c0330
[    5.347309] pci 0005:05:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[    5.348029] pci 0005:05:00.0: PME# supported from D0 D3hot
[    5.348307] iommu: Adding device 0005:05:00.0 to group 74
[    5.348777] pci_bus 0005:05: busn_res: [bus 05-ff] end is updated to 05
[    5.349213] pci 0005:06:00.0: [1912:0015] type 00 class 0x0c0330
[    5.349347] pci 0005:06:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[    5.350090] pci 0005:06:00.0: PME# supported from D0 D3hot
[    5.350372] iommu: Adding device 0005:06:00.0 to group 75
[    5.350882] pci_bus 0005:06: busn_res: [bus 06-ff] end is updated to 06
[    5.350904] pci_bus 0005:02: busn_res: [bus 02-ff] end is updated to 06
[    5.351889] pci 0005:00:00.0: BAR 14: assigned [mem 0x1f40000000-0x1f403fffff]
[    5.351896] pci 0005:01:00.0: BAR 14: assigned [mem 0x1f40000000-0x1f403fffff]
[    5.351920] pci 0005:02:01.0: BAR 14: assigned [mem 0x1f40000000-0x1f400fffff]
[    5.351944] pci 0005:02:02.0: BAR 14: assigned [mem 0x1f40100000-0x1f401fffff]
[    5.351948] pci 0005:02:03.0: BAR 14: assigned [mem 0x1f40200000-0x1f402fffff]
[    5.351966] pci 0005:02:04.0: BAR 14: assigned [mem 0x1f40300000-0x1f403fffff]
[    5.351972] pci 0005:03:00.0: BAR 0: assigned [mem 0x1f40000000-0x1f40001fff 64bit]
[    5.352094] pci 0005:02:01.0: PCI bridge to [bus 03]
[    5.352145] pci 0005:02:01.0:   bridge window [mem 0x1f40000000-0x1f400fffff]
[    5.352208] pci 0005:04:00.0: BAR 0: assigned [mem 0x1f40100000-0x1f40101fff 64bit]
[    5.352311] pci 0005:02:02.0: PCI bridge to [bus 04]
[    5.352345] pci 0005:02:02.0:   bridge window [mem 0x1f40100000-0x1f401fffff]
[    5.352398] pci 0005:05:00.0: BAR 0: assigned [mem 0x1f40200000-0x1f40201fff 64bit]
[    5.352466] pci 0005:02:03.0: PCI bridge to [bus 05]
[    5.352493] pci 0005:02:03.0:   bridge window [mem 0x1f40200000-0x1f402fffff]
[    5.352559] pci 0005:06:00.0: BAR 0: assigned [mem 0x1f40300000-0x1f40301fff 64bit]
[    5.352632] pci 0005:02:04.0: PCI bridge to [bus 06]
[    5.352672] pci 0005:02:04.0:   bridge window [mem 0x1f40300000-0x1f403fffff]
[    5.352723] pci 0005:01:00.0: PCI bridge to [bus 02-06]
[    5.352751] pci 0005:01:00.0:   bridge window [mem 0x1f40000000-0x1f403fffff]
[    5.352814] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[    5.352820] pci 0005:00:00.0:   bridge window [mem 0x1f40000000-0x1f403fffff]
[    5.352845] pci 0005:00:00.0: Max Payload Size set to  128/ 256 (was  256), Max Read Rq  512
[    5.352890] pci 0005:01:00.0: Max Payload Size set to  128/ 256 (was  128), Max Read Rq  128
[    5.352933] pci 0005:02:01.0: Max Payload Size set to  128/ 256 (was  128), Max Read Rq  128
[    5.352982] pci 0005:03:00.0: Max Payload Size set to  128/ 128 (was  128), Max Read Rq  512
[    5.353037] pci 0005:02:02.0: Max Payload Size set to  128/ 256 (was  128), Max Read Rq  128
[    5.353085] pci 0005:04:00.0: Max Payload Size set to  128/ 128 (was  128), Max Read Rq  512
[    5.353128] pci 0005:02:03.0: Max Payload Size set to  128/ 256 (was  128), Max Read Rq  128
[    5.353176] pci 0005:05:00.0: Max Payload Size set to  128/ 128 (was  128), Max Read Rq  512
[    5.353219] pci 0005:02:04.0: Max Payload Size set to  128/ 256 (was  128), Max Read Rq  128
[    5.353267] pci 0005:06:00.0: Max Payload Size set to  128/ 128 (was  128), Max Read Rq  512
[    5.353643] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[    5.353666] pci 0005:01:00.0: Signaling PME through PCIe PME interrupt
[    5.353670] pci 0005:02:01.0: Signaling PME through PCIe PME interrupt
[    5.353674] pci 0005:03:00.0: Signaling PME through PCIe PME interrupt
[    5.353677] pci 0005:02:02.0: Signaling PME through PCIe PME interrupt
[    5.353681] pci 0005:04:00.0: Signaling PME through PCIe PME interrupt
[    5.353685] pci 0005:02:03.0: Signaling PME through PCIe PME interrupt
[    5.353688] pci 0005:05:00.0: Signaling PME through PCIe PME interrupt
[    5.353703] pci 0005:02:04.0: Signaling PME through PCIe PME interrupt
[    5.353707] pci 0005:06:00.0: Signaling PME through PCIe PME interrupt
[    5.353715] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[    5.353987] aer 0005:00:00.0:pcie002: service driver aer loaded
[    5.359874] pci 0005:03:00.0: enabling device (0000 -> 0002)

Any advice and comments are welcome!
Thank you very much in advance!

1 Like
2

We guess the suspicious part of dmesg is below (though we don’t have idea what it means):

[    5.317714] pcieport 0005:01:00.0: of_irq_parse_pci() failed with rc=134
[    5.329639] pcieport 0005:01:00.0: Refused to change power state, currently in D3
4

I’m suspecting that the device is not getting ready by the time system goes for enumeration. Although there is a PCIe spec defined way for that through CRS (Configuration Request Retry), I’m not sure why this particular endpoint is not using that (assuming that is the problem).
Could you please try the below patch?

diff --git a/drivers/pci/dwc/pcie-tegra.c b/drivers/pci/dwc/pcie-tegra.c
index 4cd746b8b..06bd6b427 100644
--- a/drivers/pci/dwc/pcie-tegra.c
+++ b/drivers/pci/dwc/pcie-tegra.c
@@ -4499,6 +4499,8 @@ static int tegra_pcie_dw_runtime_resume(struct device *dev)

        config_plat_gpio(pcie, 1);

+       msleep(100);
+
        ret = regulator_enable(pcie->pex_ctl_reg);
        if (ret < 0) {
                dev_err(pcie->dev, "regulator enable failed: %d\n", ret);
5

@vidyas Thank you for sharing the patch!
I tried to apply patch for kernel and installed it to our device and rebooted many times.
We found that Xavier still occasionally did not find PCI express card.

Here is the output of dmesg when PCI express card is not detected.

It looks the same error as we posted before.

of_irq_parse_pci() failed with rc=134

6

@vidyas

Is there any comment on the issue?

FYI, I also tested on the case that the value of msleep is increased to 1000, but it still occurs the same error.
(The error occurs every once in about 20 times booting.)

7

The fact that we are observing errors like “Failed attempting to set the MPS”, “of_irq_parse_pci() failed with rc=134” and “Refused to change power state, currently in D3” tells me that there is something wrong with the device’s readiness w.r.t configuration accesses.
Could you please apply the below patch and check once?
(this patch is different from the one above)

diff --git a/drivers/pci/dwc/pcie-tegra.c b/drivers/pci/dwc/pcie-tegra.c
index 4cd746b8b..f1f0682ae 100644
--- a/drivers/pci/dwc/pcie-tegra.c
+++ b/drivers/pci/dwc/pcie-tegra.c
@@ -2856,7 +2856,7 @@ static int tegra_pcie_dw_host_init(struct pcie_port *pp)
        val |= APPL_PINMUX_PEX_RST;
        writel(val, pcie->appl_base + APPL_PINMUX);

-       msleep(100);
+       msleep(1000);

        val = readl(pci->dbi_base + CFG_LINK_STATUS_CONTROL);
        while (!(val & CFG_LINK_STATUS_DLL_ACTIVE)) {
8

it doesn’t work without external sata power from separte power source
we used computer box sata power to get the startech usb pci hub powered

9

@vidyas Thank you for another patch. Let me try it…

10

@Andrey1984 Thank you for your advice! Yes, and I’m using the PCI card with SATA power input. (Though the error still occurs.)

11

which
model of the startech hub are you using?
we were not able to run into any issues with our 4 port solution for pcie

12

@Andrey1984 We are using Startech PEXUSB3S44V model. It works almost perfect except for the issue which occurs on boot less than once in twenty times.

1 Like