Xavier not routing PCI interrupts across PEX8112 bridge

BConklin · July 23, 2019, 5:26pm

On a TX1 system with a PCIe to PCI bridge the PCI / interrupt configuration looks like this:

TX1, running Ubuntu, with kernel version 4.4.38
Interrupt for the 8112 and the 2 DDC parts are all at #368

On Xavier the 8112 interrupt says IRQ38, and the 2 DDC devices are at IRQ0, which is wrong!

This is lspci on a working Jetson-TK1. Notice that the interrupts are all assigned to IRQ 368:

nvidia@tegra-ubuntu:~/DDC/acextremeSDK_4.0.1.3/samples$ lspci -vv

00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1) (prog-if 00 [Normal decode])
                Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B DisINTx-
                Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
                Latency: 0
                Interrupt: pin A routed to IRQ 368

                Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
                Memory behind bridge: 13000000-130fffff
                Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
                BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
                Capabilities: <access denied>
                Kernel driver in use: pcieport

01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
                Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
                Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
                Latency: 0
                Interrupt: pin A routed to IRQ 368

                Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
                Memory behind bridge: 13000000-130fffff
                Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
                BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
                Capabilities: <access denied>
 
02:00.0 Communication controller: ILC Data Device Corp Device 1a00 (rev 10)
                Subsystem: ILC Data Device Corp Device 1a00
                Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
                Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
                Latency: 0
                Interrupt: pin A routed to IRQ 368

                Region 0: Memory at 13000000 (32-bit, non-prefetchable) 
                Region 1: Memory at 13080000 (32-bit, non-prefetchable) 
 
02:01.0 Communication controller: ILC Data Device Corp Device 1a00 (rev 10)
                Subsystem: ILC Data Device Corp Device 1a00
                Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
                Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
                Latency: 0
                Interrupt: pin A routed to IRQ 368

                Region 0: Memory at 13040000 (32-bit, non-prefetchable) 
                Region 1: Memory at 13081000 (32-bit, non-prefetchable)

On Jetson Xavier 4.9.140-tegra dev kit
This is lspci on Jetson-Xavier. Notice that the bridge has IRQ 38 and the parts behind the bridge are unassigned:

0005:01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 38

        Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
        Memory behind bridge: 3a200000-3a2fffff
        Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: ...
               ...
               ...
        Capabilities: [100 v1] Power Budgeting <?>

0005:02:00.0 Communication controller: ILC Data Device Corp Device 1a00 (rev 10)
        Subsystem: ILC Data Device Corp Device 1a00
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 0

        Region 0: Memory at 3a200000 (32-bit, non-prefetchable) 
        Region 1: Memory at 3a280000 (32-bit, non-prefetchable) 

0005:02:01.0 Communication controller: ILC Data Device Corp Device 1a00 (rev 10)
        Subsystem: ILC Data Device Corp Device 1a00
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 0

        Region 0: Memory at 3a240000 (32-bit, non-prefetchable) 
        Region 1: Memory at 3a281000 (32-bit, non-prefetchable)

There is no 0 in /proc/interrupts

dmesg says nothing about interrupts

[    7.626360] pci_bus 0005:00: root bus resource [bus 00-ff]
[    7.626383] pci_bus 0005:00: root bus resource [io  0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[    7.626386] pci_bus 0005:00: root bus resource [mem 0x3a200000-0x3bffffff]
[    7.626389] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1fffffffff pref]
[    7.626488] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[    7.626656] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[    7.626847] iommu: Adding device 0005:00:00.0 to group 65
[    7.627257] pci 0005:01:00.0: [10b5:8112] type 01 class 0x060400
[    7.627884] pci 0005:01:00.0: supports D1
[    7.627886] pci 0005:01:00.0: PME# supported from D0 D1 D3hot
[    7.628113] iommu: Adding device 0005:01:00.0 to group 66
[    7.628213] pci 0005:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    7.628240] pci 0005:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    7.628618] pci 0005:02:00.0: [4ddc:1a00] type 00 class 0x078000
[    7.628723] pci 0005:02:00.0: reg 0x10: [mem 0x00000000-0x0003ffff]
[    7.628777] pci 0005:02:00.0: reg 0x14: [mem 0x00000000-0x00000fff]
[    7.629259] iommu: Adding device 0005:02:00.0 to group 67
[    7.629377] pci 0005:02:01.0: [4ddc:1a00] type 00 class 0x078000
[    7.629468] pci 0005:02:01.0: reg 0x10: [mem 0x00000000-0x0003ffff]
[    7.629524] pci 0005:02:01.0: reg 0x14: [mem 0x00000000-0x00000fff]
[    7.630019] iommu: Adding device 0005:02:01.0 to group 68
[    7.630409] pci_bus 0005:02: busn_res: [bus 02-ff] end is updated to 02
[    7.631087] pci 0005:00:00.0: BAR 14: assigned [mem 0x3a200000-0x3a2fffff]
[    7.631092] pci 0005:01:00.0: BAR 14: assigned [mem 0x3a200000-0x3a2fffff]
[    7.631096] pci 0005:02:00.0: BAR 0: assigned [mem 0x3a200000-0x3a23ffff]
[    7.631120] pci 0005:02:01.0: BAR 0: assigned [mem 0x3a240000-0x3a27ffff]
[    7.631143] pci 0005:02:00.0: BAR 1: assigned [mem 0x3a280000-0x3a280fff]
[    7.631166] pci 0005:02:01.0: BAR 1: assigned [mem 0x3a281000-0x3a281fff]
[    7.631190] pci 0005:01:00.0: PCI bridge to [bus 02]
[    7.631223] pci 0005:01:00.0:   bridge window [mem 0x3a200000-0x3a2fffff]
[    7.631286] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[    7.631291] pci 0005:00:00.0:   bridge window [mem 0x3a200000-0x3a2fffff]
[    7.631305] pci 0005:00:00.0: Max Payload Size set to  128/ 256 (was  256), Max Read Rq  512
[    7.631357] pci 0005:01:00.0: Max Payload Size set to  128/ 128 (was  128), Max Read Rq  512
[    7.631548] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[    7.631550] pci 0005:01:00.0: Signaling PME through PCIe PME interrupt
[    7.631551] pci 0005:02:00.0: Signaling PME through PCIe PME interrupt
[    7.631572] pci 0005:02:01.0: Signaling PME through PCIe PME interrupt
[    7.631584] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[    7.631659] aer 0005:00:00.0:pcie002: service driver aer loaded

These devices / drivers work on TK and TX, but don’t work on Xavier

vidyas · July 24, 2019, 4:20am

Is the issue here only with IRQ being 0 on Xavier or even the functionality is also broken on Xavier?
Also, are these devices use only legacy interrupt and not MSI/MSI-X?

BConklin · July 24, 2019, 3:40pm

The IRQ number is invalid. This board works on other Jetsons, just not Xavier.

The library that opens the driver does not even try to open the device if the interrupt is invalid. This is a PCI Express to PCI bridge with 2 PCI chips on it. So they are not MSI capable. The bridge could be programmed to convert to MSI, but they would have to be assigned a value, and they are not.

vidyas · July 24, 2019, 5:57pm

Can you please give the output of ‘cat /proc/interrupts’?
Actually, IRQ number doesn’t really matter here. Can you please bypass the check in the library and try?

BConklin · July 24, 2019, 7:52pm

The library is a proprietary binary.

alphi@alphi-xavier:~$ CPU0 CPU1 CPU2 2: 35085753 14638339 5: 8689473 0 6: 0 0 7: 0 0 8: 0 0 9: 0 0 10: 0 0 11: 0 0 12: 0 0 13: 0 0 14: 0 0 15: 0 0 16: 0 0 17: 0 0 18: 0 0 19: 0 0 20: 0 0 21: 0 0 22: 0 0 23: 0 0 24: 0 0 25: 0 0 26: 0 0 27: 0 0 28: 0 0 29: 0 0 30: 0 0 31: 0 0 32: 1 0 33: 144 0 34: 1 0 35: 0 0 36: 0 0 37: 0 0 38: 0 0 39: 0 0 40: 2655084 0 42: 1015200 0 43: 33990 0 50: 17 0 53: 0 0 54: 140245 0 55: 0 0 56: 1 0 57: 0 0 58: 0 0 59: 354 0 60: 0 0 61: 0 0 62: 0 0 63: 131 0 64: 710 0 65: 1 0 68: 1975 0 69: 0 0 70: 113 0 71: 2888660 0 72: 0 0 73: 0 0 74: 0 0 75: 0 0 76: 85849 0 77: 0 0 78: 0 0 79: 0 0 80: 0 0 87: 0 0 88: 0 0 97: 0 0 98: 0 0 99: 0 0 100: 0 0 102: 0 0 113: 29 0 117: 0 0 119: 0 0 120: 0 0 121: 0 0 122: 0 0 123: 0 0 124: 0 0 125: 0 0 126: 0 0 127: 0 0 128: 0 0 129: 0 0 130: 0 0 131: 0 0 132: 0 0 133: 0 0 134: 0 0 135: 0 0 136: 0 0 137: 0 0 138: 0 0 139: 0 0 140: 0 0 141: 0 0 142: 0 0 143: 0 0 144: 0 0 145: 0 0 146: 0 0 147: 0 0 148: 0 0 149: 0 0 247: 0 0 251: 4 0 254: 0 0 257: 0 0 297: 3 0 348: 4 0 391: 0 0 437: 21 0 459: 0 0 463: 2 0 467: 14891 0 471: 46697 0 472: 2148 0 473: 5 0 474: 0 0 475: 6739332 0 476: 0 0 477: 0 0 478: 0 0 479: 0 0 480: 0 0 481: 0 0 482: 0 0 483: 0 0 484: 0 0 485: 0 0 486: 0 0 487: 0 0 488: 0 0 489: 0 0 490: 0 0 491: 0 0 492: 0 0 493: 0 0 494: 0 0 496: 0 0 500: 0 0 501: 0 0 505: 0 0 506: 0 0 529: 115 0 530: 106 0 561: 0 0 563: 66 0 564: 13 0 565: 10 0 566: 23 0 567: 4 0 568: 1 0 569: 27 0 595: 0 0 IPI0: 6394520 9076963 IPI1: 2170 IPI2: 0 0 IPI3: 0 0 IPI4: 5 2 IPI5: 0 0 Err: 0 cat /proc/interrupts
CPU3
17165610 19137934 GICv2 30 Level arch_timer
0 0 GICv2 208 Level hsp
0 0 GICv2 202 Level arm-smmu global fault
0 0 GICv2 203 Level arm-smmu global fault
0 0 GICv2 264 Level arm-smmu global fault
0 0 GICv2 265 Level arm-smmu global fault
0 0 GICv2 272 Level arm-smmu global fault
0 0 GICv2 273 Level arm-smmu global fault
0 0 GICv2 368 Level tegra-p2u-intr
0 0 GICv2 369 Level tegra-p2u-intr
0 0 GICv2 370 Level tegra-p2u-intr
0 0 GICv2 371 Level tegra-p2u-intr
0 0 GICv2 372 Level tegra-p2u-intr
0 0 GICv2 373 Level tegra-p2u-intr
0 0 GICv2 374 Level tegra-p2u-intr
0 0 GICv2 375 Level tegra-p2u-intr
0 0 GICv2 376 Level tegra-p2u-intr
0 0 GICv2 377 Level tegra-p2u-intr
0 0 GICv2 253 Level tegra-p2u-intr
0 0 GICv2 254 Level tegra-p2u-intr
0 0 GICv2 378 Level tegra-p2u-intr
0 0 GICv2 379 Level tegra-p2u-intr
0 0 GICv2 380 Level tegra-p2u-intr
0 0 GICv2 381 Level tegra-p2u-intr
0 0 GICv2 382 Level tegra-p2u-intr
0 0 GICv2 383 Level tegra-p2u-intr
0 0 GICv2 235 Level tegra-p2u-intr
0 0 GICv2 252 Level tegra-p2u-intr
0 0 GICv2 104 Level tegra-pcie-intr, PCIe PME, aerdrv
0 0 GICv2 105 Level tegra-pcie-msi
0 0 GICv2 77 Level tegra-pcie-intr, PCIe PME, aerdrv
0 0 GICv2 78 Level tegra-pcie-msi
0 0 GICv2 81 Level tegra-pcie-intr
0 0 GICv2 82 Level tegra-pcie-msi
0 0 GICv2 85 Level tegra-pcie-intr, PCIe PME, aerdrv
0 0 GICv2 86 Level tegra-pcie-msi
0 0 GICv2 226 Level ether_qos.common_irq
0 0 GICv2 222 Level 2490000.ether_qos.rx0
0 0 GICv2 218 Level 2490000.ether_qos.tx0
0 0 GICv2 144 Level 3100000.serial
0 0 GICv2 152 Level combined_uart rx
0 0 GICv2 97 Level mmc0
0 0 GICv2 94 Level mmc1
0 0 GICv2 76 Level ufshcd
0 0 GICv2 69 Level c260000.spi
0 0 GICv2 57 Level 3160000.i2c
0 0 GICv2 58 Level c240000.i2c
0 0 GICv2 59 Level 3180000.i2c
0 0 GICv2 60 Level 3190000.i2c
0 0 GICv2 62 Level 31b0000.i2c
0 0 GICv2 63 Level 31c0000.i2c
0 0 GICv2 64 Level c250000.i2c
0 0 GICv2 65 Level 31e0000.i2c
0 0 GICv2 193 Level snd_hda_tegra
0 0 GICv2 51 Level bc00000.rtcpu
0 0 GICv2 242 Level d230000.actmon
0 0 GICv2 297 Level host_syncpt
0 0 GICv2 295 Level host_status
0 0 GICv2 238 Level vic
0 0 GICv2 268 Level nvdla0
0 0 GICv2 269 Level nvdla1
0 0 GICv2 185 Level 15200000.nvdisplay
0 0 GICv2 186 Level 15210000.nvdisplay
0 0 GICv2 187 Level 15220000.nvdisplay
0 0 GICv2 191 Level tegra_dp
0 0 GICv2 192 Level tegra_dp
0 0 GICv2 266 Level pva-isr
0 0 GICv2 267 Level pva-isr
0 0 GICv2 270 Level noc_nonsecure_irq
0 0 GICv2 271 Level noc_secure_irq
0 0 PM 42 Level tegra_rtc
0 0 GICv2 255 Level mc_status
0 0 GICv2 165 Level c150000.tegra-hsp
0 0 GICv2 214 Level b950000.tegra-hsp, b950000.tegra-hsp, b950000.tegra-hsp
0 0 GICv2 315 Level 3ad0000.se_elp
0 0 GICv2 108 Level gpcdma.0
0 0 GICv2 109 Level gpcdma.1
0 0 GICv2 110 Level gpcdma.2
0 0 GICv2 111 Level gpcdma.3
0 0 GICv2 112 Level gpcdma.4
0 0 GICv2 113 Level gpcdma.5
0 0 GICv2 114 Level gpcdma.6
0 0 GICv2 115 Level gpcdma.7
0 0 GICv2 116 Level gpcdma.8
0 0 GICv2 117 Level gpcdma.9
0 0 GICv2 118 Level gpcdma.10
0 0 GICv2 119 Level gpcdma.11
0 0 GICv2 120 Level gpcdma.12
0 0 GICv2 121 Level gpcdma.13
0 0 GICv2 122 Level gpcdma.14
0 0 GICv2 123 Level gpcdma.15
0 0 GICv2 124 Level gpcdma.16
0 0 GICv2 125 Level gpcdma.17
0 0 GICv2 126 Level gpcdma.18
0 0 GICv2 127 Level gpcdma.19
0 0 GICv2 128 Level gpcdma.20
0 0 GICv2 129 Level gpcdma.21
0 0 GICv2 130 Level gpcdma.22
0 0 GICv2 131 Level gpcdma.23
0 0 GICv2 132 Level gpcdma.24
0 0 GICv2 133 Level gpcdma.25
0 0 GICv2 134 Level gpcdma.26
0 0 GICv2 135 Level gpcdma.27
0 0 GICv2 136 Level gpcdma.28
0 0 GICv2 137 Level gpcdma.29
0 0 GICv2 138 Level gpcdma.30
0 0 tegra-gpio 48 Edge force-recovery
0 0 tegra-gpio 52 Level phy_interrupt
0 0 tegra-gpio 55 Edge 3400000.sdhci cd
0 0 tegra-gpio 58 Level tmp451
0 0 tegra-gpio 98 Edge 15200000.nvdisplay
0 0 tegra-gpio 149 Edge rt5659
0 0 tegra-gpio 192 Edge bluetooth hostwake
0 0 tegra-gpio-aon 10 Level ccg_irq
0 0 tegra-gpio-aon 32 Edge ufs_cd_gpio
0 0 tegra-gpio-aon 36 Edge power-key
0 0 GICv2 39 Level 30c0000.watchdog
0 0 GICv2 198 Level 3550000.xudc
0 0 PM 195 Level xhci-hcd:usb1
0 0 PM 196 Level 3610000.xhci
0 0 PM 199 Level 3610000.xhci
0 0 GICv2 102 Level gk20a_stall
0 0 GICv2 103 Level gk20a_nonstall
0 0 GICv2 424 Level ras-fhi
0 0 GICv2 425 Level ras-fhi
0 0 GICv2 426 Level ras-fhi
0 0 GICv2 427 Level ras-fhi
0 0 GICv2 428 Level ras-fhi
0 0 GICv2 429 Level ras-fhi
0 0 GICv2 430 Level ras-fhi
0 0 GICv2 431 Level ras-fhi
0 0 GICv2 262 Level noc_nonsecure_irq
0 0 GICv2 263 Level noc_secure_irq
0 0 GICv2 292 Level noc_nonsecure_irq
0 0 GICv2 204 Level noc_secure_irq
0 0 GICv2 294 Level noc_nonsecure_irq
0 0 GICv2 206 Level noc_secure_irq
0 0 GICv2 291 Level noc_nonsecure_irq
0 0 GICv2 207 Level noc_secure_irq
0 0 GICv2 293 Level noc_nonsecure_irq
0 0 GICv2 205 Level noc_secure_irq
0 0 PM 241 Edge max77620-top
0 0 max77620-top 3 Edge max77620-gpio
0 0 max77620-top 4 Edge max77686-rtc
0 0 max77620-top 8 Edge max77620-thermal
0 0 max77620-top 9 Edge max77620-thermal
0 0 agic-controller 32 Level
0 0 agic-controller 33 Level
0 0 max77686-rtc 1 Edge rtc-alarm1
0 0 PCI-MSI 0 Edge nvme0q0, nvme0q1
0 0 PCI-MSI 1 Edge nvme0q2
0 0 PCI-MSI 2 Edge nvme0q3
0 0 PCI-MSI 3 Edge nvme0q4
0 0 PCI-MSI 4 Edge
0 0 PCI-MSI 5 Edge
0 0 PCI-MSI 6 Edge
0 0 PCI-MSI 0 Edge ahci[0001:01:00.0]
7892575 7461853 Rescheduling interrupts
2101 17227 20319 Function call interrupts
0 0 CPU stop interrupts
0 0 Timer broadcast interrupts
4 2 IRQ work interrupts
0 0 CPU wake-up interrupts

vidyas · July 25, 2019, 10:02am

There seems to be something wrong. I don’t see any ISR registered for legacy interrupt of the controller.
Can you please give info about the release you are using? like 32.2 or 32.1 etc…?

BConklin · July 25, 2019, 3:30pm

JetPack 4.2
git tag was tegra-l4t-r32.1
I added uio_pci_generic=m and re-compiled. Those devices work.
Linux alphi-xavier 4.9.140-tegra #2 SMP PREEMPT Wed Jul 17 12:56:13 MST 2019 aarch64 aarch64 aarch64 GNU/Linux

alphi@alphi-xavier:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

BConklin · July 25, 2019, 3:32pm

Yes, looks like it’s not routing PCI interrupts across the PEX8112 bridge.

vidyas · July 25, 2019, 5:26pm

So, is this more of your setup issue where you missed setting “uio_pci_generic=m” ??

BConklin · July 25, 2019, 5:40pm

No, the problem is “Xavier not routing PCI interrupts across PEX8112 bridge”.

Me successfully building a .ko has nothing to do with it.

I was just letting you know that this a compiled kernel with defconfig except for one line.

From the git tag tegra-l4t-r32.1. Because you asked the question.

Once again, on this dev kit, the interrupts are not being assigned on the PCI side of the bridge. We have other Jetson TK1 and TX2 dev kits that work.

BConklin · July 26, 2019, 6:49pm

Anyone?

vidyas · July 31, 2019, 1:09pm

Did you happen to try the same kernel version on both Jetson-AGX and TX2 and it didn’t work on Jetson-AGX but worked on TX2? If so, that confirms that there is nothing wrong in the kernel sub-system at least.
Unfortunately, we don’t have this kind of switch to repro the issue and looking at the code, it seems things are fine. Is it possible to give the link of the switch from where you purchased this switch?

BConklin · July 31, 2019, 3:38pm

The PLX chips (now Broadcom) are the backbone of most computer PCI systems.
https://www.broadcom.com/products/pcie-switches-bridges/pcie-bridges/pex8112

This is the first time that I have put it into any kind of PCI system and had any issues.

abhishekxyz17 · August 1, 2019, 5:49am

I faced similar issue. Waiting for a good solution. Meanwhile searching the web.

BConklin · August 13, 2019, 6:14pm

Are legacy PCI interrupts supported on jetson Xavier?

vidyas · August 14, 2019, 5:52am

Yes. Legacy interrupts are supported. To verify that, ‘pci=nomsi’ can be added to the kernel command line to let the kernel know that only Legacy interrupts need to be enabled.

BConklin · August 14, 2019, 9:34pm

We took a working TX2i with L4T 28.2 and flashed it with Our Xavier host with L4T 32.1, and the interrupt routes. Both kernels use int 381 for the bridge and both PCI devices. The exact same kernel on the Xavier puts int 38 on the bridge and 0? on the PCI devices behind the bridge.

The ONLY difference here is the SOC specific drivers in the device tree.

So, we have this board working on TK1, TX1, TX2, but not Xavier. It took us almost a month, but we did it.

What else do I have to do to get someone help fix the Xavier specific drivers?

BConklin · August 19, 2019, 11:54pm

Anyone?

BConklin · August 20, 2019, 11:46pm

Xavier just doesn’t support PCI interrupts. It won’t until you fix your driver bug Vidya.

In my driver probe function and in lspci and in the sys irq file, there is a zero where the correct interrupt number should be. The Tegra194 drivers have put that zero, or probably put nothing.

I did a hack and replaced the u16Irq in the device context of my driver with the 38 that it should have before the request_irq() and that made everything work. The Tegra194 PCIe driver is not storing that irq number, so the driver and lspci just grab a zero.

vidyas · August 21, 2019, 6:24am

I think I found the issue.
Can you please apply the following patch and update your observations?

diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
index 192e7b681b96..11167c65ca37 100644
--- a/drivers/pci/pci-driver.c
+++ b/drivers/pci/pci-driver.c
@@ -412,6 +412,8 @@ static int pci_device_probe(struct device *dev)
        struct pci_dev *pci_dev = to_pci_dev(dev);
        struct pci_driver *drv = to_pci_driver(dev->driver);
 
+       pci_assign_irq(pci_dev);
+
        error = pcibios_alloc_irq(pci_dev);
        if (error < 0)
                return error;

Topic		Replies	Views
Xavier AGX PCIe configuration Jetson AGX Xavier pcie , kernel	22	2877	April 13, 2022
Use PCIe to communicate between two Xaviers: RP phy dosen't up Jetson AGX Xavier	17	2076	October 6, 2021
PCIe Bus Error message Jetson TK1	40	8483	October 18, 2021
Root-endpoint PCIe communication between Xaviers with a bridge in the way Jetson AGX Xavier pcie	10	1721	October 18, 2021
Jetson AGX Xavier Pcie(Root) detection of Device needs jetson reboot Jetson AGX Xavier pcie , fpga	25	2015	July 12, 2023
Boot stuck while enumerating NVMe via PFX switch, seems to be PCIe driver issue Jetson Xavier NX pcie , nvbugs	70	2277	April 13, 2024
Xavier can't maintain Network connection Jetson AGX Xavier wifi	3	663	October 5, 2020
PCIe not being recognized Jetson TX2	5	2078	October 18, 2021
The ethernet interface over PCIe(endpoint mode) not work Jetson AGX Xavier pcie , nvbugs	59	3893	September 1, 2023
Jetson TX2 PCIe not detecting endpoint Jetson TX2 pcie	16	1743	October 3, 2021

Xavier not routing PCI interrupts across PEX8112 bridge

Related topics