Hi, I’m running a Jetpack 6 R36.4.3 Orin AGX board with a custom carrier, and attaching an external x8 PCIe device that requires a large BAR1 region while the Orin is in root-complex mode. The BAR1 space it requires is larger than the default carveout for the C5 controller (x8 lanes), so PCIe resource allocation fails during enumeration.
For context the end-point device is a L40s Nvidia dGPU and requests a 64GB BAR1 size for its VRAM aperture. I’d previously successfully used a consumer RTX6000 dGPU as well, but this only required a 256MB BAR1 size and therefore worked with the default device tree entry for this C5 controller.
# dmesg enumeration logs from boot:
[ 4.752958] tegra194-pcie 141a0000.pcie: Adding to iommu group 4
[ 4.865845] tegra194-pcie 141a0000.pcie: host bridge /bus@0/pcie@141a0000 ranges:
[ 4.865866] tegra194-pcie 141a0000.pcie: MEM 0x2800000000..0x2b27ffffff -> 0x2800000000
[ 4.865873] tegra194-pcie 141a0000.pcie: MEM 0x2b28000000..0x2b2fffffff -> 0x0040000000
[ 4.865878] tegra194-pcie 141a0000.pcie: IO 0x003a100000..0x003a1fffff -> 0x003a100000
[ 4.866785] tegra194-pcie 141a0000.pcie: iATU unroll: enabled
[ 4.866791] tegra194-pcie 141a0000.pcie: Detected iATU regions: 8 outbound, 2 inbound
[ 5.072457] tegra194-pcie 141a0000.pcie: Link up
[ 5.083780] tegra194-pcie 141a0000.pcie: Link up
[ 5.083825] tegra194-pcie 141a0000.pcie: PCI host bridge to bus 0005:00
[ 5.083829] pci_bus 0005:00: root bus resource [io 0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[ 5.083831] pci_bus 0005:00: root bus resource [mem 0x2b28000000-0x2b2fffffff] (bus address [0x40000000-0x47ffffff])
[ 5.083834] pci_bus 0005:00: root bus resource [bus 00-ff]
[ 5.083836] pci_bus 0005:00: root bus resource [mem 0x2800000000-0x2b27ffffff pref]
[ 5.083873] pci 0005:00:00.0: [10de:229a] type 01 class 0x060400
[ 5.084014] pci 0005:00:00.0: PME# supported from D0 D3hot
[ 5.087540] pci 0005:01:00.0: [10de:26b9] type 00 class 0x030200
[ 5.087669] pci 0005:01:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[ 5.087766] pci 0005:01:00.0: reg 0x14: [mem 0x00000000-0xfffffffff 64bit pref]
[ 5.087863] pci 0005:01:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff 64bit pref]
[ 5.088138] pci 0005:01:00.0: Enabling HDA controller
[ 5.088906] pci 0005:01:00.0: PME# supported from D0 D3hot
[ 5.089716] pci 0005:01:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0005:00:00.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 5.092984] pci 0005:00:00.0: BAR 9: no space for [mem size 0x1800000000 64bit pref]
[ 5.092987] pci 0005:00:00.0: BAR 9: failed to assign [mem size 0x1800000000 64bit pref]
[ 5.092988] pci 0005:00:00.0: BAR 8: assigned [mem 0x2b28000000-0x2b28ffffff]
[ 5.092991] pci 0005:01:00.0: BAR 1: no space for [mem size 0x1000000000 64bit pref]
[ 5.092992] pci 0005:01:00.0: BAR 1: failed to assign [mem size 0x1000000000 64bit pref]
[ 5.092994] pci 0005:01:00.0: BAR 3: no space for [mem size 0x02000000 64bit pref]
[ 5.092995] pci 0005:01:00.0: BAR 3: failed to assign [mem size 0x02000000 64bit pref]
[ 5.092997] pci 0005:01:00.0: BAR 0: assigned [mem 0x2b28000000-0x2b28ffffff]
[ 5.093020] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[ 5.093024] pci 0005:00:00.0: bridge window [mem 0x2b28000000-0x2b28ffffff]
[ 5.093097] pcieport 0005:00:00.0: Adding to iommu group 4
[ 5.093172] pcieport 0005:00:00.0: PME: Signaling with IRQ 227
[ 5.094233] pcieport 0005:00:00.0: AER: enabled with IRQ 227
From my understanding, I gather that I need to edit the ‘ranges’ field in the device tree entry for this C5 controller, to allow it to allocate a larger BAR in address space. I want to know how to do so without conflicting with other PCIe controllers or hitting some system limits.
Some questions:
Q1. Although my EP device requests a 64GB BAR1, lspci shows that it supports smaller sizes. Is there a way on Jetson to force a smaller allocation, e.g: 256MB? Or would this have to be done on the GPU driver side instead of from the Jetson root port?
# Full lspci output linked at the end:
$ sudo lspci -vv -s 0005:01:00.0
0005:01:00.0 3D controller: NVIDIA Corporation Device 26b9 (rev a1)
Subsystem: NVIDIA Corporation Device 1851
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 0
Region 0: Memory at 2b28000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Region 1: Memory at <unassigned> (64-bit, prefetchable) [disabled]
Region 3: Memory at <unassigned> (64-bit, prefetchable) [disabled]
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 64GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB 64GB
BAR 3: current size: 32MB, supported: 32MB
Q2. I notice that the C5 controller address is fixed (e.g: 0x141a0000), but is there a way to change the base address for the BAR1 region, so that it can grow into the higher 64-bit address space and not conflict with the address range of other PCIe controllers in the SoC?
# Can I relocate this pcie@141a0000 start address after that of the pcie@141e0000 end address to use higher address space without conflict?
2800000000-2b27ffffff : pcie@141a0000
2b28000000-2b2fffffff : pcie@141a0000
2b28000000-2b28ffffff : PCI Bus 0005:01
2b28000000-2b28ffffff : 0005:01:00.0
3000000000-3227ffffff : pcie@141e0000
3228000000-322fffffff : pcie@141e0000
3228000000-32280fffff : PCI Bus 0007:01
3228000000-3228003fff : 0007:01:00.0
3228000000-3228003fff : nvme
Q3. Is there some upper-limit on Orin for a BAR AMAP? The default for this C5 range was12.6GB, but I couldn’t find why that’s the case.
Q4: I’m assuming that I don’t need a working dGPU driver to get PCIe enumeration / BAR region allocation to succeed. Is this a correct assumption?
Addtional Info:
lspci -vvv output
$ sudo lspci -vv -s 0005:01:00.0
0005:01:00.0 3D controller: NVIDIA Corporation Device 26b9 (rev a1)
Subsystem: NVIDIA Corporation Device 1851
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 0
Region 0: Memory at 2b28000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Region 1: Memory at <unassigned> (64-bit, prefetchable) [disabled]
Region 3: Memory at <unassigned> (64-bit, prefetchable) [disabled]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Null
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s (ok), Width x8 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Via message, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
Vector table: BAR=0 offset=00b90000
PBA: BAR=0 offset=00ba0000
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [250 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [258 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us
L1SubCtl2: T_PwrOn=10us
Capabilities: [128 v1] Power Budgeting <?>
Capabilities: [420 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 64GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB 64GB
BAR 3: current size: 32MB, supported: 32MB
Capabilities: [bcc v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 32, Total VFs: 32, Number of VFs: 0, Function Dependency Link: 00
VF offset: 4, stride: 1, Device ID: 26b9
Supported Page Size: 00000573, System Page Size: 00000001
Region 1: Memory at 0000000000000000 (64-bit, prefetchable)
Region 3: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [c14 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [c1c v1] Physical Layer 16.0 GT/s <?>
Capabilities: [d00 v1] Lane Margining at the Receiver <?>
Capabilities: [e00 v1] Data Link Feature <?>
Kernel modules: nouveau, nvidia_drm, nvidia%
proc/iomem contents
01700000-0170ffff : 2c00000.memory-controller ch8
01710000-0171ffff : 2c00000.memory-controller ch9
01720000-0172ffff : 2c00000.memory-controller ch10
01730000-0173ffff : 2c00000.memory-controller ch11
01740000-0174ffff : 2c00000.memory-controller ch12
01750000-0175ffff : 2c00000.memory-controller ch13
01760000-0176ffff : 2c00000.memory-controller ch14
01770000-0177ffff : 2c00000.memory-controller ch15
02080000-021a0fff : 2080000.timer timer@2080000
02200000-0220ffff : 2200000.gpio security
02210000-0221ffff : 2200000.gpio gpio
0242c000-0242cffe : nvidia
02430000-024490ff : 2430000.pinmux pinmux@2430000
02600000-0280ffff : 2600000.dma-controller dma-controller@2600000
02900800-02900fff : 2900800.ahub ahub@2900800
02901000-029010ff : 2901000.i2s i2s@2901000
02901100-029011ff : 2901100.i2s i2s@2901100
02901200-029012ff : 2901200.i2s i2s@2901200
02901300-029013ff : 2901300.i2s i2s@2901300
02901400-029014ff : 2901400.i2s i2s@2901400
02901500-029015ff : 2901500.i2s i2s@2901500
02902000-029021ff : 2902000.sfc sfc@2902000
02902200-029023ff : 2902200.sfc sfc@2902200
02902400-029025ff : 2902400.sfc sfc@2902400
02902600-029027ff : 2902600.sfc sfc@2902600
02903000-029030ff : 2903000.amx amx@2903000
02903100-029031ff : 2903100.amx amx@2903100
02903200-029032ff : 2903200.amx amx@2903200
02903300-029033ff : 2903300.amx amx@2903300
02903800-029038ff : 2903800.adx adx@2903800
02903900-029039ff : 2903900.adx adx@2903900
02903a00-02903aff : 2903a00.adx adx@2903a00
02903b00-02903bff : 2903b00.adx adx@2903b00
02904000-029040ff : 2904000.dmic dmic@2904000
02904100-029041ff : 2904100.dmic dmic@2904100
02904200-029042ff : 2904200.dmic dmic@2904200
02904300-029043ff : 2904300.dmic dmic@2904300
02905000-029050ff : 2905000.dspk dspk@2905000
02905100-029051ff : 2905100.dspk dspk@2905100
02907000-029070ff : 2907000.afc afc@2907000
02907100-029071ff : 2907100.afc afc@2907100
02907200-029072ff : 2907200.afc afc@2907200
02907300-029073ff : 2907300.afc afc@2907300
02907400-029074ff : 2907400.afc afc@2907400
02907500-029075ff : 2907500.afc afc@2907500
02908000-029080ff : 2908000.processing-engine processing-engine@2908000
02908100-029081ff : 2908000.processing-engine equalizer@2908100
02908200-029083ff : 2908000.processing-engine dynamic-range-compressor@2908200
0290a000-0290a1ff : 290a000.mvc mvc@290a000
0290a200-0290a3ff : 290a200.mvc mvc@290a200
0290bb00-0290c2ff : 290bb00.amixer amixer@290bb00
0290e400-0290e7ff : 290e400.arad arad@290e400
0290f000-0290ffff : 290f000.admaif admaif@290f000
02910000-02911fff : 2910000.asrc asrc@2910000
02930000-0294ffff : 2930000.dma-controller dma-controller@2930000
02b80000-02b8ffff : 2c00000.memory-controller ch4
02b90000-02b9ffff : 2c00000.memory-controller ch5
02ba0000-02baffff : 2c00000.memory-controller ch6
02bb0000-02bbffff : 2c00000.memory-controller ch7
02c00000-02c0ffff : 2c00000.memory-controller sid
02c10000-02c1ffff : 2c00000.memory-controller broadcast
02c20000-02c2ffff : 2c00000.memory-controller ch0
02c30000-02c3ffff : 2c00000.memory-controller ch1
02c40000-02c4ffff : 2c00000.memory-controller ch2
02c50000-02c5ffff : 2c00000.memory-controller ch3
03100000-0310ffff : 3100000.serial serial@3100000
03110000-0311ffff : 3110000.serial serial@3110000
03130000-0313ffff : 3130000.serial serial@3130000
03140000-0314ffff : 3140000.serial serial@3140000
03160000-031600ff : 3160000.i2c i2c@3160000
03180000-031800ff : 3180000.i2c i2c@3180000
03190000-031900ff : 3190000.i2c i2c@3190000
031b0000-031b00ff : 31b0000.i2c i2c@31b0000
031c0000-031c00ff : 31c0000.i2c i2c@31c0000
031e0000-031e00ff : 31e0000.i2c i2c@31e0000
03210000-03210fff : 3210000.spi spi@3210000
03230000-03230fff : 3230000.spi spi@3230000
03280000-0328ffff : 3280000.pwm pwm@3280000
032a0000-032affff : 32a0000.pwm pwm@32a0000
032c0000-032cffff : 32c0000.pwm pwm@32c0000
032f0000-032fffff : 32f0000.pwm pwm@32f0000
03400000-0341ffff : 3400000.mmc mmc@3400000
03460000-0347ffff : 3460000.mmc mmc@3460000
03520000-0353ffff : 3520000.padctl padctl
03540000-0354ffff : 3520000.padctl ao
03550000-03557fff : 3550000.usb base
03558000-0355ffff : 3550000.usb fpci
03600000-0360ffff : 3610000.usb fpci
03610000-0364ffff : 3610000.usb hcd
03650000-0365ffff : 3610000.usb bar2
03810000-0381ffff : 3810000.fuse fuse@3810000
03990000-0399fffe : nvidia
039c0000-039c000f : 39c0000.tachometer tachometer@39c0000
03c00000-03c9ffff : 3c00000.hsp hsp@3c00000
03d00000-03d9ffff : 3d00000.hsp hsp@3d00000
03e00000-03e0ffff : 3e00000.phy ctl
03e10000-03e1ffff : 3e10000.phy ctl
03e20000-03e2ffff : 3e20000.phy ctl
03e30000-03e3ffff : 3e30000.phy ctl
03e40000-03e4ffff : 3e40000.phy ctl
03e50000-03e5ffff : 3e50000.phy ctl
03e60000-03e6ffff : 3e60000.phy ctl
03e70000-03e7ffff : 3e70000.phy ctl
03e90000-03e9ffff : 3e90000.phy ctl
03ea0000-03eaffff : 3ea0000.phy ctl
03eb0000-03ebffff : 3eb0000.phy ctl
03ec0000-03ecffff : 3ec0000.phy ctl
03ed0000-03edffff : 3ed0000.phy ctl
03ee0000-03eeffff : 3ee0000.phy ctl
03ef0000-03efffff : 3ef0000.phy ctl
03f00000-03f0ffff : 3f00000.phy ctl
03f20000-03f2ffff : 3f20000.phy ctl
03f30000-03f3ffff : 3f30000.phy ctl
03f40000-03f4ffff : 3f40000.phy ctl
03f50000-03f5ffff : 3f50000.phy ctl
03f60000-03f6ffff : 3f60000.phy ctl
03f70000-03f7ffff : 3f70000.phy ctl
03f80000-03f8ffff : 3f80000.phy ctl
03f90000-03f9ffff : 3f90000.phy ctl
06800000-0680ffff : 6810000.ethernet hypervisor
06810000-0681ffff : 6810000.ethernet mac
068a0000-068affff : 6810000.ethernet xpcs
068d0000-068dffff : 6810000.ethernet macsec-base
06900000-0690ffff : 6910000.ethernet hypervisor
06910000-0691ffff : 6910000.ethernet mac
069a0000-069affff : 6910000.ethernet xpcs
069d0000-069dffff : 6910000.ethernet macsec-base
07000000-07ffffff : 8000000.iommu iommu@8000000
08000000-08ffffff : 8000000.iommu iommu@8000000
0b950000-0b9dffff : b950000.tegra-hsp tegra-hsp@b950000
0b9f0000-0ba2ffff : bc00000.rtcpu rce-pm
0be00000-0be3ffff : be00000.rce-fabric rce-fabric@be00000
0c150000-0c1dffff : c150000.hsp hsp@c150000
0c1e0000-0c1effff : c1e0000.hardware-timestamp hardware-timestamp@c1e0000
0c240000-0c2400ff : c240000.i2c i2c@c240000
0c250000-0c2500ff : c250000.i2c i2c@c250000
0c260000-0c260fff : c260000.spi spi@c260000
0c2f0000-0c2f0fff : c2f0000.gpio security
0c2f1000-0c2f1fff : c2f0000.gpio gpio
0c300000-0c303fff : c300000.pinmux pinmux@c300000
0c310000-0c310143 : c310000.mttcan can-regs
0c311000-0c311031 : c310000.mttcan glue-regs
0c312000-0c312fff : c310000.mttcan msg-ram
0c320000-0c320143 : c320000.mttcan can-regs
0c321000-0c321031 : c320000.mttcan glue-regs
0c322000-0c322fff : c320000.mttcan msg-ram
0c360000-0c36ffff : c360000.pmc pmc
0c370000-0c37ffff : c360000.pmc wake
0c380000-0c38ffff : c360000.pmc aotag
0c390000-0c39ffff : c360000.pmc scratch
0c600000-0c63ffff : c600000.aon-fabric aon-fabric@c600000
0d230000-0d230fff : d230000.actmon actmon@d230000
0d600000-0d63ffff : d600000.bpmp-fabric bpmp-fabric@d600000
0de00000-0de3ffff : de00000.dce-fabric dce-fabric@de00000
0e000000-0e05fffe : e000000.ccplex ccplex@e000000
0e100000-0e10ffff : e100000.tegra_mce tegra_mce@e100000
0e110000-0e11ffff : e100000.tegra_mce tegra_mce@e100000
0e120000-0e12ffff : e100000.tegra_mce tegra_mce@e100000
0e130000-0e13ffff : e100000.tegra_mce tegra_mce@e100000
0e140000-0e14ffff : e100000.tegra_mce tegra_mce@e100000
0e150000-0e15ffff : e100000.tegra_mce tegra_mce@e100000
0e160000-0e16ffff : e100000.tegra_mce tegra_mce@e100000
0e170000-0e17ffff : e100000.tegra_mce tegra_mce@e100000
0e180000-0e18ffff : e100000.tegra_mce tegra_mce@e100000
0e190000-0e19ffff : e100000.tegra_mce tegra_mce@e100000
0e1a0000-0e1affff : e100000.tegra_mce tegra_mce@e100000
0e1b0000-0e1bffff : e100000.tegra_mce tegra_mce@e100000
10000000-10ffffff : 10000000.iommu iommu@10000000
11000000-11ffffff : 12000000.iommu iommu@12000000
12000000-12ffffff : 12000000.iommu iommu@12000000
13800000-138efffd : nvidia
13a00000-13dfffff : 13a00000.cbb-fabric cbb-fabric@13a00000
13e00000-13e0ffff : 13e00000.host1x common
13e10000-13e1ffff : 13e00000.host1x hypervisor
13e40000-13e4ffff : 13e00000.host1x vm
13ef0000-13f4ffff : 13e00000.host1x actmon
14100000-1411ffff : 14100000.pcie appl
14160000-1417ffff : 14160000.pcie appl
14180000-1419ffff : 14180000.pcie appl
141a0000-141bffff : 141a0000.pcie appl
141e0000-141fffff : 141e0000.pcie appl
14800000-1480ffff : 14800000.isp isp@14800000
15340000-1537ffff : 15340000.vic vic@15340000
15380000-153bffff : 15380000.nvjpg nvjpg@15380000
15480000-154bffff : 15480000.nvdec nvdec@15480000
154c0000-154fffff : 154c0000.nvenc nvenc@154c0000
15500000-1553ffff : 15500000.tsec tsec@15500000
15540000-1557ffff : 15540000.nvjpg nvjpg@15540000
155c0000-155cfffd : nvidia
15880000-158bffff : 15880000.nvdla0 nvdla0@15880000
158c0000-158fffff : 158c0000.nvdla1 nvdla1@158c0000
15a50000-15a8ffff : 15a50000.ofa ofa@15a50000
16000000-167fffff : 16000000.pva0 pva0@16000000
17000000-17ffffff : 17000000.gpu gpu@17000000
18000000-18ffffff : 17000000.gpu gpu@17000000
24700000-2477ffff : 16000000.pva0 pva0@16000000
30000000-3003ffff : 14100000.pcie config
30040000-3007ffff : 14100000.pcie atu_dma
30080000-300bffff : 14100000.pcie dbi
36000000-3603ffff : 14160000.pcie config
36040000-3607ffff : 14160000.pcie atu_dma
36080000-360bffff : 14160000.pcie dbi
38000000-3803ffff : 14180000.pcie config
38040000-3807ffff : 14180000.pcie atu_dma
38080000-380bffff : 14180000.pcie dbi
3a000000-3a03ffff : 141a0000.pcie config
3a040000-3a07ffff : 141a0000.pcie atu_dma
3a080000-3a0bffff : 141a0000.pcie dbi
3e000000-3e03ffff : 141e0000.pcie config
3e040000-3e07ffff : 141e0000.pcie atu_dma
3e080000-3e0bffff : 141e0000.pcie dbi
40070000-40070fff : 40000000.sram sram@70000
40071000-40071fff : 40000000.sram sram@71000
80000000-fffdffff : System RAM
fbfe0000-fffdffff : reserved
fffe0000-ffffffff : reserved
100000000-e15bbffff : System RAM
d26db0000-d2928ffff : Kernel code
d29290000-d29aaffff : reserved
d29ab0000-d29fbffff : Kernel data
dca200000-e10ffffff : reserved
e11390000-e113dafff : reserved
e11590000-e12983fff : reserved
e15bc0000-e15d3ffff : reserved
e15d30000-e15d30fff : reserved
e15d40000-e2322ffff : System RAM
e15d46000-e15d46fff : reserved
e15d48000-e15d48fff : reserved
e15d49000-e15d4afff : reserved
e15d4b000-e15d4bfff : reserved
e1bc90000-e1bc90fff : reserved
e1bc91000-e1bc91fff : reserved
e1d230000-e2322ffff : reserved
e23230000-e2516ffff : reserved
e25170000-e2c5fffff : System RAM
e25a29000-e25ae8fff : reserved
e25ae9000-e25ae9fff : reserved
e25aea000-e25c45fff : reserved
e25c48000-e25c48fff : reserved
e25c49000-e25c4bfff : reserved
e25c4c000-e25c4ffff : reserved
e25c50000-e25c64fff : reserved
e25c65000-e2c5fffff : reserved
e2c600000-e2c7fffff : reserved
e2c800000-e2cd6ffff : System RAM
e2c800000-e2cd6ffff : reserved
e2d700000-e2d7fffff : reserved
e32000000-e33ffffff : reserved
2080000000-20a7ffffff : pcie@14100000
20a8000000-20afffffff : pcie@14100000
20a8000000-20a81fffff : PCI Bus 0001:01
20a8000000-20a80fffff : 0001:01:00.0
20a8000000-20a80fffff : igc
20a8100000-20a8103fff : 0001:01:00.0
20a8100000-20a8103fff : igc
2140000000-2427ffffff : pcie@14160000
2428000000-242fffffff : pcie@14160000
2428000000-24280fffff : PCI Bus 0004:01
2428000000-2428003fff : 0004:01:00.0
2428000000-2428003fff : iwlwifi
2440000000-2727ffffff : pcie@14180000
2728000000-272fffffff : pcie@14180000
2728000000-27281fffff : PCI Bus 0000:01
2728000000-27280fffff : 0000:01:00.0
2728000000-27280fffff : igc
2728100000-2728103fff : 0000:01:00.0
2728100000-2728103fff : igc
2800000000-2b27ffffff : pcie@141a0000
2b28000000-2b2fffffff : pcie@141a0000
2b28000000-2b28ffffff : PCI Bus 0005:01
2b28000000-2b28ffffff : 0005:01:00.0
3000000000-3227ffffff : pcie@141e0000
3228000000-322fffffff : pcie@141e0000
3228000000-32280fffff : PCI Bus 0007:01
3228000000-3228003fff : 0007:01:00.0
3228000000-3228003fff : nvme
C5 PCIe Controller Device Tree Entry
pcie@141a0000 {
power-domains = <0x03 0x05>;
iommus = <0x06 0x14>;
vpcie3v3-supply = <0x153>;
#address-cells = <0x03>;
dma-coherent;
interconnect-names = "dma-mem\0write";
phy-names = "p2u-0\0p2u-1\0p2u-2\0p2u-3\0p2u-4\0p2u-5\0p2u-6\0p2u-7";
nvidia,bpmp = <0x03 0x05>;
bus-range = <0x00 0xff>;
clock-names = "core";
interconnects = <0x05 0xe2 0x07 0x05 0xe3 0x07>;
reg-names = "appl\0config\0atu_dma\0dbi\0ecam";
nvidia,aspm-l0s-entrance-latency-us = <0x03>;
nvidia,enable-srns;
resets = <0x03 0x82 0x03 0x81>;
interrupts = <0x00 0x35 0x04 0x00 0x36 0x04>;
clocks = <0x03 0xe1>;
interrupt-map = <0x00 0x00 0x00 0x00 0x01 0x00 0x35 0x04>;
#size-cells = <0x02>;
vpcie12v-supply = <0x11a>;
device_type = "pci";
interrupt-map-mask = <0x00 0x00 0x00 0x00>;
num-lanes = <0x08>;
compatible = "nvidia,tegra234-pcie";
vddio-pex-ctl-supply = <0x108>;
ranges = <0x43000000 0x28 0x00 0x28 0x00 0x03 0x28000000 0x2000000 0x00 0x40000000 0x2b 0x28000000 0x00 0x8000000 0x1000000 0x00 0x3a100000 0x00 0x3a100000 0x00 0x100000>;
iommu-map-mask = <0x00>;
#interrupt-cells = <0x01>;
status = "okay";
interrupt-names = "intr\0msi";
phys = <0x10a 0x10b 0x10c 0x10d 0x10e 0x10f 0x110 0x111>;
num-viewport = <0x08>;
reg = <0x00 0x141a0000 0x00 0x20000 0x00 0x3a000000 0x00 0x40000 0x00 0x3a040000 0x00 0x40000 0x00 0x3a080000 0x00 0x40000 0x2b 0x30000000 0x00 0x10000000>;
linux,pci-domain = <0x05>;
iommu-map = <0x00 0x06 0x14 0x1000>;
reset-names = "apb\0core";
nvidia,aspm-pwr-on-t-us = <0x14>;
nvidia,aspm-cmrt-us = <0x3c>;
};
Another experiment I tried was appending pci=realloc to the boot cmdline args in /boot/extlinux/extlinux.conf, but it didn’t have any effect on BAR allocation.
Would appreciate any pointers for PCIe enumeration.
