Hello. I use PCIe endpoint connected to Zynq Ultrascale+ root port.
I have a problem with not being able to assign memory areas (BARs) to PCIe devices.
Address ranges do not overlap
root@zynqup:~# dmesg | grep pci
[ 2.056044] ehci-pci: EHCI PCI platform driver
[ 2.473723] nwl-pcie fd0e0000.pcie: Link is UP
[ 2.478193] nwl-pcie fd0e0000.pcie: host bridge /amba/pcie@fd0e0000 ranges:
[ 2.485150] nwl-pcie fd0e0000.pcie: Parsing ranges property...
[ 2.485165] nwl-pcie fd0e0000.pcie: MEM 0xe0000000..0xefffffff -> 0xe0000000
[ 2.492380] nwl-pcie fd0e0000.pcie: MEM 0x600000000..0x7ffffffff -> 0x600000000
[ 2.499968] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[ 2.506146] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.511629] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[ 2.518494] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[ 2.525970] pci_bus 0000:00: scanning bus
[ 2.525993] pci 0000:00:00.0: [10ee:d011] type 01 class 0x058000
[ 2.526007] pci 0000:00:00.0: ignoring class 0x058000 (doesn't match header type 01)
[ 2.533776] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[ 2.533782] pci 0000:00:00.0: PME# disabled
[ 2.535144] pci_bus 0000:00: fixups for bus
[ 2.535150] pci 0000:00:00.0: scanning [bus 01-0c] behind bridge, pass 0
[ 2.535207] pci_bus 0000:01: scanning bus
[ 2.535235] pci 0000:01:00.0: [10de:0001] type 00 class 0x050000
[ 2.535309] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff]
[ 2.535338] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0001ffff 64bit pref]
[ 2.535357] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00000fff 64bit]
[ 2.535382] pci 0000:01:00.0: Max Payload Size set to 128 (was 256, max 256)
[ 2.542553] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 2.542560] pci 0000:01:00.0: PME# disabled
[ 2.542596] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:00.0 (capable of 126.024 Gb/s with 16 GT/s x8 link)
[ 2.557805] pci_bus 0000:01: fixups for bus
[ 2.557810] pci_bus 0000:01: bus scan returning with max=01
[ 2.557817] pci 0000:00:00.0: scanning [bus 01-0c] behind bridge, pass 1
[ 2.557823] pci_bus 0000:00: bus scan returning with max=0c
[ 2.557837] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00020000 64bit pref]
[ 2.565403] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00020000 64bit pref]
[ 2.573309] pci 0000:01:00.0: BAR 0: no space for [mem size 0x00010000]
[ 2.579914] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x00010000]
[ 2.586866] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00001000 64bit]
[ 2.593992] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00001000 64bit]
[ 2.601466] pci 0000:00:00.0: not setting up bridge for bus 0000:01
root@zynqup:~# lspci -vvv
00:00.0 Non-VGA unclassified device: Xilinx Corporation Device d011
!!! Invalid class 0000 for header type 01
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 0
Bus: primary=00, secondary=01, subordinate=0c, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: 00000000-000fffff [size=1M]
Prefetchable memory behind bridge: 0000000000000000-00000000000fffff [size=1M]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [60] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend+
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [10c v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [128 v1] Vendor Specific Information: ID=1234 Rev=1 Len=018 <?>
01:00.0 RAM memory: NVIDIA Corporation Device 0001
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 255
Region 2: Memory at <unassigned> (64-bit, prefetchable) [disabled]
Region 4: Memory at <unassigned> (64-bit, non-prefetchable) [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Vector table: BAR=2 offset=00000000
PBA: BAR=2 offset=00010000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Secondary PCI Express <?>
Capabilities: [168 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [190 v1] Lane Margining at the Receiver <?>
Capabilities: [1b8 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1c0 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1- L1_PM_Substates+
PortCommonModeRestoreTime=60us PortTPowerOnTime=40us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us
L1SubCtl2: T_PwrOn=10us
Capabilities: [1d0 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [2d0 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [308 v1] Data Link Feature <?>
Capabilities: [314 v1] Precision Time Measurement
PTMCap: Requester:+ Responder:- Root:-
PTMClockGranularity: Unimplemented
PTMControl: Enabled:- RootSelected:-
PTMEffectiveGranularity: Unknown
Capabilities: [320 v1] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>
Capabilities: [388 v1] Vendor Specific Information: ID=0006 Rev=0 Len=018 <?>
jetson@ubuntu:~$ sudo dmesg | grep pci
[sudo] password for jetson:
[ 3.890117] ohci-pci: OHCI PCI platform driver
[ 5.833006] tegra194-pcie 141a0000.pcie_ep: Adding to iommu group 9
[ 5.845833] tegra194-pcie 141a0000.pcie_ep: Failed to get PERST GPIO: -517
[ 5.845842] tegra194-pcie 141a0000.pcie_ep: Failed to parse device tree: -517
[ 5.846083] tegra194-pcie 14100000.pcie: Adding to iommu group 10
[ 5.858636] tegra194-pcie 14100000.pcie: Using GICv2m MSI allocator
[ 5.867373] tegra194-pcie 14160000.pcie: Adding to iommu group 11
[ 5.879708] tegra194-pcie 14160000.pcie: Using GICv2m MSI allocator
[ 7.213867] tegra194-pcie 141a0000.pcie_ep: Using GICv2m MSI allocator
[ 7.220607] tegra194-pcie 141a0000.pcie_ep: Failed to get slot regulators: -517
[ 7.229918] tegra194-pcie 14100000.pcie: Using GICv2m MSI allocator
[ 7.245083] tegra194-pcie 14100000.pcie: host bridge /pcie@14100000 ranges:
[ 7.265638] tegra194-pcie 14100000.pcie: IO 0x0030100000..0x00301fffff -> 0x0030100000
[ 7.274323] tegra194-pcie 14100000.pcie: MEM 0x20a8000000..0x20afffffff -> 0x0040000000
[ 7.283006] tegra194-pcie 14100000.pcie: MEM 0x2080000000..0x20a7ffffff -> 0x2080000000
[ 8.401234] tegra194-pcie 14100000.pcie: Phy link never came up
[ 8.407404] tegra194-pcie 14100000.pcie: PCI host bridge to bus 0001:00
[ 8.414216] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 8.419859] pci_bus 0001:00: root bus resource [io 0x0000-0xfffff] (bus address [0x30100000-0x301fffff])
[ 8.429708] pci_bus 0001:00: root bus resource [mem 0x20a8000000-0x20afffffff] (bus address [0x40000000-0x47ffffff])
[ 8.440544] pci_bus 0001:00: root bus resource [mem 0x2080000000-0x20a7ffffff pref]
[ 8.448474] pci 0001:00:00.0: [10de:229e] type 01 class 0x060400
[ 8.454811] pci 0001:00:00.0: PME# supported from D0 D3hot
[ 8.469360] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[ 8.474753] pci 0001:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 8.483740] pcieport 0001:00:00.0: Adding to iommu group 10
[ 8.489739] pcieport 0001:00:00.0: PME: Signaling with IRQ 65
[ 8.496419] pcieport 0001:00:00.0: AER: enabled with IRQ 65
[ 8.502514] pci_bus 0001:01: busn_res: [bus 01-ff] is released
[ 8.508603] pci 0001:00:00.0: Removing from iommu group 10
[ 8.514254] pci_bus 0001:00: busn_res: [bus 00-ff] is released
[ 8.522237] tegra194-pcie 14160000.pcie: Using GICv2m MSI allocator
[ 8.529512] tegra194-pcie 14160000.pcie: host bridge /pcie@14160000 ranges:
[ 8.536690] tegra194-pcie 14160000.pcie: IO 0x0036100000..0x00361fffff -> 0x0036100000
[ 8.545400] tegra194-pcie 14160000.pcie: MEM 0x2428000000..0x242fffffff -> 0x0040000000
[ 8.554093] tegra194-pcie 14160000.pcie: MEM 0x2140000000..0x2427ffffff -> 0x2140000000
[ 9.669388] tegra194-pcie 14160000.pcie: Phy link never came up
[ 9.675542] tegra194-pcie 14160000.pcie: PCI host bridge to bus 0004:00
[ 9.682354] pci_bus 0004:00: root bus resource [bus 00-ff]
[ 9.687985] pci_bus 0004:00: root bus resource [io 0x100000-0x1fffff] (bus address [0x36100000-0x361fffff])
[ 9.698105] pci_bus 0004:00: root bus resource [mem 0x2428000000-0x242fffffff] (bus address [0x40000000-0x47ffffff])
[ 9.708944] pci_bus 0004:00: root bus resource [mem 0x2140000000-0x2427ffffff pref]
[ 9.716863] pci 0004:00:00.0: [10de:229c] type 01 class 0x060400
[ 9.723213] pci 0004:00:00.0: PME# supported from D0 D3hot
[ 9.737164] pci 0004:00:00.0: PCI bridge to [bus 01-ff]
[ 9.742555] pci 0004:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 9.751521] pcieport 0004:00:00.0: Adding to iommu group 11
[ 9.757487] pcieport 0004:00:00.0: PME: Signaling with IRQ 67
[ 9.763721] pcieport 0004:00:00.0: AER: enabled with IRQ 67
[ 9.769664] pci_bus 0004:01: busn_res: [bus 01-ff] is released
[ 9.775762] pci 0004:00:00.0: Removing from iommu group 11
[ 9.781410] pci_bus 0004:00: busn_res: [bus 00-ff] is released
[ 9.788775] vdd-3v3-pcie: supplied by vdd-3v3-sys
[ 9.846380] tegra194-pcie 141a0000.pcie_ep: Using GICv2m MSI allocator
[ 16.229567] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x12a088000
[ 16.229585] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[ 16.229619] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0x0000000067398ec0
[ 42.420163] vdd-3v3-pcie: disabling
[ 42.420193] vdd-12v-pcie: disabling
Root port have range
[ 2.506146] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.511629] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[ 2.518494] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
In Jetson PCIe endpoint:
[ 2.535309] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff]
[ 2.535338] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0001ffff 64bit pref]
[ 2.535357] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00000fff 64bit]
As I understand it, the driver assigns BAR addresses.
I would like to know how I can change the range of addresses for Jetson so that they overlap with
Zynq Ultrascale+ addresses?