PCIe Base adress registers range setting

Hello. I use PCIe endpoint connected to Zynq Ultrascale+ root port.

I have a problem with not being able to assign memory areas (BARs) to PCIe devices.
Address ranges do not overlap

root@zynqup:~# dmesg | grep pci
[    2.056044] ehci-pci: EHCI PCI platform driver
[    2.473723] nwl-pcie fd0e0000.pcie: Link is UP
[    2.478193] nwl-pcie fd0e0000.pcie: host bridge /amba/pcie@fd0e0000 ranges:
[    2.485150] nwl-pcie fd0e0000.pcie: Parsing ranges property...
[    2.485165] nwl-pcie fd0e0000.pcie:   MEM 0xe0000000..0xefffffff -> 0xe0000000
[    2.492380] nwl-pcie fd0e0000.pcie:   MEM 0x600000000..0x7ffffffff -> 0x600000000
[    2.499968] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[    2.506146] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.511629] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[    2.518494] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[    2.525970] pci_bus 0000:00: scanning bus
[    2.525993] pci 0000:00:00.0: [10ee:d011] type 01 class 0x058000
[    2.526007] pci 0000:00:00.0: ignoring class 0x058000 (doesn't match header type 01)
[    2.533776] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    2.533782] pci 0000:00:00.0: PME# disabled
[    2.535144] pci_bus 0000:00: fixups for bus
[    2.535150] pci 0000:00:00.0: scanning [bus 01-0c] behind bridge, pass 0
[    2.535207] pci_bus 0000:01: scanning bus
[    2.535235] pci 0000:01:00.0: [10de:0001] type 00 class 0x050000
[    2.535309] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff]
[    2.535338] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0001ffff 64bit pref]
[    2.535357] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00000fff 64bit]
[    2.535382] pci 0000:01:00.0: Max Payload Size set to 128 (was 256, max 256)
[    2.542553] pci 0000:01:00.0: PME# supported from D0 D3hot
[    2.542560] pci 0000:01:00.0: PME# disabled
[    2.542596] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:00.0 (capable of 126.024 Gb/s with 16 GT/s x8 link)
[    2.557805] pci_bus 0000:01: fixups for bus
[    2.557810] pci_bus 0000:01: bus scan returning with max=01
[    2.557817] pci 0000:00:00.0: scanning [bus 01-0c] behind bridge, pass 1
[    2.557823] pci_bus 0000:00: bus scan returning with max=0c
[    2.557837] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00020000 64bit pref]
[    2.565403] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00020000 64bit pref]
[    2.573309] pci 0000:01:00.0: BAR 0: no space for [mem size 0x00010000]
[    2.579914] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x00010000]
[    2.586866] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00001000 64bit]
[    2.593992] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00001000 64bit]
[    2.601466] pci 0000:00:00.0: not setting up bridge for bus 0000:01
root@zynqup:~# lspci -vvv
00:00.0 Non-VGA unclassified device: Xilinx Corporation Device d011
        !!! Invalid class 0000 for header type 01
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 0
        Bus: primary=00, secondary=01, subordinate=0c, sec-latency=0
        I/O behind bridge: 00000000-00000fff [size=4K]
        Memory behind bridge: 00000000-000fffff [size=1M]
        Prefetchable memory behind bridge: 0000000000000000-00000000000fffff [size=1M]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [60] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend+
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [10c v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [128 v1] Vendor Specific Information: ID=1234 Rev=1 Len=018 <?>

01:00.0 RAM memory: NVIDIA Corporation Device 0001
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 255
        Region 2: Memory at <unassigned> (64-bit, prefetchable) [disabled]
        Region 4: Memory at <unassigned> (64-bit, non-prefetchable) [disabled]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00010000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Secondary PCI Express <?>
        Capabilities: [168 v1] Physical Layer 16.0 GT/s <?>
        Capabilities: [190 v1] Lane Margining at the Receiver <?>
        Capabilities: [1b8 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [1c0 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1- L1_PM_Substates+
                          PortCommonModeRestoreTime=60us PortTPowerOnTime=40us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us
                L1SubCtl2: T_PwrOn=10us
        Capabilities: [1d0 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Capabilities: [2d0 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
        Capabilities: [308 v1] Data Link Feature <?>
        Capabilities: [314 v1] Precision Time Measurement
                PTMCap: Requester:+ Responder:- Root:-
                PTMClockGranularity: Unimplemented
                PTMControl: Enabled:- RootSelected:-
                PTMEffectiveGranularity: Unknown
        Capabilities: [320 v1] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>
        Capabilities: [388 v1] Vendor Specific Information: ID=0006 Rev=0 Len=018 <?>
jetson@ubuntu:~$ sudo dmesg | grep pci
[sudo] password for jetson:
[    3.890117] ohci-pci: OHCI PCI platform driver
[    5.833006] tegra194-pcie 141a0000.pcie_ep: Adding to iommu group 9
[    5.845833] tegra194-pcie 141a0000.pcie_ep: Failed to get PERST GPIO: -517
[    5.845842] tegra194-pcie 141a0000.pcie_ep: Failed to parse device tree: -517
[    5.846083] tegra194-pcie 14100000.pcie: Adding to iommu group 10
[    5.858636] tegra194-pcie 14100000.pcie: Using GICv2m MSI allocator
[    5.867373] tegra194-pcie 14160000.pcie: Adding to iommu group 11
[    5.879708] tegra194-pcie 14160000.pcie: Using GICv2m MSI allocator
[    7.213867] tegra194-pcie 141a0000.pcie_ep: Using GICv2m MSI allocator
[    7.220607] tegra194-pcie 141a0000.pcie_ep: Failed to get slot regulators: -517
[    7.229918] tegra194-pcie 14100000.pcie: Using GICv2m MSI allocator
[    7.245083] tegra194-pcie 14100000.pcie: host bridge /pcie@14100000 ranges:
[    7.265638] tegra194-pcie 14100000.pcie:       IO 0x0030100000..0x00301fffff -> 0x0030100000
[    7.274323] tegra194-pcie 14100000.pcie:      MEM 0x20a8000000..0x20afffffff -> 0x0040000000
[    7.283006] tegra194-pcie 14100000.pcie:      MEM 0x2080000000..0x20a7ffffff -> 0x2080000000
[    8.401234] tegra194-pcie 14100000.pcie: Phy link never came up
[    8.407404] tegra194-pcie 14100000.pcie: PCI host bridge to bus 0001:00
[    8.414216] pci_bus 0001:00: root bus resource [bus 00-ff]
[    8.419859] pci_bus 0001:00: root bus resource [io  0x0000-0xfffff] (bus address [0x30100000-0x301fffff])
[    8.429708] pci_bus 0001:00: root bus resource [mem 0x20a8000000-0x20afffffff] (bus address [0x40000000-0x47ffffff])
[    8.440544] pci_bus 0001:00: root bus resource [mem 0x2080000000-0x20a7ffffff pref]
[    8.448474] pci 0001:00:00.0: [10de:229e] type 01 class 0x060400
[    8.454811] pci 0001:00:00.0: PME# supported from D0 D3hot
[    8.469360] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    8.474753] pci 0001:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.483740] pcieport 0001:00:00.0: Adding to iommu group 10
[    8.489739] pcieport 0001:00:00.0: PME: Signaling with IRQ 65
[    8.496419] pcieport 0001:00:00.0: AER: enabled with IRQ 65
[    8.502514] pci_bus 0001:01: busn_res: [bus 01-ff] is released
[    8.508603] pci 0001:00:00.0: Removing from iommu group 10
[    8.514254] pci_bus 0001:00: busn_res: [bus 00-ff] is released
[    8.522237] tegra194-pcie 14160000.pcie: Using GICv2m MSI allocator
[    8.529512] tegra194-pcie 14160000.pcie: host bridge /pcie@14160000 ranges:
[    8.536690] tegra194-pcie 14160000.pcie:       IO 0x0036100000..0x00361fffff -> 0x0036100000
[    8.545400] tegra194-pcie 14160000.pcie:      MEM 0x2428000000..0x242fffffff -> 0x0040000000
[    8.554093] tegra194-pcie 14160000.pcie:      MEM 0x2140000000..0x2427ffffff -> 0x2140000000
[    9.669388] tegra194-pcie 14160000.pcie: Phy link never came up
[    9.675542] tegra194-pcie 14160000.pcie: PCI host bridge to bus 0004:00
[    9.682354] pci_bus 0004:00: root bus resource [bus 00-ff]
[    9.687985] pci_bus 0004:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x36100000-0x361fffff])
[    9.698105] pci_bus 0004:00: root bus resource [mem 0x2428000000-0x242fffffff] (bus address [0x40000000-0x47ffffff])
[    9.708944] pci_bus 0004:00: root bus resource [mem 0x2140000000-0x2427ffffff pref]
[    9.716863] pci 0004:00:00.0: [10de:229c] type 01 class 0x060400
[    9.723213] pci 0004:00:00.0: PME# supported from D0 D3hot
[    9.737164] pci 0004:00:00.0: PCI bridge to [bus 01-ff]
[    9.742555] pci 0004:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    9.751521] pcieport 0004:00:00.0: Adding to iommu group 11
[    9.757487] pcieport 0004:00:00.0: PME: Signaling with IRQ 67
[    9.763721] pcieport 0004:00:00.0: AER: enabled with IRQ 67
[    9.769664] pci_bus 0004:01: busn_res: [bus 01-ff] is released
[    9.775762] pci 0004:00:00.0: Removing from iommu group 11
[    9.781410] pci_bus 0004:00: busn_res: [bus 00-ff] is released
[    9.788775] vdd-3v3-pcie: supplied by vdd-3v3-sys
[    9.846380] tegra194-pcie 141a0000.pcie_ep: Using GICv2m MSI allocator
[   16.229567] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x12a088000
[   16.229585] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[   16.229619] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0x0000000067398ec0
[   42.420163] vdd-3v3-pcie: disabling
[   42.420193] vdd-12v-pcie: disabling

Root port have range
[ 2.506146] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.511629] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[ 2.518494] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]

In Jetson PCIe endpoint:

[ 2.535309] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff]
[ 2.535338] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0001ffff 64bit pref]
[ 2.535357] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00000fff 64bit]

As I understand it, the driver assigns BAR addresses.
I would like to know how I can change the range of addresses for Jetson so that they overlap with
Zynq Ultrascale+ addresses?

This issue has been resolved. If anyone runs into this:
In Vivado, after changing the port type to root,
you need to remember to change the device class to
Base class 0x06
Sub class 0x4
Xilinx Answer 71493…

1 Like