I have two FPGA boards that work in a PC. One is an Actel board using a PCIe 2.0 link speed of 5 GT/s the other is a Lattice board using a PCIe 1.0 link speed of 2.5 GT/s. Both have two BARs <= 256KB. ON the TX1, the Actel board links but the BARs are not assigned any memory during enumeration. The Lattice board links and its BARs ARE assigned memory during enumeration. Anyone have an idea of where I should be looking to get the Actel board BAR region memory allocated?
What does “lspci -vvv” show when you soft start TX1?
If the card does not show by “lspci” it may not be able to handle Spread Spectrum Clocking of TX1.
I ran ‘dmesg | grep pci’ > a1.txt and ‘sudo lspci -vv > a2.txt’ with the Actel board and then the same with the Lattice baord so that I could compare the results. Both outputs for each board were almost identical except of course for the fact that one has PCIe 1 and the other has PCIe 2 link rates.
The only significant difference for the dmesg log is that the Actel board has a line about "PME# suported from D0 D3hot’ and the Lattice doesn’t.
For lspci the only thing that stands out, other than what I assume are PCIe 2.0 messages is a line about Capabilities: Virtual Channel for the Actel board.
lspci -x looks reasonable for both boards.
Looking at the same information for the Actel board in my Centos PC I see that the dmesg line regarding ‘PME# supported from D0 D3hot’ is followed by the line PME# disabled. Everything else is pretty much the same except the board regions are allocated memory in the PC and works as expected.
Is it possible to attach dmesg logs with Actel and Lattice boards connecting only one board to the system at a time?
Hi vidyas,
I don’t understand your question. Please expand…
I mean, please attach
a) dmesg log with only Actel board connected to TX1 system
b) dmesg log with only Lattice board connected to TX1 system
result of ‘dmesg | grep pci’ with Lattice board installed:
[ 3.068502] tegra-pcie 1003000.pcie-controller: PCIE: Enable power rails
[ 3.074207] tegra-pcie 1003000.pcie-controller: probing port 0, using 4 lanes and lane map as 0x14
[ 3.083661] tegra-pcie 1003000.pcie-controller: probing port 1, using 1 lanes and lane map as 0x14
[ 3.211590] tegra-pcie 1003000.pcie-controller: link 1 down, ignoring
[ 3.216236] tegra-pcie 1003000.pcie-controller: PCI host bridge to bus 0000:00
[ 3.223245] pci_bus 0000:00: root bus resource [mem 0x13000000-0x1fffffff]
[ 3.230136] pci_bus 0000:00: root bus resource [mem 0x20000000-0x3fffffff pref]
[ 3.237389] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.242825] pci_bus 0000:00: root bus resource [io 0x1000-0xffff]
[ 3.249062] pci 0000:00:01.0: [10de:0fae] type 01 class 0x060400
[ 3.249203] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 3.249498] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.257200] pci 0000:01:00.0: [1204:ec30] type 00 class 0x000000
[ 3.257249] pci 0000:01:00.0: reg 10: [mem 0x13000000-0x1303ffff]
[ 3.257282] pci 0000:01:00.0: reg 14: [mem 0x13040000-0x1307ffff]
[ 3.273530] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 3.273590] pci 0000:00:01.0: BAR 8: assigned [mem 0x13000000-0x130fffff]
[ 3.278395] pci 0000:00:01.0: PCI bridge to [bus 01]
[ 3.283334] pci 0000:00:01.0: bridge window [mem 0x13000000-0x130fffff]
[ 3.295941] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
[ 3.302658] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[ 3.309208] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded
[ 3.309359] aer 0000:00:01.0:pcie02: service driver aer loaded
[ 3.309461] tegra-pcie 1003000.pcie-controller: PCIE: No Link speed change happened
[ 5.044698] ehci-pci: EHCI PCI platform driver
[ 14.224342] vgaarb: this pci device is not a vga device
[ 14.284284] vgaarb: this pci device is not a vga device
result of ‘dmesg | grep pci’ with Altera board installed:
[ 3.069309] tegra-pcie 1003000.pcie-controller: PCIE: Enable power rails
[ 3.074984] tegra-pcie 1003000.pcie-controller: probing port 0, using 4 lanes and lane map as 0x14
[ 3.084633] tegra-pcie 1003000.pcie-controller: probing port 1, using 1 lanes and lane map as 0x14
[ 3.128526] tegra-pcie 1003000.pcie-controller: link 1 down, ignoring
[ 3.133155] tegra-pcie 1003000.pcie-controller: PCI host bridge to bus 0000:00
[ 3.140203] pci_bus 0000:00: root bus resource [mem 0x13000000-0x1fffffff]
[ 3.147069] pci_bus 0000:00: root bus resource [mem 0x20000000-0x3fffffff pref]
[ 3.154307] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.159775] pci_bus 0000:00: root bus resource [io 0x1000-0xffff]
[ 3.166000] pci 0000:00:01.0: [10de:0fae] type 01 class 0x060400
[ 3.166144] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 3.166437] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.174112] pci 0000:01:00.0: [11aa:11aa] type 00 class 0x000000
[ 3.174157] pci 0000:01:00.0: reg 10: [mem 0x00000000-0x0003ffff]
[ 3.174189] pci 0000:01:00.0: reg 14: [mem 0x00000000-0x0000ffff]
[ 3.174373] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 3.194502] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 3.194563] pci 0000:00:01.0: BAR 8: assigned [mem 0x13000000-0x130fffff]
[ 3.199367] pci 0000:00:01.0: PCI bridge to [bus 01]
[ 3.204307] pci 0000:00:01.0: bridge window [mem 0x13000000-0x130fffff]
[ 3.216914] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
[ 3.223630] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[ 3.230188] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded
[ 3.230354] aer 0000:00:01.0:pcie02: service driver aer loaded
[ 4.960108] ehci-pci: EHCI PCI platform driver
[ 13.901780] vgaarb: this pci device is not a vga device
[ 14.018055] vgaarb: this pci device is not a vga device
Oops, substitute the work Actel for Altera in the dmesg log…
Are you sure that it is Lattice board that is working fine and not the Actel?
because, I see
[ 3.257249] pci 0000:01:00.0: reg 10: [mem 0x13000000-0x1303ffff]
[ 3.257282] pci 0000:01:00.0: reg 14: [mem 0x13040000-0x1307ffff]
for Lattice which looks incorrect… I mean it should’ve been
reg 10: [mem 0x00000000-0x0003ffff]
reg 14: [mem 0x00000000-0x0003ffff]
Wondering if it because of config space not being reset?? (and hence previously assigned locations are showing up?)
for Actel board, things looks fine i.e.
[ 3.174157] pci 0000:01:00.0: reg 10: [mem 0x00000000-0x0003ffff]
[ 3.174189] pci 0000:01:00.0: reg 14: [mem 0x00000000-0x0000ffff]
But, in both cases, I don’t see BARs getting some memory space assigned by kernel.
there is no prints of the sort
pci 0000:01:00.0: BAR x: assigned [mem 0xyyyyyyyy-0xzzzzzzzz]
(note … this is for 01:00.0)
Can you also give ‘lspci -vvvv’ and ‘dmesg | grep -i pci’ output? for all cases?
i.e.
a) only Lattice card connected to system
b) Only Actel card connected to system
c) both Lattice and Actel connected to system
HuH?
Here’s information about the Actel board in my PC under Centos 6.8. I use this board without issue.
-[0000:00]-±00.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
±01.0-[01]–±00.0 NVIDIA Corporation GK104 [GeForce GTX 760]
| -00.1 NVIDIA Corporation GK104 HDMI Audio Controller
±14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
±16.0 Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
±1a.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
±1b.0 Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
±1c.0-[02]----00.0 Actel Device 11aa
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: f7200000-f72fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities:
Kernel modules: shpchp
02:00.0 Non-VGA unclassified device: Actel Device 11aa
Subsystem: Actel Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f7200000 (32-bit, non-prefetchable)
Region 1: Memory at f7240000 (32-bit, non-prefetchable)
Capabilities:
So, I expect that the BARs are assigned memory within the space behind whatever bridge the are connected to. The resources for Actel board in the TX1 are not reachable.
Agree, but looks like on TX1 for some reason that is not happening.
Can you please give the following as I asked earlier on TX1 target?
‘lspci -vvvv’ and ‘dmesg | grep -i pci’ outputs for all cases?
i.e.
a) only Lattice card connected to system
b) Only Actel card connected to system
c) both Lattice and Actel connected to system
since there’s only 1 pcie slot I can’t install both boards at the same time. Let’s concentrate on the Actel board as it’s currently my problem. Here is the output as requested for the Actel board on the TX1:
00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: 13000000-130fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: NVIDIA Corporation Device 0000
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
Mapping Address Base: 00000000fee00000
Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd On, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet+ LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=30us PortTPowerOnTime=70us
Kernel driver in use: pcieport
01:00.0 Non-VGA unclassified device: Actel Device 11aa
Subsystem: Actel Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 130
Region 0: Memory at (32-bit, non-prefetchable) [disabled]
Region 1: Memory at (32-bit, non-prefetchable) [disabled]
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <64ns, L1 <16us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [800 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
[ 3.137205] PCI: CLS 0 bytes, default 64
[ 3.179343] tegra-pcie 1003000.pcie-controller: PCIE: Enable power rails
[ 3.185046] tegra-pcie 1003000.pcie-controller: probing port 0, using 4 lanes and lane map as 0x14
[ 3.194435] tegra-pcie 1003000.pcie-controller: probing port 1, using 1 lanes and lane map as 0x14
[ 3.238331] tegra-pcie 1003000.pcie-controller: link 1 down, ignoring
[ 3.242957] tegra-pcie 1003000.pcie-controller: PCI host bridge to bus 0000:00
[ 3.250008] pci_bus 0000:00: root bus resource [mem 0x13000000-0x1fffffff]
[ 3.256874] pci_bus 0000:00: root bus resource [mem 0x20000000-0x3fffffff pref]
[ 3.264112] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.269582] pci_bus 0000:00: root bus resource [io 0x1000-0xffff]
[ 3.275803] pci 0000:00:01.0: [10de:0fae] type 01 class 0x060400
[ 3.275942] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 3.276236] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.283917] pci 0000:01:00.0: [11aa:11aa] type 00 class 0x000000
[ 3.283964] pci 0000:01:00.0: reg 10: [mem 0x00000000-0x0003ffff]
[ 3.283996] pci 0000:01:00.0: reg 14: [mem 0x00000000-0x0000ffff]
[ 3.284181] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 3.304305] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 3.304366] pci 0000:00:01.0: BAR 8: assigned [mem 0x13000000-0x130fffff]
[ 3.309170] pci 0000:00:01.0: PCI bridge to [bus 01]
[ 3.314110] pci 0000:00:01.0: bridge window [mem 0x13000000-0x130fffff]
[ 3.320925] PCI: enabling device 0000:00:01.0 (0140 → 0143)
[ 3.326717] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
[ 3.333433] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[ 3.339983] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded
[ 3.340152] aer 0000:00:01.0:pcie02: service driver aer loaded
[ 5.070082] ehci-pci: EHCI PCI platform driver
[ 14.412285] vgaarb: this pci device is not a vga device
[ 14.468164] vgaarb: this pci device is not a vga device
For the record, I haven’t developed code for the Lattice board on the TX1 yet; I just know that its resources are assigned valid memory.