TK1 mini-PCIe stuck at 2.5GT/s

The documentation for the PCIe controller on the Tegra K1 states that both Gen1 and Gen2 speeds are supported, but I’ve been unable to make Gen2 speeds work with the mini-PCIe slot on the Jetson kit.

I’ve added an Intel i350-AM2 on the mini-PCIe, with the internal topology then looking like this:

-+-[0000:02]---00.0-[03]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
 \-[0000:00]---00.0-[01]--+-00.0  Intel Corporation I350 Gigabit Network Connection
                          \-00.1  Intel Corporation I350 Gigabit Network Connection

However while the i350 is requesting 5GT/sec, it’s only getting 2.5:

LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
         Compliance De-emphasis: -6dB

The bridge reports 5GT/sec supported on Port #0:

LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
        ClockPM- Surprise- LLActRep+ BwNot+

One thing to note is that the onboard RTL8111 is Gen1, so is it forcing the entire bridge to operate in Gen1 mode? If so, is there any way to disable the on-board ethernet? I tried poking around in /proc a bit to see if I could disable the power on the device, but haven’t come up with anything so far. Any ideas would be welcome… :-)

I’m curious, what is the lspci -vvv of this device? What’s the exact model of this NIC, and is there more than one function on the device (perhaps each function has 2.5GT/s and add to 5GT/s)?

I was able to remove the Realtek from at least the OS perspective by using:

echo 1 > /sys/devices/pci0000:02/0000:02:00.0/0000:03:00.0/remove
echo 1 > /sys/devices/pci0000:02/0000:02:00.0/remove

As root which removed the 8111 and the port on the pcie bridge. lspci reflects this change - it doesn’t even show the realtek or the port anymore - but nothing changes. I reset the remaining PCIe devices but they still come up as 2.5GT/s (and perform poorly).

There are two ports, but you can’t split lane transfers in that way. Both functions should be reporting 5GT/sec on x1 width.

This particular mPCIe slot has a USB DP/DM USB differential pair on it which can be used instead of the PCIe TX/RX. I don’t have one of these NICs, so I am still curious about how they chose to wire it, and looking for clues as to whether this could be part of the answer. Thus the reason for wanting to know the lspci -vvv and the exact NIC model. Some manufacturers may have chosen to take a USB version and make a mini-PCIe by skipping the actual PCIe TX/RX, or perhaps even using both…if they did use USB I’m wondering if the USB is sharing with something else (does lsusb show anything?)…I’m just exploring the possibilities. The PCIe lanes are themselves packet switched, so I’m thinking that sharing the TX/RX with another device would not account (by itself) for the lower speed. I’m leaning towards the idea of the driver not choosing a higher speed.

I built the card. it works fine at 5GT/sec in a Kabini board I have, so I know it works in general (it even works in the Jetson, it just maxes out at about 1.4Gbps in each direction because of the bad lane speed). The Tegra also has a strange limitation with the maximum PCIe packet size (at least on the Jetson) being 128 bytes, which is half the size of a typical PCIe configuration (sadly adding to the overhead), although I don’t believe that should impact the Gen1 vs. Gen2 negotiation.

I’m leaning towards the Tegra bridge not being able to operate at more than one speed - I’m going to get another Jetson and remove the realtek from the board and see if that changes anything, barring any other suggestions.

For reference, here is the lspci -vvv output from both endpoints and the bridge port:

00:00.0 PCI bridge: NVIDIA Corporation Device 0e12 (rev a1) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 32200000-323fffff
        Prefetchable memory behind bridge: 0000000012200000-00000000122fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Subsystem: NVIDIA Corporation Device 0000
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/2 Maskable- 64bit+
                Address: 00000000ab650000  Data: 0000
        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
                Mapping Address Base: 00000000fee00000
        Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Off, PwrInd On, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet+ LinkState+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel driver in use: pcieport

01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 130
        Region 0: Memory at 32200000 (32-bit, non-prefetchable) 
        Region 2: I/O ports at 1000 
        Region 3: Memory at 32300000 (32-bit, non-prefetchable) 
        [virtual] Expansion ROM at 12200000 [disabled] 
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 <32us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140 v1] Device Serial Number 00-30-18-ff-ff-a1-f7-9e
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 384, stride: 4, Device ID: 1520
                Supported Page Size: 00000553, System Page Size: 00000001
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1a0 v1] Transaction Processing Hints
                Device specific mode supported
                Steering table in TPH capability structure
        Capabilities: [1c0 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [1d0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: igb

01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 130
        Region 0: Memory at 32280000 (32-bit, non-prefetchable) 
        Region 2: I/O ports at 1020 
        Region 3: Memory at 32304000 (32-bit, non-prefetchable) 
        [virtual] Expansion ROM at 12280000 [disabled] 
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 <32us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140 v1] Device Serial Number 00-30-18-ff-ff-a1-f7-9e
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 01
                VF offset: 384, stride: 4, Device ID: 1520
                Supported Page Size: 00000553, System Page Size: 00000001
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1a0 v1] Transaction Processing Hints
                Device specific mode supported
                Steering table in TPH capability structure
        Capabilities: [1d0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: igb

Is your Kabini board running linux? I’m guessing it is, and if so, would also use pcieport driver. I’m not all that familiar with Kabini, but it looks to be fully x86, whereas Jetson is ARMv7.

I had first thought perhaps you were using PHY, data, and transaction layer in a way that restricted the driver to 2.5GT/s, but the fact that Kabini made it to 5GT/s with the x86 version of the same driver means this wouldn’t be true (plus it seems the chip you are using has built in PCIe support). On the other hand, even if both systems use linux pcieport driver, there would be significant ARMv7-x86 differences between the two drivers. The tegra124 SoC has a register which could be set to force speed restriction. See the TegraK1 TRM:
30.3.5.13 T_PCIE2_RP_LINK_CONTROL_STATUS_2 (page 2112)

Since Jetson is a development board and not for general use I would suspect perhaps pcieport driver/hardware development was simplified by setting that driver to 2.5GT/s.

I happen to have a Realtek wired NIC in that slot, and it has the same issue…5GT/s capability, 2.5GT/s actual. It’s possible that PCIe would be intentionally throttling back the rate if it detected signals requiring this for quality reasons, but then both our NICs would have this issue and would tend to mean the trace routing on the Jetson itself was an issue (do you see errors in logs?). Had trace routing been the issue I think general operation of the mPCIe would be a bit more “flaky” (that’s a technical term!). You might want to check on that register to see if the driver on the Jetson version was told to limit to 2.5GT/s.

I doubt link control status is the problem (because the bridge does actually report 5GT/sec capacity, and I think it wouldn’t do that if the registers were set to Gen1).

At the moment I’m leaning towards the problem being the integrated RTL8111. I have another few boards on the way, so I’ll try removing that chip if all else fails. (Going to try building a new system using u-boot first and see if I can disable it in software).

I’m not sure, but it would seem capability might be part of a hardware query…in which case the PHY/link would show what they could do but not necessarily that the driver isn’t currently in that state. I would not be surprised either way…whether it is the RTL8111 or the driver…but I would still lean strongly towards the driver doing this.

If you can afford to pop the RTL8111 off, this is probably the fastest/easiest way to test. Should it still fail, I would be very suspicious of link training to be intentionally limited via that register. If I were developing the PCIe driver, I’d certainly add a log message when a device tries for 5GT/s and fallback to 2.5GT/s is required, but I have not seen any such message on my own mini-PCIe.

When you find out, I’ll be curious to know what happens.

Hi,

I’ve got my Jetson to connect occasionally at 5 GT/s:

LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

However, it’s not usable and fills the kernel messages with:

[ 740.901555] pcieport 0000:00:00.0: AER: Corrected error received: id=0010
[ 740.902079] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[ 740.912578] pcieport 0000:00:00.0: device [10de:0e12] error status/mask=00000001/00002000
[ 740.921143] pcieport 0000:00:00.0: [ 0] Receiver Error (First)

When I force my design to connect at 2.5 GT/s it works fine, so I’m thinking this is a signal integrity issue. In my case I have a mini PCIe to PCIe adapter:

http://linitx.com/product/mini-pci-express-card-to-pci-express-x1-socket-9cm-cable-left-sided/12894

which means four connectors and consequent impedance discontinuities.

Mark.

Those messages are pretty much a smoking gun guarantee you are right about signal integrity. What I still have to wonder about is that the original poster did not see error messages. It’s possible that he was not “forcing” the issue and thus PCIe drivers had no reason to report signal problems, that they just did their job and switched to the best reliable speed without complaint. In that case it is very hard to know with certainty without some very expensive tools to actually view signals.

What was the exact method of forcing speed to Gen2? I might experiment with that on one of my mPCIe slot cards.

Yes, I think that’s likely the case. I also tested a design that will do gen2 but usually connects at gen1 speeds to the K1. When it occasionally connects at gen2 speeds I get errors.

In this case the PCIe endpoint is inside an Altera FPGA, which gives me lots of control.

Sorry - just reread what you actually posted. I didn’t force my design to gen2 - I forced it to gen1. The Tegra occasionally negotiates gen2 with my board but it doesn’t actually work reliably.

It occurred to me last night that the Tegra may not have tried gen2 at all until I installed the R21.1 update. I’m in the middle of things right now so I don’t want to go back to R19.3 but it could be that the gen1/gen2 thing is something that got fixed in R21.1. The original post was before R21.1 was out.

The R19 software releases forced PCIe to gen1 speed, but is capable of gen2. R21 releases detect the capability of the cards on the bus and should negotiate gen1/gen2 properly based upon the card.

Something like the following in R19.3 can force gen2, but best to move forward to R21:

diff --git a/drivers/pci/host/pci-tegra.c b/drivers/pci/host/pci-tegra.c
index fe7329e…657bdd0 100644
— a/drivers/pci/host/pci-tegra.c
+++ b/drivers/pci/host/pci-tegra.c
@@ -1105,7 +1105,7 @@ static int tegra_pcie_power_ungate(void)
pr_err(“PCIE: mselect clk enable failed: %d\n”, err);
return err;
}

  • clk_set_rate(tegra_pcie.pcie_mselect, 102000000);
  • clk_set_rate(tegra_pcie.pcie_mselect, 408000000);
    /* pciex is reset only but need to be enabled for dvfs support */
    err = clk_enable(tegra_pcie.pcie_xclk);
    if (err) {
    @@ -1769,7 +1769,7 @@ static void tegra_pcie_enable_features(void)
    PR_FUNC_LINE;

    /* configure all links to gen2 speed by default */

  • if (!tegra_pcie_link_speed(false))
  • if (!tegra_pcie_link_speed(true))
    pr_info(“PCIE: No Link speed change happened\n”);

    tegra_pcie_pll_pdn();

This makes me curious as to the nature of gen1 versus gen2 speed failing on gen2 from R19.x versus R21.1. Restated, would software on R21.1 have exactly the same issues versus R19.3 with the card listed above that had to be backed down to gen1 speeds? How much is trace layout on the card, trace layout on the Jetson, and software? (Just a rhetorical question because the answers are interesting and hard to know).