PCIe C1 connect 5G module facing MSI interrupts issue

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Hi,

Currently we are using a custom carrier board with an external Quectel RM520N 5G module via PCIe C1, the device tree is as follows:

output_agx.txt (317.1 KB)

Currently meeting the timing requirements of the 5G module, PCIe devices can be scanned. The 5G module responded to the MSI interrupt sent by AGX and replied, but it seems that AGX did not receive the response.

Hardware question: The AGX has only one PCIe_WAKE pin. Is it possible to connect multiple devices to it? we have NVME and 5G module.

tegra-ubuntu:~$ sudo cat /proc/interrupts | grep MSI
239:     22     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395200 Edge   nvme0q0
240:   17219     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395201 Edge   nvme0q1
241:   19865     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395202 Edge   nvme0q2
242:   22818     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395203 Edge   nvme0q3
243:   18634     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395204 Edge   nvme0q4
244:    5410     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395205 Edge   nvme0q5
245:    4597     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395206 Edge   nvme0q6
246:    4044     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395207 Edge   nvme0q7
247:    4326     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 537395208 Edge   nvme0q8
263:     0     0     0     0     0     0     0     0     0     0     0     0  PCI-MSI 134742016 Edge   mhi

0001:01:00.0 Unassigned class [ff00]: Qualcomm Device 0308
        Subsystem: Qualcomm Device 5201
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin ? routed to IRQ 263
        IOMMU group: 5
        Region 0: Memory at 20a8000000 (64-bit, non-prefetchable) [size=4K]
        Region 2: Memory at 20a8001000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
                Address: 00000000fffff950  Data: 0000
                Masking: fffffffe  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 16GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <1us, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (downgraded), Width x1 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp+ ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: Upstream Port
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [168 v1] Physical Layer 16.0 GT/s <?>
        Capabilities: [18c v1] Lane Margining at the Receiver <?>
        Capabilities: [19c v1] Transaction Processing Hints
                No steering table available
        Capabilities: [228 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [230 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=70us PortTPowerOnTime=0us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=98304ns
                L1SubCtl2: T_PwrOn=40us
        Capabilities: [240 v1] Data Link Feature <?>
        Kernel driver in use: mhi_q
        Kernel modules: pcie_mhi

AGX log:

dmesg.txt (87.3 KB)

5G module log:

modem.txt (396.4 KB)

3

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, I’ll address the questions and provide a comprehensive answer.

PCIe Interface and Signal Connections

The Jetson Orin NX Series and Jetson Orin Nano Series Product Design Guide provides information on the PCIe interface and signal connections. The guide states that the Orin module brings four PCIe interfaces to the module pins for up to seven total lanes (1 x4 + 1 x1 + 1x2) for use on the carrier board. The PCIe x4 interface (PCIE0) operates up to Gen4 speed and supports both Root Port and Endpoint operation.

PCIeWAKE Pin and Multiple Device Connection

Regarding the PCIeWAKE pin, the guide does not explicitly state that it is possible to connect multiple devices to it. However, it does mention that the PCIeWAKE pin is used for wake-up signals. In general, it is possible to connect multiple devices to a single wake-up signal pin, but it depends on the specific implementation and the requirements of the devices.

In this case, you have an NVME device and a 5G module connected to the AGX via PCIe. The 5G module is responding to the MSI interrupt sent by AGX, but AGX is not receiving the response. This could be due to various reasons such as signal integrity issues, incorrect device tree configuration, or hardware limitations.

Device Tree Configuration and Signal Integrity

The provided device tree configuration and signal integrity requirements are crucial for ensuring proper communication between the AGX and the connected devices. The guide provides recommendations for signal routing, trace impedance, and via proximity to ensure reliable communication.

To resolve the issue, I recommend:

  1. Verify the device tree configuration to ensure that it is correctly configured for the connected devices.
  2. Check the signal integrity of the PCIe signals, including the PCIeWAKE pin, to ensure that they meet the recommended requirements.
  3. Consult the datasheet of the Quectel RM520N 5G module to ensure that it is compatible with the AGX and that the wake-up signal is properly configured.
  4. Consider using a signal analyzer or oscilloscope to monitor the PCIe signals and wake-up signals to identify any potential issues.

By following these steps, you should be able to identify and resolve the issue preventing the AGX from receiving the response from the 5G module.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

5

We found that after disabling the IOMMU of PCIe C1( followed this topic), the AGX can receive the MSI interrupts sent by the 5G module. We are currently using Jetpack 6.0 and have configured PCIe according to the PCIe configuration method in the Jetson AGX Orin Platform Adaptation and Bring-Up. Is there any other IOMMU configuration that needs to be done?

Thanks

6

Sorry for the late response.
Is this still an issue to support? Any result can be shared?

7

Hi kayccc,

We are able to receive MSI interrupts after disabling the IOMMU for PCIe C1.
Is this a known issue or I missed something? Could you please explain the reason behind this behavior?

Thanks.

8

Probably due to this specific PCIe device. No other user reporting this issue before.

9

Sure, thank you, WayneWWW.