Jetson Thor Custom Board XDMA DMA Transfer Timeout (PCIe C3)

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

Hi：

I am working on a custom carrier board with Jetson AGX Thor (P3834-0008) and a Xilinx FPGA connected via PCIe C3. I have enabled PCIe C3 by modifying the device tree (uphy0-config=6, status=okay) and the XDMA driver (v2025.2.0) loads successfully. The PCIe device is detected by lspci, and the XDMA driver binds to it (Kernel driver in use: xdma). However, any DMA transfer (read or write) to the FPGA results in a Connection timed out error, even though the interrupt count is incrementing.

Hardware and software environment

  • JetPack version: R38.4.0 (L4T R38.4.0)
  • Kernel version: output of uname - r
  • Development board: self-developed carrier board, unofficial DevKit (Custom Carrier Board, based on P4071)
  • FPGA：Xilinx FPGA with XDMA IP
  • XDMA driver version: Xilinx XDMA v2025.2.0
  • PCIe configuration: C3 enabled, UPHY0-config changed from 0x07 to 0x06, UPHYVNet cleared

Completed troubleshooting steps

  • PCIe device correctly recognized (lspci - d 10ee: display device)

  • XDMA driver successfully loaded and bound (lsmod | grep xdma, dmesg shows probe successful)

  • Control channel (/dev/xdma0_control) reads and writes normally (reg_rw can read 0x1fc00006)

  • Interrupts have been allocated (lspci - v shows IRQ 435), and when attempting DMA transfer, the interrupt count in/doc/interrupts will increase from 48 to 54 (an increase of 6 times)

  • Attempted to use 64 bit addresses (0x1000000 and 0x1100000000) for DMA transfer, both timed out

  • Read XDMA IP core status registers (0x2044, 0x2048, 0x204C), all return values are 0

  • Key log: PCIe bus error in kernel log after DMA timeout:

[ 9793.941469] xdma 0003:01:00.0: PCIe Bus Error: severity=Correctable, type=Transaction Layer, (Receiver ID)
[ 9793.941491] xdma 0003:01:00.0: device [10ee:9032] error status/mask=00002000/0000c000
[ 9793.944979] xdma 0003:01:00.0: [13] NonFatalErr

And the DMA engine status register that drives the dump:

[ 9804.005557] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(…) = 0x00000000 (status).

question

  1. The PCIe link has been established, the driver has been bound, and the interrupt has been triggered, but why does the DMA request still timeout?

  2. Does the PCIe Bus Error: NonFatalErr and xdma: engine_ded_dump: status=0x0000000 in the log indicate that the problem is on the FPGA side (such as AXI bus response, address decoding, or XDMA IP core configuration)?

  3. When enabling PCIe C3 on a self-developed carrier board, besides modifying uphy0 config and status, are there any other necessary hardware (such as Pinmux, reset/clock signals) or software configurations (such as IOMMU/SMMU) that need to be adjusted? For example, I saw in the document that it is necessary to check settings such as PERST #, REFCLK, and hot debug programmable.

  4. A community developer mentioned that on Orin, PCIe DMA needs to handle SMMU/IOMMU correctly, otherwise it will trigger EMEM address decode error. Do similar issues exist on Thor?

2

thor@thor:~/xdma/tools$ sudo lspci -vvv -s 0003:00:00.0
[sudo] password for thor:
0003:00:00.0 PCI bridge: NVIDIA Corporation Device 22da (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin ? routed to IRQ 201
IOMMU group: 11
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled] [32-bit]
Memory behind bridge: 38000000-380fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Express (v2) Root Port (Slot-), MSI 0f
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 512 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x2, ASPM not supported
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 128 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x2
TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
RootCap: CRSVisible+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS+ LN System CLS Not Supported, TPHComp+ ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing- 32bit+ 64bit+ 128bitCAS+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled, ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS+
LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: Downstream Port, DRS-
DownstreamComp: Link Up - Present
Capabilities: [84] MSI: Enable+ Count=16/16 Maskable+ 64bit+
Address: 00000000fffff040 Data: 0000
Masking: 00007ff9 Pending: 00000000
Capabilities: [100 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [12c v1] Data Link Feature
Capabilities: [168 v1] Extended Capability ID 0x2a
Capabilities: [198 v1] Lane Margining at the Receiver
Capabilities: [254 v1] Downstream Port Containment
DpcCap: INT Msg #1, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
DpcCtl: Trigger:1 Cmpl- INT+ ErrCor+ PoisonedTLP- SwTrigger- DL_ActiveErr-
DpcSta: Trigger- Reason:00 INT- RPBusy- TriggerExt:00 RP PIO ErrPtr:1f
Source: 0000
Capabilities: [298 v1] Hierarchy ID <?>
Capabilities: [2b8 v1] Extended Capability ID 0x30
Kernel driver in use: pcieport

3

Moved to Jetson Thor forum

7

Excuse me, isn’t this the Jetson Thor forum?