PCIe communication between 2 jetson agx orin industrials

tcreighton · March 16, 2026, 8:20pm

Hello,

I am currently trying to set up a PCIe link between 2 jetson agx orin industrials on custom carriers using internal refclks.

I have one of the boards setup as endpoint on C7

ODMData string: gbe-uphy-config-1,hsstp-lane-map-3,hsio-uphy-config-16,nvhs-uphy-config-0

DT node:

pcie-ep@141e0000 {
status = “okay”;
num-lanes = <8>;
nvidia,enable-srns;
/delete-property/ nvidia,enable-ext-refclk;
vddio-pex-ctl-supply = <&vpcie3v3>;
reset-gpios = <&gpio TEGRA234_MAIN_GPIO(K, 3) GPIO_ACTIVE_LOW>;

phys = <&p2u_gbe_0>, <&p2u_gbe_1>, <&p2u_gbe_2>, <&p2u_gbe_3>,
       <&p2u_gbe_4>, <&p2u_gbe_5>, <&p2u_gbe_6>, <&p2u_gbe_7>;
phy-names = "p2u-0", "p2u-1", "p2u-2", "p2u-3", 
            "p2u-4", "p2u-5", "p2u-6", "p2u-7";

};

BPMP edit:

/ {
clocks {
 	clock@plle {
  		clk-id = <0x64>;
  		disable-spread = <1>;
 	};
  
  	clock@pllnvhs {
		clk-id = <0xf3>;
		disable-spread = <1>;
	};
  
	clock@pllgbe {
		clk-id = <0x13f>;
	  	disable-spread = <1>;
	};
};

uphy {
	pcie-c7-endpoint-use-int-refclk;
};
};

I am configuring the endpoint with the following commands:

modprobe pci-epf-test

mkdir -p /sys/kernel/config/pci_ep/functions/pci_epf_test/func1
echo 0x10de > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/vendorid
echo 0x229a > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/deviceid
ln -s /sys/kernel/config/pci_ep/functions/pci_epf_test/func1 func1
echo 1 > /sys/kernel/config/pci_ep/controllers/141e0000.pcie-ep/start

and the other configured as root on C7

ODMData: gbe-uphy-config-0,hsstp-lane-map-3,hsio-uphy-config-16,nvhs-uphy-config-0

DT entry:

pcie@141e0000 {
status = “okay”;
vddio-pex-ctl-supply = <&vpcie3v3>;
nvidia,enable-srns;
num-lanes = <8>;
nvidia,disable-aspm-states = <0xf>;
phys = <&p2u_gbe_0>, <&p2u_gbe_1>, <&p2u_gbe_2>, <&p2u_gbe_3>,
<&p2u_gbe_4>, <&p2u_gbe_5>, <&p2u_gbe_6>, <&p2u_gbe_7>;
phy-names = “p2u-0”, “p2u-1”, “p2u-2”, “p2u-3”,
“p2u-4”, “p2u-5”, “p2u-6”, “p2u-7”;
};

bpmp edit:

clocks {
clock@plle {
clk-id = <0x64>;
disable-spread = <1>;
};

clock@pllnvhs {
	clk-id = <0xf3>;
	disable-spread = <1>;
};

clock@pllgbe {
	clk-id = <0x13f>;
  	disable-spread = <1>;
};

};

Through use of kernel debugging, I have determined the pcie link is up during the UEFI boot, but fails to come up during normal linux boot.

I have verified the TX and RX signals are routed correctly, and verified the reset signal is functional
I have verified the expected device tree and ODMData are being flashed to the boards.

The physical hardware is a 4 lane connection, however when configured for four lanes the endpoint board crashes upon receiving a reset from the host. I found this post: AGX Orin, JP5.1.2 and PCIE Endpoint Troubleshooting - #17 by WayneWWW
that seemed to show switching the endpoint to 8 lanes was a solution to this issue. I have tried with both 4 and 8 lanes on the root side, in both cases the link is up during UEFI boot but fails to come up during normal linux boot.

dmesg on root:

[ 11.706261] tegra194-pcie 141e0000.pcie: Adding to iommu group 8
[ 11.715984] sdhci-tegra 3400000.mmc: Got CD GPIO
[ 11.814165] tegra194-pcie 141e0000.pcie: host bridge /bus@0/pcie@141e0000 ranges:
[ 11.814181] tegra194-pcie 141e0000.pcie: MEM 0x3000000000..0x3227ffffff → 0x3000000000
[ 11.814186] tegra194-pcie 141e0000.pcie: MEM 0x3228000000..0x322fffffff → 0x0040000000
[ 11.814189] tegra194-pcie 141e0000.pcie: IO 0x003e100000..0x003e1fffff → 0x003e100000
[ 11.814662] tegra194-pcie 141e0000.pcie: iATU unroll: enabled
[ 11.814664] tegra194-pcie 141e0000.pcie: Detected iATU regions: 8 outbound, 2 inbound
[ 12.922124] tegra194-pcie 141e0000.pcie: Phy link never came up
[ 13.923981] tegra194-pcie 141e0000.pcie: Phy link never came up
[ 13.924034] tegra194-pcie 141e0000.pcie: PCI host bridge to bus 0007:00
[ 13.924038] pci_bus 0007:00: root bus resource [io 0x300000-0x3fffff] (bus address [0x3e100000-0x3e1fffff])
[ 13.924040] pci_bus 0007:00: root bus resource [mem 0x3228000000-0x322fffffff] (bus address [0x40000000-0x47ffffff])
[ 13.924043] pci_bus 0007:00: root bus resource [bus 00-ff]
[ 13.924045] pci_bus 0007:00: root bus resource [mem 0x3000000000-0x3227ffffff pref]
[ 13.924089] pci 0007:00:00.0: [10de:229a] type 01 class 0x060400
[ 13.924233] pci 0007:00:00.0: PME# supported from D0 D3hot
[ 13.930861] pci 0007:00:00.0: PCI bridge to [bus 01-ff]
[ 13.930951] pcieport 0007:00:00.0: Adding to iommu group 8
[ 13.931027] pcieport 0007:00:00.0: PME: Signaling with IRQ 198
[ 13.931535] pcieport 0007:00:00.0: AER: enabled with IRQ 198
[ 13.931730] sdhci-tegra 3400000.mmc: Got CD GPIO
[ 13.931775] pci_bus 0007:01: busn_res: [bus 01-ff] is released
[ 13.931851] pci 0007:00:00.0: Removing from iommu group 8
[ 13.931862] pci_bus 0007:00: busn_res: [bus 00-ff] is released
[ 13.943992] sdhci-tegra 3400000.mmc: Got CD GPIO

carolyuu · March 16, 2026, 8:34pm

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, it appears that you are experiencing issues with setting up a PCIe link between two Jetson AGX Orin Industrial boards on custom carriers using internal refclks. The link is up during UEFI boot but fails to come up during normal Linux boot.

Here are some key observations and potential solutions:

Lane configuration: You have tried configuring both 4 and 8 lanes on the root side, but the link still fails to come up during normal Linux boot. You may want to try configuring the endpoint to 8 lanes as suggested in the post you mentioned, but also ensure that the root side is configured to match the endpoint’s lane configuration.
Reset signal: You have verified that the reset signal is functional, which is good. However, you may want to double-check that the reset signal is being asserted correctly during normal Linux boot.
Device tree and ODMData: You have verified that the expected device tree and ODMData are being flashed to the boards, which is good. However, you may want to double-check that the device tree and ODMData are correct and match the hardware configuration.
dmesg output: The dmesg output on the root side shows that the PCIe host bridge is being added to the iommu group, and the PCI host bridge is being detected. However, it also shows that the Phy link never came up, which suggests that there may be an issue with the PCIe link.
Comparison with other configurations: You may want to try comparing your configuration with other configurations that are known to work. For example, you could try using a different PCIe endpoint configuration or a different root side configuration.
Kernel debugging: You have already used kernel debugging to determine that the PCIe link is up during UEFI boot but fails to come up during normal Linux boot. You may want to continue using kernel debugging to try to identify the root cause of the issue.
Hardware verification: You have verified that the TX and RX signals are routed correctly, and that the reset signal is functional. However, you may want to double-check that the hardware is functioning correctly and that there are no issues with the PCIe link.

In terms of specific commands or configurations, you may want to try the following:

Configure the endpoint to 8 lanes as suggested in the post you mentioned.
Ensure that the root side is configured to match the endpoint’s lane configuration.
Double-check that the device tree and ODMData are correct and match the hardware configuration.
Use kernel debugging to try to identify the root cause of the issue.
Verify that the hardware is functioning correctly and that there are no issues with the PCIe link.

Here is an example of how you could configure the endpoint to 8 lanes:

PCIE_EP_ADDR=141e0000
modprobe pci-epf-test
mkdir /sys/kernel/config/pci_ep/functions/pci_epf_test/func1
echo 0x10de > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/vendorid
echo 0x229a > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/deviceid
echo 8 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/num_lanes
ln -s /sys/kernel/config/pci_ep/functions/pci_epf_test/func1 /sys/kernel/config/pci_ep/controllers/${PCIE_EP_ADDR}.pcie-ep/
echo 1 > /sys/kernel/config/pci_ep/controllers/${PCIE_EP_ADDR}.pcie-ep/start

Note that this is just an example, and you may need to modify the commands to match your specific configuration.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

tcreighton · March 18, 2026, 2:06pm

I went thought the “Debug PCIe Link-Up Failure” procedure found here:

Adding that kernel patch solved the issue and the link is now coming up. It appears the default ~2s timeout was to short, taking close to 6 seconds to fully come on on my system

Topic		Replies	Views
PCIe to PCIe communication on custom board Jetson AGX Orin pcie , board-design	21	481	August 26, 2025
PCIe Communication Between 2 Jetson Agx Orin Jetson AGX Orin pcie	19	614	March 27, 2025
Connecting AGX Orin (PCIe EP mode) to non-NVIDIA AP via PCIe Jetson AGX Orin pcie , board-design	20	406	March 16, 2026
JP6.2.1 AGX ORIN ethernet switch to pcie can't Link-up Jetson AGX Orin pcie , device-tree	2	170	February 3, 2026
Jetson AGX Orin Endpoint Mode - nothing is working, HELP! Jetson AGX Orin pcie	8	202	October 29, 2025
AGX Orin, JP5.1.2 and PCIE Endpoint Troubleshooting Jetson AGX Orin pcie , board-design	17	1984	November 19, 2023
Two orin agx pcie connect failed Jetson AGX Orin pcie	17	221	April 23, 2026
Jetson AGX Orin PCIe C7 issue Jetson AGX Orin pcie , board-design	2	658	July 18, 2023
Jetson Orin AGX PCIe Endpoint mode ping issues Jetson AGX Orin pcie	4	146	September 10, 2025
PCIE(C7) - is not working for Jetson AGX orin custom board Jetson AGX Orin pcie , board-design	1	625	October 26, 2023

PCIe communication between 2 jetson agx orin industrials

Related topics