PCIE-EP mode in AGX orin

salary_korea · August 26, 2024, 9:29am

I’m using AGX orin + custom carrier board
And I’m using L4T R36.3

I want to communicate between two boards by setting one board(pcie-ep mode) and the other board(pcie-rp mode)

PCIE is connected to C7 port with x8 lane,

Set the pcie ep board as follows by referring to the link below,

modprobe pci-epf-dma-test
cd /sys/kernel/config/pci_ep/
mkdir functions/tegra_pcie_dma_epf/func1
echo 0x10de > functions/tegra_pcie_dma_epf/func1/vendorid
echo 0x229a > functions/tegra_pcie_dma_epf/func1/deviceid
echo 16 > functions/tegra_pcie_dma_epf/func1/msi_interrupts
ln -s functions/tegra_pcie_dma_epf/func1 controllers/141e0000.pcie-ep/
echo 1 > controllers/141e0000.pcie-ep/start

https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/Communications/PcieEndpointMode.html?highlight=endpoint

After that, when booting the pcie rp board, it seems that the pcie ep board is not recognized properly.

What parts should I check?

salary_korea · August 26, 2024, 10:10am

Looking at the log below, it looks like it is trying to recognize pcie-ep.

*pcie-ep log

[ 111.339737] tegra194-pcie 141e0000.pcie-ep: iATU unroll: enabled
[ 111.339752] tegra194-pcie 141e0000.pcie-ep: Detected iATU regions: 8 outbound, 2 inbound
[ 111.339845] pcie_dma_epf tegra_pcie_dma_epf.0: BAR0 phy_addr: f0000000 size: 10000000

*pcie-rp log

[ 17.887144] pci 0007:00:00.0: [10de:229a] type 01 class 0x060400
[ 17.887305] pci 0007:00:00.0: PME# supported from D0 D3hot
[ 17.937464] pci 0007:01:00.0: [10de:229a] type 00 class 0x050000
[ 17.937943] pci 0007:01:00.0: reg 0x10: [mem 0x00000000-0x0fffffff 64bit pref]
[ 17.938226] pci 0007:01:00.0: reg 0x18: [mem 0x00000000-0x0001ffff 64bit pref]
[ 17.938510] pci 0007:01:00.0: reg 0x20: [mem 0x00000000-0x00000fff 64bit]
[ 17.941532] pci 0007:01:00.0: PME# supported from D0 D3hot
[ 17.943996] pci 0007:01:00.0: 31.506 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x2 link at 0007:00:00.0 (capable of 126.024 Gb/s with 16.0 GT/s PCIe x8 link)
[ 18.125691] pci 0007:01:00.0: calc_l1ss_pwron: Invalid T_PwrOn scale: 3
[ 18.131599] pci 0007:00:00.0: BAR 15: assigned [mem 0x3000000000-0x3017ffffff 64bit pref]
[ 18.131610] pci 0007:00:00.0: BAR 14: assigned [mem 0x3228000000-0x32280fffff]
[ 18.131615] pci 0007:01:00.0: BAR 0: assigned [mem 0x3000000000-0x300fffffff 64bit pref]
[ 18.131623] pci 0007:01:00.0: BAR 0: error updating (0x00000c != 0xffffffff)
[ 18.131628] pci 0007:01:00.0: BAR 0: error updating (high 0x000030 != 0xffffffff)
[ 18.131632] pci 0007:01:00.0: BAR 2: assigned [mem 0x3010000000-0x301001ffff 64bit pref]
[ 18.131639] pci 0007:01:00.0: BAR 2: error updating (0x1000000c != 0xffffffff)
[ 18.131642] pci 0007:01:00.0: BAR 2: error updating (high 0x000030 != 0xffffffff)
[ 18.131646] pci 0007:01:00.0: BAR 4: assigned [mem 0x3228000000-0x3228000fff 64bit]
[ 18.131652] pci 0007:01:00.0: BAR 4: error updating (0x40000004 != 0xffffffff)
[ 18.131656] pci 0007:01:00.0: BAR 4: error updating (high 0x000000 != 0xffffffff)
[ 18.131660] pci 0007:00:00.0: PCI bridge to [bus 01-ff]
[ 18.131665] pci 0007:00:00.0: bridge window [mem 0x3228000000-0x32280fffff]
[ 18.131669] pci 0007:00:00.0: bridge window [mem 0x3000000000-0x3017ffffff 64bit pref]
[ 18.131924] pcieport 0007:00:00.0: Adding to iommu group 58
[ 18.132067] pcieport 0007:00:00.0: PME: Signaling with IRQ 202
[ 18.132243] pcieport 0007:00:00.0: AER: enabled with IRQ 202
[ 18.132643] pcie_dma_host 0007:01:00.0: Adding to iommu group 58
[ 18.132667] pcie_dma_host 0007:01:00.0: of_irq_parse_pci: failed with rc=134
[ 18.132689] pcie_dma_host 0007:01:00.0: can’t change power state from D3cold to D0 (config space inaccessible)
[ 18.132835] pcie_dma_host 0007:01:00.0: Failed to enable MSI interrupt
[ 18.133375] pci 0007:01:00.0: Removing from iommu group 58
[ 18.133400] pci_bus 0007:01: busn_res: [bus 01-ff] is released
[ 18.133801] pci 0007:00:00.0: Removing from iommu group 58
[ 18.133834] pci_bus 0007:00: busn_res: [bus 00-ff] is released

WayneWWW · August 27, 2024, 2:31am

Could you also try if this would work on rel-35.5?

WayneWWW · August 27, 2024, 3:07am

Actually there are some extra patches added for PCIe C5 to support endpoint.

If you want to use C7, then you need to make sure similar thing exists.

diff --git a/nv-platform/tegra234-p3737-0000+p3701-xxxx-nv-common.dtsi b/nv-platform/tegra234-p3737-0000+p3701-xxxx-nv-common.dtsi
index 848f943..e190bbd 100644
--- a/nv-platform/tegra234-p3737-0000+p3701-xxxx-nv-common.dtsi
+++ b/nv-platform/tegra234-p3737-0000+p3701-xxxx-nv-common.dtsi
@@ -1,5 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0-only
-// SPDX-FileCopyrightText: Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
 
 #include "nv-soc/tegra234-overlay.dtsi"
 #include "nv-soc/tegra234-soc-thermal.dtsi"
@@ -306,6 +306,12 @@
 		gpu@17000000 {
 			status = "okay";
 		};
+
+		pcie-ep@141a0000 {
+			nvidia,refclk-select-gpios = <&gpio
+							TEGRA234_MAIN_GPIO(Q, 4)
+							GPIO_ACTIVE_HIGH>;
+		};
 	};
 
 	tegra-hsp@b950000 {
diff --git a/nv-soc/tegra234-soc-overlay.dtsi b/nv-soc/tegra234-soc-overlay.dtsi
index d77f1c8..7ff1bc8 100644
--- a/nv-soc/tegra234-soc-overlay.dtsi
+++ b/nv-soc/tegra234-soc-overlay.dtsi
@@ -1,5 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0-only
-// SPDX-FileCopyrightText: Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
 //
 // This file contains the DT nodes of T234 which are not in base/tegra234.dtsi
 
@@ -99,7 +99,49 @@
 					nvidia,lpdr = <TEGRA_PIN_DISABLE>;
 				};
 			};
-
+			pex_rst_c5_in_state: pex_rst_c5_in {
+				pex_rst {
+					nvidia,pins = "pex_l5_rst_n_paf1";
+					nvidia,function = "rsvd1";
+					nvidia,pull = <TEGRA_PIN_PULL_NONE>;
+					nvidia,tristate = <TEGRA_PIN_ENABLE>;
+					nvidia,enable-input = <TEGRA_PIN_ENABLE>;
+					nvidia,lpdr = <TEGRA_PIN_DISABLE>;
+				};
+			};
+			pex_rst_c6_in_state: pex_rst_c6_in {
+				pex_rst {
+					nvidia,pins = "pex_l6_rst_n_paf3";
+					nvidia,function = "rsvd1";
+					nvidia,pull = <TEGRA_PIN_PULL_NONE>;
+					nvidia,tristate = <TEGRA_PIN_ENABLE>;
+					nvidia,enable-input = <TEGRA_PIN_ENABLE>;
+					nvidia,io-high-voltage = <TEGRA_PIN_ENABLE>;
+					nvidia,lpdr = <TEGRA_PIN_DISABLE>;
+				};
+			};
+			pex_rst_c7_in_state: pex_rst_c7_in {
+				pex_rst {
+					nvidia,pins = "pex_l7_rst_n_pag1";
+					nvidia,function = "rsvd1";
+					nvidia,pull = <TEGRA_PIN_PULL_NONE>;
+					nvidia,tristate = <TEGRA_PIN_ENABLE>;
+					nvidia,enable-input = <TEGRA_PIN_ENABLE>;
+					nvidia,io-high-voltage = <TEGRA_PIN_ENABLE>;
+					nvidia,lpdr = <TEGRA_PIN_DISABLE>;
+				};
+			};
+			pex_rst_c10_in_state: pex_rst_c10_in {
+				pex_rst {
+					nvidia,pins = "pex_l10_rst_n_pag7";
+					nvidia,function = "rsvd1";
+					nvidia,pull = <TEGRA_PIN_PULL_NONE>;
+					nvidia,tristate = <TEGRA_PIN_ENABLE>;
+					nvidia,enable-input = <TEGRA_PIN_ENABLE>;
+					nvidia,io-high-voltage = <TEGRA_PIN_ENABLE>;
+					nvidia,lpdr = <TEGRA_PIN_DISABLE>;
+				};
+			};
 			eqos_mii_rx_input_state_disable: eqos_rx_disable {
 				eqos {
 					nvidia,pins = "eqos_rd0_pe6","eqos_rd1_pe7",
@@ -868,6 +910,38 @@
 			status = "disabled";
 		};
 
+		pcie-ep@141a0000 {
+			pinctrl-names = "default";
+			pinctrl-0 = <&pex_rst_c5_in_state>;
+
+			num-ib-windows = <2>;
+			num-ob-windows = <8>;
+		};
+
+		pcie-ep@141c0000 {
+			pinctrl-names = "default";
+			pinctrl-0 = <&pex_rst_c6_in_state>;
+
+			num-ib-windows = <2>;
+			num-ob-windows = <8>;
+		};
+
+		pcie-ep@141e0000 {
+			pinctrl-names = "default";
+			pinctrl-0 = <&pex_rst_c7_in_state>;
+
+			num-ib-windows = <2>;
+			num-ob-windows = <8>;
+		};
+
+		pcie-ep@140e0000 {
+			pinctrl-names = "default";
+			pinctrl-0 = <&pex_rst_c10_in_state>;
+
+			num-ib-windows = <2>;
+			num-ob-windows = <8>;
+		};
+
 		hsp_top2: hsp@1600000 {
 			compatible = "nvidia,tegra234-hsp";
 			reg = <0x0 0x1600000 0x0 0x90000>;

These were for C5.

salary_korea · August 27, 2024, 6:16am

The parts you mentioned seem to be already apply it.

I am using a sperated clock architecture on a custom carrier board.
Could this affect it?

salary_korea · August 27, 2024, 7:13am

Here’s my PCIE endpoint dtsi
(It’s the result of “dtc -I fs -O dts /proc/device-tree”)

pcie-ep@141e0000 {
  	power-domains = <0x03 0x10>;
  	iommus = <0xe4 0x08>;
  	nvidia,host1x = <0x101>;
  	pinctrl-names = "default";
  	dma-coherent;
  	interconnect-names = "dma-mem\0write";
  	phy-names = "p2u-0\0p2u-1\0p2u-2\0p2u-3\0p2u-4\0p2u-5\0p2u-6\0p2u-7";
  	nvidia,bpmp = <0x03 0x07>;
  	pinctrl-0 = <0x11d>;
  	clock-names = "core";
  	interconnects = <0x51 0x2a 0x52 0x51 0x30 0x52>;
  	reg-names = "appl\0atu_dma\0dbi\0addr_space";
  	nvidia,aspm-l0s-entrance-latency-us = <0x03>;
  	num-ob-windows = <0x08>;
  	resets = <0x03 0x0f 0x03 0x0e>;
  	interrupts = <0x00 0x162 0x04>;
  	clocks = <0x03 0xab>;
  	nvidia,enable-ext-refclk;
  	reset-gpios = <0xe7 0xc1 0x01>;
  	num-lanes = <0x08>;
  	compatible = "nvidia,tegra234-pcie-ep";
  	iommu-map-mask = <0x00>;
  	status = "okay";
  	interrupt-names = "intr";
  	phys = <0x115 0x116 0x117 0x118 0x119 0x11a 0x11b 0x11c>;
  	reg = <0x00 0x141e0000 0x00 0x20000 0x00 0x3e040000 0x00 0x40000 0x00 0x3e080000 0x00 0x40000 0x2e 0x40000000 0x04 0x00>;
  	nvidia,refclk-select-gpios = <0xe7 0xc0 0x00>;
  	iommu-map = <0x00 0xe4 0x08 0x1000>;
  	reset-names = "apb\0core";
  	nvidia,aspm-pwr-on-t-us = <0x14>;
  	nvidia,aspm-cmrt-us = <0x3c>;
  	num-ib-windows = <0x02>;
  };

WayneWWW · August 27, 2024, 7:29am

Did you ever test this hardware on rel-35 before or this is your first time using it?

Do you have full dmesg to share?

And please share all the configuration you’ve done during flash.

WayneWWW · August 27, 2024, 7:38am

Do you have schematic to share for this part? This does affecting.

salary_korea · August 27, 2024, 8:31am

reference clock schematic as below
Y5200 is oscilator(100Mhz)

salary_korea · August 27, 2024, 8:38am

here’s ep, rp dmesg log

pcie-ep_dmesg.txt (63.0 KB)
pcie-rp_dmesg.txt (100.0 KB)

WayneWWW · August 27, 2024, 8:40am

You need to dump ep log after enabling the driver.

salary_korea · August 27, 2024, 8:53am

After enabling pcie-ep, log files

at the pcie-rp log, there are a lot of error messages.

pcie-ep_dmesg_24082701.txt (64.6 KB)
pcie-rp_serial_log_24082701.txt (135.5 KB)

WayneWWW · August 27, 2024, 8:57am

Answer this previous missing question. I think this is needed.

salary_korea · August 27, 2024, 9:01am

What file should I give you?

WayneWWW · August 27, 2024, 9:01am

I don’t really need any file yet. Just describe what did you change first…
And please only tell the change related to this PCIe EP thing.

salary_korea · August 27, 2024, 9:08am

I added the following on tegra234-p3737-0000+p3701-0000-dynamic.dts

   /* PCIe C7 endpoint */
   fragment-t234-p3737-0000-p3701-0000-pcie-c7-ep@0 {
           target-path = "/bus@0";
           board_config {
                   odm-data = "gbe-uphy-config-1";
           };
           __overlay__ {
                   pcie@141e0000 {
                           status = "disabled";
                   };
                   pcie-ep@141e0000 {
                           status = "okay";
                   };
           };
   };

and I added the following on tegra234-p3737-0000+p3701-0000.dts

			pcie-ep@141e0000 {
                   status = "disabled";

                   vddio-pex-ctl-supply = <&vdd_1v8_ls>;

                   reset-gpios = <&gpio TEGRA234_MAIN_GPIO(AG, 1) GPIO_ACTIVE_LOW>;

                   nvidia,refclk-select-gpios = <&gpio TEGRA234_MAIN_GPIO(AG, 0) GPIO_ACTIVE_HIGH>;

                   phys = <&p2u_gbe_0>, <&p2u_gbe_1>, <&p2u_gbe_2>,
                           <&p2u_gbe_3>, <&p2u_gbe_4>, <&p2u_gbe_5>,
                           <&p2u_gbe_6>, <&p2u_gbe_7>;
                   phy-names = "p2u-0", "p2u-1", "p2u-2", "p2u-3", "p2u-4",
                           "p2u-5", "p2u-6", "p2u-7";
           };

WayneWWW · August 27, 2024, 9:13am

And ODMDATA is correctly set?

salary_korea · August 27, 2024, 9:15am

*RP setting
ODMDATA=“gbe-uphy-config-0,hsstp-lane-map-3,hsio-uphy-config-16,nvhs-uphy-config-0”;

*EP setting
ODMDATA=“gbe-uphy-config-1,hsstp-lane-map-3,hsio-uphy-config-16,nvhs-uphy-config-0”;

WayneWWW · August 27, 2024, 9:41am

Please check the interrupts on your EP side for tegra_pcie_7_pex_rst_irq.

root@jetson:/sys/kernel/config/pci_ep# cat /proc/interrupts | grep -i pex

The count of this interrupt shall go up when RP goes for the link up. If the count is going up, then, it confirms that EP is able to perceive the PERST# transitions from RP.

If it cannot, I think hardware review is needed.

salary_korea · August 27, 2024, 9:53am

cat /proc/interrupts |grep -i pex
300: 14 0 0 0 0 0 0 0 0 0 0 0 2200000.gpio 157 Edge tegra_pcie_7_pex_rst_irq

It(tegra_pcie_7_pex_rst_irq) seems to be increasing

Topic		Replies	Views
Pcie offline Jetson AGX Orin pcie	13	1334	September 13, 2024
PCIE(C0) - is not working for Jetson AGX orin Jetson AGX Orin pcie	15	1055	October 10, 2023
Jetson Orin NX, custom board, add M.2 on pcie2, pcieport error Jetson Orin NX pcie , board-design	2	267	April 23, 2024
PCIe adoptation Jetson AGX Orin pcie	58	676	July 2, 2024
Enabling Orin Dev Kit PCIe EP mode Jetson AGX Orin pcie	22	5445	December 20, 2022
Use PCIe to communicate between two Xaviers: RP phy dosen't up Jetson AGX Xavier	17	2077	October 6, 2021
PCIe Bus Error message Jetson TK1	40	8489	October 18, 2021
PCIE export error Jetson TX1	4	2737	July 5, 2018
Assert on NVME PCIE Boot Jetson Orin NX boot , board-design , nvme	17	1514	August 30, 2023
AGX Orin, JP5.1.2 and PCIE Endpoint Troubleshooting Jetson AGX Orin pcie , board-design	18	1189	November 19, 2023

PCIE-EP mode in AGX orin

Related topics