How to configure pcie endpoint mode on jetson orin nx

[jowonjin96]

To enable PCIe communication between a custom board and the Jetson Orin NX by setting the Jetson Orin NX as an endpoint, I followed the process outlined in
https://docs.nvidia.com/jetson/archives/r36.4/DeveloperGuide/SD/Communications/PcieEndpointMode.html#flashing-pcie-as-endpoint-on-a-jetson-orin-nx-nano-series-system

After modifying the ODM DATA and running the commands shown in the image on the Jetson device, I booted the custom board.

However, the system logs on the Jetson device only displayed the following message:

Are there additional options or steps required to properly configure the PCIe endpoint?
For example, should I modify the DTB-related settings or make other adjustments to properly configure the PCIe endpoint?

[WayneWWW]

You better sharing full dmesg first and which Jetpack release are you using?

[jowonjin96]

I am currently using NVIDIA Jetson Linux [L4T version 36.4] without JetPack. Please refer to the dmesg logs below.
orin_nx_dmesg.log (62.3 KB)

[WayneWWW]

that regulator log could be ignored.

[jowonjin96]

How can I verify if the PCIe endpoint is properly configured on the Jetson Orin NX? Does the log I shared earlier indicate that the PCIe endpoint has been successfully set up?

I want to connect it to a custom board acting as the Root Complex, but when I run lspci, the Jetson Orin NX does not appear. Could this indicate an issue with the custom board? If it does appear, I am curious about the name it would display under.

[WayneWWW]

Is the dmesg you shared based on the case after you run those commands?

There should be log printed after you run the commands. But I didn’t see them.

[jowonjin96]

jetson_orin_ep_disabled.log (81.0 KB)

After performing “Connecting and Configuring the Devices” and booting the root device, I moved to the directory:

/sys/kernel/debug/14160000.pcie-ep_epf_dma_test/

There, I input the required configuration values and executed cat edmalib_test. However, it returned RP dma address null.

Additionally, when running lspci on the custom board, the NVIDIA-related device is still not detected.
Are there any additional settings I need to configure for the “custom carrier board root complex dma address” or “jetson orin nx endpoint”

or have to change switchutils settings on custom carrier board?

[WayneWWW]

Hi,

Your RP needs to restart after you run this on EP side.

root@jetson-desktop:/sys/kernel/config/pci_ep# echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start

[jowonjin96]

We kept the custom board in BIOS mode and executed the command on the jetson:

echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start
before starting the boot process of the custom board

Is this approach incorrect? Should we boot both the Jetson and the custom board first, execute this command, and then reboot the custom board?

[jowonjin96]

Even after following your suggestion to reboot, the same warning, ’ RP DMA address is null,’ still appears

[WayneWWW]

Hi,

I am talking about lspci on your RP side shall see the device first. Make sure that happened first. If that one didn’t happen, then no need to care about that error log.

Also,

We kept the custom board in BIOS mode and executed the command on the jetson:

What are you talking about here? I have no idea what you are trying to say. What custom board and what BIOS mode you are talking about?

[jowonjin96]

Summary of Steps Taken and Current Issue

1. Compiled the Module
   The file nvidia-oot/drivers/pci/endpoint/functions/pci-epf-dma-test.c was compiled into a kernel module.

2. Modified ODMDATA
   The ODMDATA was overridden with the following configuration: The board was then flashed with the modified configuration.

ODMDATA="gbe-uphy-config-8,hsstp-lane-map-3,hsio-uphy-config-41";

3. Executed Commands on Jetson After Booting
   After booting the Jetson board, the following commands were executed to set up the PCIe Endpoint function:

modprobe pci-epf-dma-test
cd /sys/kernel/config/pci_ep/
mkdir functions/tegra_pcie_dma_epf/func1
echo 0x10de > functions/tegra_pcie_dma_epf/func1/vendorid
echo 0x229a > functions/tegra_pcie_dma_epf/func1/deviceid
echo 16 > functions/tegra_pcie_dma_epf/func1/msi_interrupts
ln -s functions/tegra_pcie_dma_epf/func1 controllers/${PCIE_EP_ADDR}.pcie-ep/
echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start

4. Booted the Custom Board
   The custom board (Root Complex) was powered on.
5. Tested DMA on the Jetson Board
   The following commands were executed on the Jetson board to configure and test DMA:

cd /sys/kernel/debug/14160000.pcie-ep_epf_dma_test/
echo 16777216 > dma_size
echo 4 > nents
echo 1000 > stress_count
echo 0x11 > edma_ch

However, the following message was displayed:

RP DMA address is null

6. Checked the Custom Board
   When running lspci on the custom board, no NVIDIA-related PCIe devices were detected.

What More Can Be Done?

[WayneWWW]

@jowonjin96

Checked the Custom Board
When running lspci on the custom board, no NVIDIA-related PCIe devices were detected.

Is there anything printed in dmesg after you run this command on EP?

root@jetson-desktop:/sys/kernel/config/pci_ep# echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start

And please check cat /proc/interrupts after you run above command on EP side.

[jowonjin96]

we couldn’t find jetson orin nx pcie endpoint in custom board while using after using command in jetson orin nx

lspci
lspci -t

and we checked cat /proc/interrupts after we run command on EP side
jetson_orin_ep_command.log (95.1 KB)

[WayneWWW]

I don’t care anything about your custom board… all I need here is the Jetson side. Where is the dmesg?

[WayneWWW]

000t@jetson-desktop:/sys/kernel/config/pci_ep# cat /proc/interrupts | grep ep
199: 0 0 0 0 GICv3 83 Level tegra-pcie-ep-intr

As interrupt is not toggled, are you sure your hardware is correct?

[jowonjin96]

After checking the PCIe pin map, there doesn’t appear to be any issue between the custom board and the Jetson Orin NX

On the flashed Jetson device, I inspected the file /boot/dtb/kernel_tegra234-p3768-0000+p3767-0000-nv.dtb
using the command fdtdump kernel_tegra234-p3768-0000+p3767-0000-nv.dtb | less
and found that pcie-ep@14160000 is set to disabled.

Shouldn’t I update pcie-ep@14160000 to okay and set pcie@14160000 to disabled instead? If the DTB needs to be modified, where should it be overridden?

Would modifying the DTB actually be a meaningful action?

[WayneWWW]

pcie-ep@14160000 is already enabled. We have overlay files there to handle this.

If your pcie-ep@14160000 is not enabled, your dmesg won’t print anything about it and your first question in this post won’t exist.

[WayneWWW]

The endpoint things are not validated with x86 host. You better using something like Orin AGX to validate this first.

[jowonjin96]

PCIe C4 endpoint mode on Orin NX

Share me the dmesg after your boot up and what did you try on your side.

I am planning to use Linux rel-35 series to enable Ethernet interface over PCIe between the x86 PC and Jetson Orin NX. refer to the above statement

tvnet.zip (9.7 KB)

Additionally, I found an x86 driver related to AGX Orin for PCIe communication between x86 and AGX Orin. In this code, if I modify only the line
#define PCI_DEVICE_ID_NVIDIA_JETSON_AGX_NETWORK 0x2296,
would that be sufficient? If so, could you provide the correct Device ID?