Request a test program of PCI communication

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hi teams
The board we designed has two Xavier Socs, And two Xavier SoCs connect to each other directly by PCIe. Now I throuth the PCIe drive is ok, Both SoCs can print information as follow when exec lspci.
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0003:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)

Now I need a test program of PCIe communication, Could you share one to me ? Thanks

2

In addition, how to check the PCIe in master or slave mode?

4

Can you share more details how two SOC are connected through PCIe?

5

Thanks for your reply.
The hadrward design is show in the figure blew:

A&B
A&B733×711 43.8 KB
A
A1140×599 62.5 KB
B
B1120×602 66 KB

what I want is one Xavier be configured master the other one be configured slave. I am novice to PCIe, I need one test demo to check wither the Xavier SoCs can communicate to each other. could you give it to me? Thanks

6

AGX PCIe C5 controller can be configured as Root Port (Master) or Endpoint mode (Slave)

I hope your platform is following Endpoint design guideline mentioned in below document
https://developer.download.nvidia.com/assets/embedded/secure/jetson/xavier/docs/Jetson_AGX_Xavier_Series_PCIe_Endpoint_Design_Guidelines_Application_Note_v1.2.pdf?Siw3_byIL8-d8VLULGNvh_dyEsTXwVf2KP52OsHwE_Ia35xEWpcrBDm9FveGNLsdKEz2O3vfs1_KjuF7sxIhx1zjNq3jJf-30mOTqhXIcbS0ArBjsi2JUIUscuzk6VPkmtZz4CM6on3UqE-lBmLmyjdaKUI1-YpDwvjrMBPmLvWBpH88s4kwnQHMJOD9trZbEqBToTVTF3VmWYK2AYGF5gA6UUPQSgqKuHCs0oekladD-k4hv49AMxHECA

Jetpack release has tegra virtual Ethernet driver support for SoC to SoC communication testing though PCIe

follow: “Bringing Up an Ethernet Interface Over PCIE”
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/xavier_PCIe_endpoint_mode.html

7

Thanks for your reply.

I download the document of “Jetson_AGX_Xavier_Series_PCIe_Endpoint_Design_Guidelines_Application_Note_v1.2…pdf” and it tips File Not Found or Link has Expired. Could you share this file in the attachment.

When I didn’t change the ODMDATA value from 0x09190000 to 0x09191000，do lspci command there will be for line infomation about PCIe:
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0003:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)

But when I changed the ODMDATA value to 0x09191000 the line of “0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)” will be disappeared. This will be result in the follow step failed, the step is follow:

To allow the PCIe endpoint to respond to PCIe memory accesses on the root port system, enter this command:

setpci -s 0005:01:00.0 COMMAND=0x02

What should i do now ? Thanks.

8

Hi omp,

vendorid_deviceid
vendorid_deviceid1126×587 20.6 KB

where do the values “0x10de, 0x0001” come from?

I do “lspci -v” command, the infomation be printed as follow:
root@we-desktop:/home/we# lspci -v
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 33
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [168] #26
Capabilities: [190] #27
Capabilities: [1c0] L1 PM Substates
Capabilities: [1d0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [2d0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [308] #25
Capabilities: [314] Precision Time Measurement
Capabilities: [320] Vendor Specific Information: ID=0004 Rev=1 Len=054 <?>
Kernel driver in use: pcieport

0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 35
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 40000000-400fffff
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [158] #26
Capabilities: [17c] #27
Capabilities: [190] L1 PM Substates
Capabilities: [1a0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [2a0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [2d8] #25
Capabilities: [2e4] Precision Time Measurement
Capabilities: [2f0] Vendor Specific Information: ID=0004 Rev=1 Len=054 <?>
Kernel driver in use: pcieport

0003:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 37
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [158] #26
Capabilities: [17c] #27
Capabilities: [190] L1 PM Substates
Capabilities: [1a0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [2a0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [2d8] #25
Capabilities: [2e4] Precision Time Measurement
Capabilities: [2f0] Vendor Specific Information: ID=0004 Rev=1 Len=054 <?>
Kernel driver in use: pcieport

0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 39
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [168] #26
Capabilities: [190] #27
Capabilities: [1c0] L1 PM Substates
Capabilities: [1d0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [2d0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [308] #25
Capabilities: [314] Precision Time Measurement
Capabilities: [320] Vendor Specific Information: ID=0004 Rev=1 Len=054 <?>
Kernel driver in use: pcieport

It’s print out four PCI bridges, for example 0005:00:00.0 PCI bridge, the deviceid is “0x1ad0” is it correct ? and there are three Vendor Specific Information ID value 0002, 0001, 0004 which one is the deviceid ? I’m confused. Please help me, Thanks.

9

Please follow this link
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/xavier_PCIe_endpoint_mode.html

You need to flash one Xavier with default ODMDATA for C5 in Rootport mode

and other with 0x09191000 for C5 in Endpoint mode

image
image1343×277 24 KB

Then follow “Bringing Up an Ethernet Interface Over PCIE” section

image
image1384×687 42 KB

10

Hi omp,
Thanks for your reply,
I do step by step as you saied，when enter the lspci command in root port system, there is no info about PCIe device, no detected any PCIe device with vendor id 0x10de and device id 0x2296.
so what the problem is ?

11

Hi omp,
I hava a problem, when i changed the ODMDATA value from 0x9190000 to 0x9191000, and flashing image to the endpoint system. Doing the command “lspci” when endpoint system started, the print infomation cannot find the PCI bridge of 0005:00:00.0, while in root port mode system has this PCI bridge. I did the command (dtc -I fs -O dts -o dts.txt /proc/device-tree/) in endpoint system to decompile dts, in dts.txt file the pcie@141a0000 node had been disabled, But i had config it okay, is it right?

12

This is expected.
after flashing first booted End point mode Xavier and run below command:

cd /sys/kernel/config/pci_ep/
mkdir functions/pci_epf_tvnet/func1
echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
ln -s functions/pci_epf_tvnet/func1 controllers/141a0000.pcie_ep/
echo 1 > controllers/141a0000.pcie_ep/start

Then Boot Root port Mode Xavier. After boot do lspci. you will see PCIe device with name 0005:01:00.0 Nvidia Ethernet

run “ifconfig -a” command on both End Point and Root port Xavier. you will notice new Ethernet interface eth1 on both side.

configure ip address for both eg.
on Root port Xavier: “ifconfig eth1 192.168.5.1 up”
on End point Xavier: “ifconfig eth1 192.168.5.2 up”

now you can verify link using ping command

13

FYI, there is an organization which maintains a registry of IDs which require a fee to register. See:
https://pcisig.com/search/node/vendor

Becoming a member can be expensive.

14

Hi omp,
Thanks to your reply.

When I do this step, There is no PCIe device with name 0005:01:00.0 Nvidia Ethernet

When I do this step, On End Point Xavier has the Ethernet interface eth1, But on Root port Xavier has no.

Does this indicate that there is no connection between the PCIe save and the master or the PCIe master didn’t detected any PCIe device? What‘s the problem ? Thanks again.

15

When I do this step, There is no PCIe device with name 0005:01:00.0 Nvidia Ethernet
When I do this step, On End Point Xavier has the Ethernet interface eth1, But on Root port Xavier has no.

The could be due to C5 controller is not linked with other Xavier C5 controller. Please check your platform schematic about controller mapping