Is jetson Xavier inter-board PCIe connection possible

[aradmosh]

Hi.
I have seen that TX1/2 interboard pcie connection is not possible as they only support root port. meaning we will need a PCIe switch with non-transparent (NT) bridging.
Now in Xavier I have read that it supports also endpoint mode (“End Point mode is supported on Interfaces C0, C4 & C5 only.”).
So my question is can we connect 2 Xavier “back to back” meaning connect 2 devices through the PCIe connector ? and if so how ? (is it feasible ).

[dusty_nv]

Hi aradmosh, please refer to these posts relevant to your question:

• https://devtalk.nvidia.com/default/topic/1039469/jetson-agx-xavier/how-another-cpu-communicate-with-xavier-through-pcie-/post/5281876/#5281876
• https://devtalk.nvidia.com/default/topic/1039469/jetson-agx-xavier/how-another-cpu-communicate-with-xavier-through-pcie-solved-/post/5288520/#5288520

[aradmosh]

Hi Dustin and thanks for the quick reply. (need a bit of elaboration).
I see here we are talking about PC (X86) to xavier.
maybe it’s more or less the same, I was asking about Xavier to Xavier communication .
I found here Tegra Xavier - Nvidia - WikiChip a quote.
“It’s worth noting that Xavier added support for an end-point mode in addition to the standard root complex support. This support meant they can connect two Xaviers directly one to another (2-way multiprocessing) without going through a PCIe switch or alike.”

1. Why is it worth nothing ? if I can connect without a switch (NT bridging) it’s more than something :) .
2. Is it 2-way multiprocessing ? meaning each xavier is root and endpoint ?
3. Can I buy (and where) a male to male pcie cable and connect it between them ?
4. Should I configure the endpoints in cross (root and endpoint connected to endpoint and root respectively).

[dusty_nv]

“It’s worth noting that Xavier added support for an end-point mode in addition to the standard root complex support. This support meant they can connect two Xaviers directly one to another (2-way multiprocessing) without going through a PCIe switch or alike.”

1. Why is it worth nothing ? if I can connect without a switch (NT bridging) it’s more than something :) .

This says it “worth noting”, i.e. noteworthy, not “worth nothing”.

2. Is it 2-way multiprocessing ? meaning each xavier is root and endpoint ?

Each Jetson Xavier has 5 PCIe endpoints, 3 of them can be switched between root or endpoint mode, and 2 of them are root-only.

3. Can I buy (and where) a male to male pcie cable and connect it between them ?

I do not currently know of a generally-available adapter for doing this @ gen 4. A dedicated carrier board may be better suited to the task.

It is warned against to try random cables for this task, as the power rails must be disconnected to prevent power back-drive between systems, and the RX/TX pairs would need to be swapped.

4. Should I configure the endpoints in cross (root and endpoint connected to endpoint and root respectively).

Yes, an endpoint will need to be connected to a root port and vice-versa.

[aradmosh]

>>worth noting
Now I noticed :)

>>I do not currently know of a generally-available adapter for doing this
I have 2 xavier devices on my desk , will try to find an adapter for that (at least for one direction). At first will try that lspci command will show the other device (PCI) (should be a good start).

I noticed in the post your refered me to , it is mentioned that “There is a platform driver (pcie-tegra-dw-ep.c) available to configure controller for endpoint mode” any where I can read a bit more about that so I’ll be able to configure one of the controllers to endpoint mode ?

[linuxdev]

I am curious…on the dev kit carrier, is the exposed x8 going to one of the controllers which can act as an end point?

[dusty_nv]

I am curious…on the dev kit carrier, is the exposed x8 going to one of the controllers which can act as an end point?

Yes, as per section 1.9.4 of the Jetson AGX Xavier Module Datasheet, the PCIe x8 controller supports both root port and endpoint modes.