PCIE Lane usage & questions: Jetson AGX Orin

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Hello, we are evaluating Jetson AGX Orin 64GB for an internal project.
For Jetson AGX Orin 64GB, can we confirm there are 2 PCIE x8 slots available?

We would also like confirmation on if our usecase would be supported:
PCIE devices 1, 2, 3 (PCIE Gen 3) is streaming in data, must be in sync with no data drops.
PCIE device 4 is SSD.

Usecase 1:
Device 1 , 2, 3 at x2 lanes each-> lane splitter → PCIE slot 1
Device 4 at x4 lane → PCIE slot 2

Usecase 2:
Device 1, 2, at x4 lanes each → lane splitter → PCIE slot 1
Device 3, 4 at x4 lanes each → lane splitter → PCIE slot 2

For usecase 2, we are concerned if we can retrieve data and send directly to GPU for CUDA in a synced way despite the 2 different slots.

2

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, I can confirm that the Jetson AGX Orin 64GB has two PCIe x8 slots available. However, to confirm the availability of these slots, I would recommend verifying the specific configuration of your Jetson AGX Orin 64GB module, as the context does not explicitly mention the 64GB variant.

Regarding your use case, I will break down the requirements and provide an analysis of each scenario.

Use Case 1:

  • Device 1, 2, 3 at x2 lanes each (via lane splitter) → PCIe slot 1
  • Device 4 at x4 lanes → PCIe slot 2

In this scenario, you are utilizing the two PCIe x8 slots, with the first slot split into three x2 lanes for Devices 1, 2, and 3, and the second slot used for Device 4 at x4 lanes. This configuration is feasible, as the Jetson AGX Orin supports PCIe Gen 3, which can provide sufficient bandwidth for your devices.

However, to ensure that the data from Devices 1, 2, and 3 is streamed in sync with no data drops, you will need to verify that the lane splitter is properly configured and that the devices are synchronized. Additionally, you should ensure that the PCIe slot 1 can handle the aggregated bandwidth of the three devices.

Use Case 2:

  • Device 1, 2 at x4 lanes each (via lane splitter) → PCIe slot 1
  • Device 3, 4 at x4 lanes each (via lane splitter) → PCIe slot 2

In this scenario, you are utilizing both PCIe x8 slots, with each slot split into two x4 lanes for Devices 1, 2 and Devices 3, 4, respectively. This configuration is also feasible, but you should verify that the lane splitters are properly configured and that the devices are synchronized.

Regarding your concern about retrieving data and sending it directly to the GPU for CUDA processing in a synced way, despite the two different slots, it is technically possible. However, it will depend on the specific implementation and the capabilities of your system.

The Jetson AGX Orin supports PCIe Gen 3, which provides a high-bandwidth interface for data transfer between the devices and the GPU. Additionally, the NVIDIA CUDA architecture allows for direct memory access (DMA) between the GPU and the PCIe devices, which can enable efficient data transfer and processing.

To achieve synced data transfer and processing, you may need to implement a custom solution that utilizes the CUDA API and the PCIe device drivers to manage the data transfer and synchronization between the devices and the GPU. This may require significant development and optimization efforts to ensure that the data is transferred and processed efficiently and in sync.

In summary, both use cases are feasible, but you should verify the specific configuration of your Jetson AGX Orin 64GB module and ensure that the lane splitters are properly configured. Additionally, you will need to implement a custom solution to achieve synced data transfer and processing between the devices and the GPU.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Please refer to the supported PCIe mappings below from the Jetson AGX Orin Series Design Guide (including AGX Orin Industrial).

Your usecase 1 would be short of one x2 lane interface since the C0 and C1 controllers only support x1. The bifurcation in your usecase 2 is not directly supported by the two x8 interfaces from the Orin module. A PCIe switch could be added in between but if the device 4 SSD is behind the switch, it is not guaranteed to work as a boot device.

For your usecase 1:

  • The SSD could be directly on one of the x8-capable PCIe controllers, C5 or C7.
  • One of the x2 lane devices could be directly on C4.
  • The other two x2 lane devices could be on the available one of C5 or C7 with a PCIe switch in between.

image
image607×798 42.1 KB

5

Hi Chris! Thanks for your response.

We aren’t planning to use the SSD as boot device.

Can you confirm if we use a PCIe switch, the device will see the 2 x2 lane devices as 2 separate PCIE devices?

Do you recommend our usecase for Jetson AGX Orin 64GB or should we consider IGX Thor ? I believe IGX Thor has x16 + x8 slot.

6

The two x2 devices on the switch would be seen as two separate PCIe devices by the root port on Orin.

(Note: the comments below only focus on the supported PCIe interfaces and do not consider other major factors like performance and power compared to Orin)

Which IGX Thor do you mean?

IGX T5000 has one less PCIe x8 compared to Jetson AGX Orin (see below from Chapter 7 of IGX T5000 Modules Design Guide).

The IGX Thor Developer Kit (IGX Thor Developer Kit Product Specification) and the IGX T7000 Board Kit (NVIDIA IGX T7000 Product Specification) have x8 and x16 slots through a ConnectX7 but the x16 is intended for a discrete GPU like the RTX Pro 6000 Blackwell MaxQ. In both cases, the Thor module is the T5000 described above, so there is only the one x8 from the SoM to the ConnectX7.

image
image602×542 33.5 KB

7

For IGX Thor Developer Kit, IGX T7000 Board Kit, are we able to use the x16 as a generic PCIE device slot?

8

We cannot confirm support for other types of PCIe devices on the x16. It is intended for and validated with the RTX Pro 6000 Blackwell MaxQ discrete GPU.