Executing RDMA accelerated path on Jetson AGX Orin with a PCIe Connectx 5 cx512a card

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
, , , ,
1

Hi,

We have a Jetson AGX Orin 64GB developer board, and we have tested the linux_imx274_player application that comes with holoscan-sensor-bridge. We now intend to shift to the network accelerated version which is imx274_player that uses RDMA. However, we are aware that jetson AGX Orin itself doesn’t have a RDMA capable NIC. As a workaround: -

Can we use a PCIe card such as “Connectx 5 cx512a” [that supports RDMA] on jetson AGX Orin to run the network accelerated imx274_player example?

3

*** Please note that this reply is generated by LLM automatically ***

To answer your question, we need to consider the PCIe configuration of the Jetson AGX Orin Developer Kit and the compatibility of the ConnectX-5 CX512A PCIe card.

According to the provided context, the Jetson AGX Orin Developer Kit has three PCIe controllers:

  1. PCIe C1: Root port, x1 link width, M.2 Key E slot (supports Wi-Fi cards)
  2. PCIe C4: Root port, up to x4 link width, M.2 Key M slot (supports NVMe cards)
  3. PCIe C5: Root port or Endpoint, up to x8 link width, PCIe slot (supports PCIe CEM endpoints, but operates in x8 mode despite being a x16 slot)

The ConnectX-5 CX512A PCIe card is a high-performance, low-latency InfiniBand and Ethernet adapter that supports RDMA. However, its compatibility with the Jetson AGX Orin Developer Kit depends on the PCIe configuration.

The ConnectX-5 CX512A card requires a PCIe x8 or x16 slot to operate at its full potential. Since the Jetson AGX Orin Developer Kit’s PCIe C5 slot is a x16 slot but operates in x8 mode, it might be possible to use the ConnectX-5 CX512A card in this slot. However, the actual performance might be limited by the x8 mode operation.

Before purchasing the ConnectX-5 CX512A card, consider the following:

  1. Compatibility: Verify that the ConnectX-5 CX512A card is compatible with the Jetson AGX Orin Developer Kit’s PCIe configuration. You may need to check the card’s documentation or contact the manufacturer for confirmation.
  2. Performance: Since the PCIe C5 slot operates in x8 mode, the ConnectX-5 CX512A card’s performance might be limited. You may need to evaluate whether the reduced performance will still meet your requirements.
  3. Power consumption: The ConnectX-5 CX512A card is a high-performance adapter that may require significant power. Ensure that the Jetson AGX Orin Developer Kit’s power supply can handle the additional power requirements.
  4. Software support: Verify that the necessary software and drivers are available for the ConnectX-5 CX512A card on the Jetson AGX Orin Developer Kit. You may need to check the manufacturer’s website or contact their support team for more information.

In summary, while it might be possible to use the ConnectX-5 CX512A PCIe card on the Jetson AGX Orin Developer Kit, you should carefully evaluate the compatibility, performance, power consumption, and software support before making a purchase.

As a comparison, you may also consider other PCIe cards that support RDMA and are specifically designed for the Jetson AGX Orin Developer Kit. These cards might be optimized for the kit’s PCIe configuration and provide better performance and compatibility.

To run the network-accelerated imx274player example, you will also need to ensure that the necessary software and drivers are installed and configured correctly on the Jetson AGX Orin Developer Kit. This may involve installing additional packages or configuring the system to use the RDMA-capable NIC.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Hi @carolyuu ,
Thanks for your response.
we had already purchased this Connectx card for a different project that’s why we asked for it specifically.
Does that mean given the compatible PCIe card that supports RDMA and is pcie compatible with Jetson AGX Orin, we can run the network accelerated example “imx274_player” successfully? Has it been done before? If yes, do you have any references that could be useful ?

5

Hi,
imx274_player application is supported on IGX Orin. Not supported on AGX Orin. Please check the statement in

Running Holoscan Sensor Bridge examples - NVIDIA Docs
Examples without “linux_” in the filename use the accelerated network receiver operator and require ConnectX SmartNIC controllers, like those on IGX and DGX Spark.
AGX Orin and AGX Thor systems cannot run these examples.

6

Hi @DaneLLL
any specific reasons why this application is not supported for AGX Orin? because we were thinking to use network acceleration i.e RDMA ROCE based receiver on AGX