ConnectX-6 Dx: NIC-to-NIC Packet Transfer Without CPU (PeerDirect Support?)

Infrastructure & Networking Software And Drivers Mellanox OFED
1

I have two ConnectX-6 Dx NICs in my server running Ubuntu 22.04.
Is there any way to transmit packets directly from one port to another between these two NICs without CPU involvement?

I couldn’t find any documentation on how to use PeerDirect between two ConnectX-6 NICs.

Any guidance would be appreciated. Thanks!

2

Hello Iwang7,

See below based on your inquiry:

Yes—but not in the way “NIC-to-NIC packet transfer without CPU” is usually imagined. ConnectX-6 Dx supports CPU-bypass data movement, but it relies on RDMA / GPUDirect / PeerDirect-style mechanisms, not a generic NIC-to-NIC forwarding path.

CPU bypass via RDMA (NIC ↔ memory / NIC ↔ NIC logically)

  • ConnectX-6 Dx provides RDMA & RDMA over Converged Ethernet (RoCE).

  • RDMA enables direct memory access between endpoints without CPU involvement, minimizing latency and CPU load.

Practically:

  • NIC A can place data directly into memory on Host B (or GPU memory with GPUDirect).

  • The CPUs orchestrate setup (queue pairs, memory registration), but data path is CPU-bypassed.

3

According to your answer:
CPU bypass appears to only work between two hosts. There seems to be no way to implement CPU-bypass packet forwarding between two NICs within a single endpoint.
Did I get it right?