ConnectX-7 RoCEv2 RC QP out-of-order receive behavior with non-NVIDIA packet-spraying switches

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
, ,
1

We are using ConnectX-7 NICs for RoCEv2 with non-NVIDIA Ethernet switches, for example Intel Tofino2. We would like to implement packet spraying / packet-level load balancing in the switch. This means that packets belonging to the same RoCEv2 RC QP may arrive out of order at the receiver.

I have two questions about the receiver-side behavior of ConnectX-7:

  1. Does ConnectX-7 officially support out-of-order receive / out-of-order data placement for RoCEv2 RC QPs when the packet reordering is caused by a non-NVIDIA switch doing packet spraying?

  2. If this is supported, is there a documented maximum reordering depth / maximum PSN gap that the receiver can tolerate? If this limit is exceeded, what behavior should we expect, e.g., PSN Sequence Error NAK, selective NAK/SACK-like feedback, timeout-based retransmission, or packet drop?

We are trying to understand whether packet-level spraying for the same RoCEv2 RC QP is safe with ConnectX-7, and what limits or configuration requirements we should consider.

2

In certain fabric configurations, RoCE traffic packets for a given QP may take up different paths in a network from the source to the destination.

This results in packets being received in an out-of-order manner.

Data will be placed into host memory in an out-of-order (OOO).

In order to achieve this, adaptive routing needs to be enabled to handle OOO.

AR will be blocked if the HCA is not connected to a Nvidia Spectrum switch (SpectrumX solution only).

3

Thanks, that’s really helpful!

A couple of follow-ups based on what we observe in our own experiments:

We see that even without AR/Spectrum, ConnectX-7 already does direct data placement for out-of-order packets with RDMA_SELECTIVE_REPEAT_EN=1. So the OOO placement capability itself clearly lives in the RNIC, independent of the switch.

Given that:

  1. Why does OOO placement still have to depend on a Spectrum switch? Since placement already works on the RNIC alone, what does the Spectrum switch actually change on the receiver side? Concretely, does the Spectrum switch mark AR-affected packets in some way (e.g. a path-change indication carried in the packet) that the ConnectX-7 receiver uses to distinguish reordering from genuine loss? Or is the Spectrum requirement just a qualification/support boundary rather than an in-band signaling dependency?
  2. In that AR mode with NVIDIA Spectrum switches, is there a max reordering depth (max PSN − expected PSN) or reordering timeout the receiver can tolerate? And once it’s exceeded, what should we expect — a PSN Sequence Error NAK, timeout-based retransmission, or packet drop?

Thanks again!