How to enable Out-of-Order Data Placement with non-Mellanox switches?

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

Greetings,

We are using CX-5, and want to use OoO data placement for adaptive routing, but using non-mellanox switches (for example, not Quantum/Switch-IB but using Intel Tofino).

In the configuration webpage of Adaptive Routing, it is stated as following:

Adaptive Routing (AR) and Self-Healing Networking (interconnect enhancement for intelligent datacenters) mechanisms in an InfiniBand fabric that utilizes Mellanox switch systems (Quantum ™/Switch-IB® 2 and above) and HCAs (ConnectX®-5 and above).

Here are our questions:

(1) Can we enable the out-of-order receive (e.g., reordering bufferings) for RDMA packets w/o MLX switches? If yes, is it supported by default or how can we enable it?

(2) In the configuration of AR (adaptive routing), we believe we need to specify the topology to “routing_engine” flag (e.g., Dragonfly or fat-tree). Do we need to specify more details of topology, e.g., number of levels in fat-tree?

Thank you in advance to your answers!
Best regards,
Mason

2

Hi,

I want to setup the same thing as you described. Did you figure out how to do it?

Best,
Matthias

3

Hi Matthias,

Unfortunately, to the best of my knowledge, it is supported only on InfiniBand system.

ᐧ

4

ok, that’s unfortunate. But thanks for the response.

5

Hi, Adaptive Routing is supported for Ethernet with NVIDIA Spectrum-X Networking and BlueField-3 devices.

6

That’s great. How can I enable it for ConnectX-5 cards when using a non-Mellanox Switch?

The only page that I could find ESPCommunity mentions a subnet manager, but no specific intructions.

Any help would be appreciated!

7

Hi, in order to get our best certified OOO (e.g. AR), there is a need to connect our NICs/DPUs to NVIDIA switches, Eth (Spectrum) or IB (Quantum)

8

Thanks to correct my answer. I knew but missed the very recent product.

ᐧ

9

In older OFED version there was the environment variable MLX5_RELAXED_PACKET_ORDERING_ON= which was configured on the hosts side. Is there still a way to use this OOO in an “uncertified” way for the current OFED?

10

Hi everyone, the Out-of-Order Data Placement feature looks like a functionality of the network cards, I’m curious as to why it requires special switch support?