RoCE MLX5: Relaxing ordering requirements for incoming Reads/Writes

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi,

I have a 100G Mellanox RDMA NIC (MT27800 Family [ConnectX-5]) installed for RoCE and setup lossy RoCE Accelerations according to https://community.mellanox.com/s/article/How-to-Enable-Disable-Lossy-RoCE-Accelerations

In my current setup it is possible that the network switch reorders packets within a QP that are sent to my RDMA NIC. The packets are in quick succession e.g. there is less than microsecond delay between the reordesr.

For example the following trace can happen:
RDMA Write Only (PSN: 12376722)
RDMA Read Req (PSN: 12376724) // comes in to early
RDMA Write Only (PSN: 12376723)

The NIC then replies with a CNP Packet and a NAK (PSN Error for 12376723)

Is there a way to relax these constraints on my NIC, e.g. by modifying some registers? I could do a similar thing to disable the iCRC32 validation. As of now I have no way to enforce correct ordering inside the network, therefore I look for a way such that the NIC accepts these packet traces.

Any help would be appreciated!

Best,
Matthias

PS:

  • Attached is a screenshot of the packet dump.
  • If I’m in the wrong sub-forum, please move this topic.

2023-08-18-132037_1097x935_scrot
2023-08-18-132037_1097x935_scrot1097×935 342 KB

2

I think further analysis would need to take place first upon reviewing the relevant data and information from your deployment and have a clear understanding of what you are trying to accomplish.
This post only refers to enable/disable these features.

My recommendation would be to open a support case should you have a support contract with us so we can further assist you.
(ESPCommunity).

3

To create a case ESPCommunity (With a valid support contract).