Connectx-5 (WinOF-2) and RoCE v1

Has anyone managed to get RoCE v1 working on WinOF-2 cards? In this case we are trying to setup ConnectX-5 cards to use RoCE v1 due to backwards compatibility for now.

Existing Infrastructure uses ConnectX-3 Pro (WinOF) with RoCE v1.

New Infrastructure uses ConnectX-5 (WinOF-2).

When setting the new cards to RoCE v1, RDMA is not working between the new hosts themselves.

RoCE v2 works instantly between the new cards.

Funny enough, initiating a connection from an old host X-3 PRO, RoCE v1 to one of the new Hosts running RoCE v2, RDMA works. Has anyone seen this issue?The RoCE Max Frame Size has been set to 2K on the WinOF-2 side to match the maximum MTU for RoCE frames to / from ConnectX-3 Pro cards.

Working and Failed Connections.PNG

Connectx-5 (WinOF-2) and RoCE v1Basically, if configured properly - RoCE v1 should work ok on CX-5 w/WinOF-2

-Global-pause (Flow control) should be enabled

-in the Registry - Set roce_mode (REG_DWORD) to 0 for mlnx5 (Parameters\Roce)

More on RoCEv1 configuration you can find in Chapter 4.2.3.6.1 -Registry Key Parameters

In User Manual od WinOF-2 v2.40 in Mellanox web-site

Hi Avi

Thanks for your reply. I have gone through the Switch and local driver settings and everything seems to be in order. However, RoCE v1 doesn’t seem to appear to be working.

Do you have any methods for troubleshooting this further ?

RoCE v1 on ConnectX-3 Pro Cards works perfectly fine.

Also, establishing a connection from ConnectX-3 Pro Cards running RoCE V1 to a X5 Card running RoCE V2 appears to be working.

“Only” the X5 cards themselves don’t seem to work with V1 so far.

Thanks,

Rob