RDMA without switch? Is it even possible?

Infrastructure & Networking Adapters and Cables
,
1

Hello,

I just want to check if it is even possible to get RDMA running on two Widnows Datacenter 2019 (only for Scale-Out File Server)

RDMA
RDMA1489×1505 526 KB
hosts with two ConnectX-3 Pro RDMA adapters - connected directly (without switch).

I’ve got them configured, IP addreses are “pingable”, however RDMA traffic is not going.

2

Hi Piotr,

Yes, RDMA communication between two Windows servers (without a switch) is possible.

Please install WinOF, which is the Windows driver for ConnectX-3 and ConnectX-3 Pro network adapter cards. Downloadable from: http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_User_Manual_v5.35.pdf

For RDMA communication, the adapter performs virtual-to-physical memory address translation internally. This translation is defined by a Memory Translation Table (MTT) that is built internally by the WinOF device drivers.

For proper installation & additional details, please follow WinOF user-manual:

https://docs.mellanox.com/display/WINOFv550/Mellanox+WinOF+VPI+Documentation

Regards,

Chen

3

thanks @Chen Hamami​ , I have a question about RoCEv2 on ConnectX-5 Adapters. I’m using vSAN and in 7.0 U3 it supports RDMA / RoCEv2.

Heres my szenario:

  • 2 Nodes each with 1 DualPort ConnectX-5 VPI Socket Direct 100GbE/EDR Cards (2x PCIe x8 Cards)
  • no switch fabric for the 100GbE…
  • Can I directly connect my 2 nodes with Host-Chaining running RoCEv2 on it for vSAN Traffic & vMotion?