Hello Team,
My question is : Can RDMA be used with lacp on windows servers with Connectx5-6 nics. I’ve checked the RDMA and RoCE terms little bit and i couldn’t able to see any article for link-redundancy ? Do you have any manual or installation guide for link redundancies ? In near future we are planning to build a IP SAN with Mellanox and redundancy is mandatory for us.
Waiting for your feedback.
Have a nice day
Hi sezgink059,
Microsoft recommends using RDMA with Switch Embedded Teaming (SET) and SMB Direct for RDMA link aggregation. SET is a new teaming mechanism that Microsoft would like to replace traditional Windows LBFO teaming. You will need to enable Hyper-V guest support and create a Hyper-V vSwitch for SET, even you only use the team for physical workload link MSSQL on metal.
After SET vSwitch is created, you can create a vNIC with option “-AllowManagementOS” to create a vNIC on host Windows OS. This vNIC is then running on pNIC team, fault tolerated, bandwidth aggregated, and running on low latency RDMA, but it still needs whole brunch of tuning, including pNIC CPU affinity, VMQ/RSS, pNIC driver configuration, SMB server/client setting, etc. The higher the NIC speed the more important those tuning will impact outcome performance. I spent months to figure most of them out and result is a super stable Hyper-V HCI that has run for 4-5 years already without major instability and performance problem. My NIC are are just the old ConnectX-3 Pro only. I believe your ConnectX-5 (or 6?) will give better result. Please avoid mixing pNIC models and speed for team members of a single vSwitch.
SET can be done via PowerShell, Windows Admin Center or System Center SCVMM. One of the requirements for SET is the teaming mode set to be “switch independent” instead of LACP, that means RDMA is not preferred to be used in conjunction with LACP in Microsoft network teaming. Once pNICs are added into a SET teamed vSwitch, you can fine tune the team pNIC to vNIC mapping via PowerShell.
If you are looking for an IP SAN, you can consider Windows native Storage Space Direct (S2D) on Windows Failover Cluster, it has a lot of exciting storage feature like automatic tiering, deduplication, volume encryption, sync/async volume replication etc., it is also a world SDS performance record holder in this specific area. The most important point is: it is free from Windows Server OS license.
You can google for “Microsoft S2D deployment guide” and search for some server hardware vendors’ reference architecture and step-by-step implementation walkthrough.
Cheers.