Windows Server 2016 configure RDMA for S2D

Hi all, i have configured cluster S2D , two nodes with rdma adapters (HP 546SFP+ ) connected to Switch Dell PowerConnect 8024 with lastest firmware.

Cluster have two networks : 1 routable for VMs, 2 non routable for Storage RDMA

I use this PS code for configure RDMA on nodes…

Install-WindowsFeature Data-Center-Bridging Set-NetQosDcbxSetting -Willing $False Enable-NetAdapterRdma –Name RDMA-NIC1 Remove-NetQosTrafficClass -confirm:$False Remove-NetQosPolicy -confirm:$False New-NetQosPolicy "SMBDIRECT" -netDirectPortMatchCondition 445 -PriorityValue8021Action 4 New-NetQosPolicy "DEFAULT" -default -PriorityValue8021Action 1 Enable-NetQosFlowControl -Priority 4 Disable-NetQosFlowControl 0,1,2,3,5,6,7 Enable-NetAdapterQos -InterfaceAlias "RDMA-NIC1" New-NetQoSTrafficClass "SMBDirect" -Priority 4 -Bandwidth 90 -Algorithm ETS

Flow control disabled on NIC

GlobalPause disabled on NIC

VlanID set to 666

Switch config:

Jumbo Frames enabled

flowcontrol globally disabled(global pause)

Cut-through mode is configured as enabled

Ports config:

interface Te1/0/20 datacenter-bridging priority-flow-control mode on priority-flow-control priority 4 no-drop exit spanning-tree portfast mtu 9216 switchport mode trunk exit ``interface Te1/0/10 datacenter-bridging priority-flow-control mode on priority-flow-control priority 4 no-drop exit spanning-tree portfast mtu 9216 switchport mode trunk exit

After configuration i see RDMA activity counters greater than zero… RDMA works, but i see Mellanox Diagnostics counters have many errors… screen shot attached

nd_send_bw test results:

C:\Program Files\Mellanox\MLNX_VPI\IB\Tools>nd_send_bw -C 10.100.100.2

#qp #bytes #iterations MR [Mmps] Gb/s CPU Util.

0 65536 100000 0.017 9.13 100.00

Test finished. Releasing resources…

Can i fix errors or this is correct RDMA work? Can verify RDMA works correctly?

thx for help…

mlnx work.png

Hello Dmitry,

Please use this document SDN/WS2016_ConvergedNIC_Configuration.docx at master · Microsoft/SDN · GitHub https://github.com/Microsoft/SDN/blob/master/Diagnostics/WS2016_ConvergedNIC_Configuration.docx

in order to configure Mellanox adapter in S2D cluster with Dell switches. Dell switch configuration provided in above document on page 15.

For additional information about S2D please look this page Storage Spaces Direct Hardware Requirements | Microsoft Docs. https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-hardware-requirements

B.R.

Vitaliy