RoCE not working on Win 2016 (ConnectX-3 Pro)

Infrastructure & Networking Software And Drivers WinOF Driver
, ,
1

Hi all!

I’m trying to configure SMB Direct on windows 2016 with ConnectX-3 Pro EN card (MCX312C-XCCT), but it’s not working. RDMA Activity counters shows constantly rising Responder CQE Errors. So, I treied to buils simplest configuration and now I have the following

Two servers with WIn 2016 w/latest updates. ConnectX-3 Pro w/latest drivers and firmware.

These servers are connected back-to-back by direct optical link (without switch, to eliminate). I’m using one port. Second port attached to switch and used for management.

I’ve used following script for configuration:

#disable everything related to rdma

Remove-NetQosTrafficClass

Remove-NetQosPolicy -PolicyStore ActiveStore -Confirm:$False

Remove-NetQosPolicy -Confirm:$False

Get-NetAdapterQos | Disable-NetAdapterQos

Get-NetAdapterRdma | Set-NetAdapterRdma -Enabled $false

#configure rdma from scratch

Set-NetQosDcbxSetting -Willing 0

Set-NetAdapterRdma -Name “CPU2 SLOT6 PCI-E 3.0 X16” -Enabled $true

New-NetQosPolicy “SMB” -PolicyStore ActiveStore -NetDirectPortMatchCondition 445 -PriorityValue8021Action 4

New-NetQosPolicy “DEFAULT” -PolicyStore Activestore -Default -PriorityValue8021Action 4

New-NetQosPolicy “TCP” -PolicyStore ActiveStore -IPProtocolMatchCondition TCP -PriorityValue8021Action 0

New-NetQosPolicy “UDP” -PolicyStore ActiveStore -IPProtocolMatchCondition UDP -PriorityValue8021Action 0

Enable-NetQosFlowControl -Priority 4

Enable-NetAdapterQos -InterfaceAlias “CPU2 SLOT6 PCI-E 3.0 X16”

New-NetQosTrafficClass -name “SMB class” -priority 4 -bandwidthPercentage 50 -Algorithm ETS

I’m testing RDMA with nd_rping and dumping traffic with ibdump, and that’s what I’ve found.

Client send frame with “CM: ConnectRequest” command to server. Destination MAC Address is correct in that frame, but source MAC addres is 00:00:00:00:00:00.

Server receives this frame and relpies with “CM: ConnectReply”. In this frame both Source and destination MACs are 00:00:00:00:00:00. Client doesn’t receive this frame, so, handshake can’t finish.

This also accompanied by strange frames with “UD Send Only” commands. They are strange because they don’t have Vlan and priority tag and part of them are with Source Address that equals Destination Address and another part - with Destination address 0.0.0.0.

I’m attaching dumps from both client and server.

Could anyone explain, what’s happening?

I also have 4 servers with ConnectX-4 EN adapters, and there are no problems with them - RDMA and SMB Direct works well there.

client.pcap.zip (1.26 KB)

server.pcap.zip (1.47 KB)

2

I’ve downgraded to oldest driver for windows 2016 (v 5.25, the first firmware for windows 2016) and choosed option flash firmware included with these drivers.

RoCE works now!

3

https://community.mellanox.com/s/article/howto-configure-smb-direct--roce--over-pfc-on-windows-2012-server

Note: “Responder CQE Error” counter may raise at an end of SMB direct tear-down session. Details: RDMA receivers need to post receive WQEs to handle incoming messages, if the application does not know how many messages are expected to be received (e.g. by maintaining high level message credits) they may post more receive WQEs than will actually be used. On application tear-down, if the application did not use up all of it’s receive WQEs the device will issue completion with error for these WQEs to indicate HW does not plan to use them, this is done with a clear syndrome indication of “Flushed with error”.

4

Hey Dmitry,

I just wrote a blog post about this issue on my site http://www.checkyourlogs.net/?p=36172 http://www.checkyourlogs.net/?p=36172 today.

Basically without reading too much below → I just pasted for those other readers looking to do some more research. We have in the community confirmed a bug with the 2.40.7000 firmware. I have had several people contact me on twitter today regarding the post and they confirmed the same issue. Basically we rolled back the firmware and in some cases driver and it fixed this.

One of my readers said he setup a Storage Spaces Direct Configuration for a customer. He left it for a month and they upgraded the firmware and it all presented the exact same issues below. They rolled back their firmware and have the same experience as me.

Anyways → Here is a snip from what I wrote up today. I haven’t heard anything back from mellanox Technical Support yet as I have a request in for someone via the Partner Channel to have them contact me regarding this. No answers yet.

Thanks,

Dave

Here is a snip from my post.

Hey Storage Spaces Direct fans I have been working on a deployment for a customer in Washington (Health Care).

We are building out a development environment for them and they have purchased Mellanox NIC’s for both DEV and Production. (Mellanox CX3-Pro MCX-312B-XCCT)

Like any normal, deployment we decided to grab the latest firmware and drivers from Mellanox’s website.

We downloaded the latest Drivers and Firmware and this is what we get for performance on the cards when copying to the CSV Volume in the Cluster.

10 MB/Sec are you kidding me… (We tried everything before looking at this firmware: Changed the Avago Controller “LSI 9300”, Updated Firmware on our drives, tore apart the Failover Cluster, tried every conceivable configuration change in Storage Spaces Direct. Finally, I noticed the Firmware version in my lab and the customer’s was different. So I upgraded mine and this is what happened.

958×608 27.1 KB

For all you Networking junkies out there… I can repo this on demand every time I upgrade the firmware to 2.40.7000

I am able to somewhat fix the issue by reverting the firmware, reinstalling the old driver, and tearing down the entire Virtual networking stack.

Here are the detailed steps I followed to fix this:

  1. Revert the Firmware on the Card to: 2.40.5030 (fw-ConnectXPro-rel-2_40_5030-MCX3112B-XCC_Ax-FlexBoot-3.4.746.bin)

  2. Uninstall the Mellanox Driver (This will break your networking Setup but that is fine)

  3. Reinstall the Mellanox Driver with the older version: 5.35.51100.0 (MLNX_VPI_WinOF-5_35_All_win2016_x64)

  4. Restart the Server

  5. Validate that the Firmware and Driver are updated in Device MGR

  6. Rebuilt the SET Team / VMSwitch / and Virtual Adapters / Re-Enable RDMA

#Remove Virtual Adapters

Get-VMNetworkAdapter -ManagementOS SMB_3 | Remove-VMNetworkAdapter

Get-VMNetworkAdapter -ManagementOS SMB_4 | Remove-VMNetworkAdapter

Get-VMSwitch -Name teamedvswitch02 | Remove-VMSwitch -confirm:$false

New-VMSwitch -Name teamedvswitch02 -NetAdapterName “Ethernet 3”, “Ethernet 4” -EnableEmbeddedTeaming $true -Confirm:$false

Add-VMNetworkAdapter -SwitchName teamedvswitch02 -name SMB_3 -ManagementOS

Add-VMNetworkAdapter -SwitchName teamedvswitch02 -name SMB_4 -ManagementOS

Enable-NetAdapterRdma “vethernet (SMB_3)”,“vethernet (SMB_4)”

Get-NetAdapterRdma

  1. Reconfigure the IP Addresses on the SMB_3 and SMB_4 virtual adapters

This is still slow as it is about 1/2 of the speed I used to get in my lab prior to touching the drivers and firmware. I was getting 500-700 MB /sec before I touched the drivers and firmware.

I also tested this configuration upgrading to the latest Mellanox Driver – MLNX_VPI_WinOF-5.35_All_Win2016_x64 (5.35.12978.0 “Device MGR” 5.35.52000.0 “on the Driver Package”. My results were the same à Better than the 10 MB/sec but not faster than 200 MB /sec yet.

I did some digging on the Firmware and this is what I have been able to come up with so far:

There only appear to be two fixes in this latest firmware:

RM#980151: Fixed an issue where a virtual MAC address which is configured by set_port (ifconfig), remained after driver restart

RM#913926: Fixed an issue where the two interfaces reported the same MAC address when bonding configuration was used.

http://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_40_7000-release_notes.pdf http://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_40_7000-release_notes.pdf

I don’t think it will be that big of a deal NOT installing this Firmware and going with the latest drivers. At least until Mellanox comes back to the community with a fix and or workaround.

There was something rather disturbing that I read in the release notes for this flakey firmware**. It appears to me that no testing was done on this firmware one Windows Server 2016.** I may to totally off base here and I am just going off the release notes published on the www.mellanox.com http://www.mellanox.com/ website for this firmware.

5

Hi, Dave!

Thanks for confirming, I’m not alone with this problem.

I’d also like to say, we have faced with this problem while running previous version of firmware (2.40.5032) and drivers (5.35.12970.0). I’ve saw, that 2.40.7000 fixes two [irrelevant] bugs, but updated this version with hope it will suddenly fix RoCE.

Actually, I’m not building S2D, but trying to connect Hyper-V hosts to SOFS via RDMA, and see, that RDMA is not working at all (nd_rping, nd_send_bw can’t establish RoCE connection).

Btw, even without using RDMA we have problems with these servers - virtual machines are constantly lose connection to storage (which is on the SOFS). I think now, in may be caused by firmware too.

I’ll try to downgrade to older fw/drivers…

6

I have another important addition.

Everything was working just after driver&firmware rollback/installation.

After that I’ve added VmSwitch (New-VMSwitch -Name “Embedded Cluster Switch” -NetAdapterName “Port2” -EnableEmbeddedTeaming $true), which was bound just to one adaper (Port 2 on physical nic) (I remind, that second adapter (Port 1 on physical nic) is used for back-to-back connection between servers and I’m using if for testing RDMA) RoCE suddenly stopped working.

After several reconfigurations I finally had seen, what everything breaks after I add VmSwitch on the server, which acts as a client in nd_rping, because source MAC address became 00:00:00:00:00:00 after adding VmSwitch.

Btw, driver release notes lists something similar to this problem in known issues: 909040 Description: When configuring ConnectX-3 Pro single port via the driver, the RDMA connection fails. WA: N/A Keywords: ConnectX-3 Pro, single port, RDMA.

But I can’t understand, what is “configuring ConnectX-3 Pro single port via the driver”…

7

Firmware version 2.42.500 is now out.

I can’t see any notes regarding this issue in the release notes, either in the fixes or known bugs…

Anyone been able to test?

8

Just an update for anyone interested.

On my Hyper-V 2016 install, I’ve updated the drivers to the latest version (5.40) along with the firmware version 2.42.500 and now RoCE appears to be working.

I haven’t run any in depth testing as yet but I am able to Live Migrate VM’s with 15GB memory in 6 seconds and it’s no longer traversing the management vNIC’s (1Gb). Additionally, I don’t see any bandwidth utilization via task manager which indicates it’s working correctly.

Happy days!