ConnectX-3 WinOF 5.35 on Win2016 Multiple Partitions

I am in the process setting up an iSCSI storage system for Vmware using IPoIB as high speed interconnect. The switch used is 4036 running 3.9.1. iSCSI cluster has two nodes each has one ConnectX-3 dual port. The second port of the HCA is directly cross connected without a switch. IB SM is running on CentOS box with the following definition:

Default=0xffff , ipoib, mtu=5: ALL=full;

vmotion20=0x8014 , ipoib, mtu=5, defmember=full: ALL=full;

iscsi40=0x8028 , ipoib, mtu=5, defmember=full: ALL=full;

iscsi50=0x8032 , ipoib, mtu=5, defmember=full: ALL=full;

On node A of the storage cluster, “part_man add IPoIB#1 ipoib_8028 8028”. The IPoIB#1 is configured with, and new virtual interface for PKey 0x8028 has

On node B of the storage cluster, “part_man add IPoIB#1 ipoib_8032 8032”. The IPoIB#1 is configured with, and new virtual interface for PKey 0x8032 has

On SM (CentOS), ib0 is configured with, ib0.8028 has, ib0.8032 has

The problem I am having is that connectivity to virtual interface on Windows server is sporadic after system restart/reboot. The ping among are always successful. ibping and ibtracert between all three nodes are always successful. The ping on and are hit and miss. Sometimes, a restart of SM will reestablish the connectivity. When the connectivity is not there, tcpdump and wireshark on the virtual interface showed ARP who-has packet out of the ping originator but never showed up on the other end.

However, if SM is cross connected to the port #1 on the storage cluster node (bypassing 4036), I was not able to reproduce the problem. Of course, by doing this, the SM will see link goes down and then up, and it will pretty much trigger an event similar to SM restart. As indicated above, restart of SM seems to fix the connectivity on virtual interface (PKey partitions) often times.

Any help pointing me to the right direction will be greatly appreciated!


Following up on this topic…

I reverted the firmware back to 2.36.5150 and driver to 5.25.12665, all the issues mentioned above are gone.

I am not sure if this applies, but Linux has a max IPoB MTU of 2048(2044) while Windows has max 4096(4092)

When connecting the 2 different OS you must use the smaller MTU 2044. This normally must be configured manually on the Windows Servers. Failure to set a common MTU in the past yielded poor performance or connectivity issues.

It has been a while since we used both together so things may have changed.

Two nodes are both Windows 2016. SM has partition configured as mtu=5. Windows driver side is also configured with 4096. In fact, on the later versions of the Windows driver, it will throw out warning message in the event log if partition MTU does not match the driver MTU. Through my testing, I know for sure FW 2.36.5150 works perfect with driver 5.25.12665. FW 2.40.7000 does not work with driver 5.25 nor 5.35. I have yet to see if FW 2.36.5150 works with driver 5.35. The server is an older unit based on the Intel Tylersburg chipset.