Infiniband HBA ports show as "cable unplugged" in Windows 2016?

Infrastructure & Networking Adapters And Cables Ethernet Adapter Cards
#1

I’ve posted this topic on other forums on the internet with mediocre results, so hopefully coming straight to the source will help me resolve this:

I am setting up a new IB storage network for my home lab and I need help establishing a connection with my Windows Server 2016 box.

The hardware is:

HP DL180 G6 server with Windows Server 2016 x64

HP Infiniband 4X DDR Connect-X PCI-e Dual Port HBA (Part number is 448397-B21 / 452372-001)

Infiniband SFF-8740 cable

Topspin 120 24-port IB switch

I have upgraded the firmware on the Topspin switch to 2.9.0/build 170 and have installed the Mellanox WinOF VPI (v5.35) on my server. I have set the MTU on both ports of the HBA to 2048 to match the switch settings.

The card detects as a Mellanox ConnectX interface card in the OS, but the network adapter appears as “cable unplugged” for both ports. I have used WinOF to configure the ports as IPoIB and as Ethernet just to see if one works over the other, but both are down. Cabling the card do the same card in another Windows box and setting both to Ethernet mode brings up the connection immediately, so the issue is definitely Infiniband related.

mst status shows:

mt25418_pci_cr0

mt25418_pciconf0

I have flashed the cards from HP to Mellanox firmware 2.9.1 using the following command:

flint -d mt25418_pciconf0 -i fw-25408-2_9_1000-MHGH28-XTC_A2-A3.bin -nofs -allow_psid_change burn

There is a link light on the switch for both connections as well as a link light on the HBA on both ports. The ports on the switch show Up/Up. The ports are set to auto-negotiate and show up at 4x-SDR(10Gbps). When I connect the adapters to the Topspin 120 switch and put them into IPoIB mode, get get a good green light on the adapter and switch, and the amber light is lit on each adapter, indicating a good connection.

In addition, the switch itself shows the two servers as up and active nodes in the IB section of the management console, and the IB HBAs show up as active ports and functional neighbors. So the subnet manager on the switch is functioning as it should, I guess.

So everything seems good and all, but the HBAs still show as Network Cable http://skimlinks.pgpartner.com/mrdr.php?url=http%3A%2F%2Fskimlinks.pgpartner.com%2Fsearch.php%2Fform_keyword%3Dnetwork%2Bcable unplugged within Windows and I am not able to send any traffic through them.

What have I missed? How come the adapters show “cable unplugged” in the OS?

#2

Try WinOF 4.80. The latest 5.35 only support connectx-3

#3

Will start with indicating the fact that Mellanox WinOF VPI (v5.35) provides modules that are containing IB & ETH drivers supporting ConnectX-3 & ConnectX-3 pro adapters only, and over FW versions 2.40.5032, so inspire upgrading switch fw & getting “green line” at the switch ports it seems to me that you may have an issue with SM (subnet manager) on the switch or incompatibility between the operating-system tack and SM. Suggesting that you liaise with Cisco to check if they can provide a compatibility matrix that confirms workability of all components above (Mellanox Connect-X PCI-e Dual Port adapter and the proper fw, IPoIB, switch fw etc…)

#4

Thanks for this.

Can I get a clarification that the Mellanox Connect-X PCI-e Dual Port adapter is supported under Windows Server 2016 in Infiniband mode?

#5

I tried it, to no avail.

It is making me crazy that the switch is showing the ports connected, and both the HBA and the switch port are lit. But the damn OS doesn’t want to turn them on.

http://bits.rainwalk.net/InfinibandAdapterStatus.PNG

http://bits.rainwalk.net/InfinibandNeighbors.PNG

http://bits.rainwalk.net/InfinibandNodes.PNG

http://bits.rainwalk.net/InfinibandPortProtocol.PNG

#6

Try to start a local subnet manager on one of the nodes

OpenSM - Subnet Manager OpenSM v3.3.11 is an InfiniBand Subnet Manager. In order to operate one host machine or more in the InfiniBand cluster., at least one Subnet Manger is required in the fabric.

OpenSM can run as a Windows service and can be started manually from the following directory: <installation_directory>\tools. OpenSM as a service will use the first active port, unless it receives a specific GUID.

To run OpenSM manually:

C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe

For additional run options, enter: “opensm.exe -h”

OpenSM can be registered as a service from the Command Line Interface (CLI)

 To register it as a service execute the OpenSM service:

sc create OpenSM binPath= “c:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe -service” start= auto

 To start OpenSM as a service:

sc start OpenSM

The following are commands used from the PowerShell:

 To register it as a service execute the OpenSM service:

New-Service –Name “OpenSM” –BinaryPathName “"C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe” --service -L 128" -DisplayName “OpenSM” –Description “OpenSM for IB subnet” -StartupType Automatic

 To start OpenSM as a service run:

Start-Service OpenSM

#7

Deleting the SM within the Topspin 120 and launching OpenSM on one of the nodes also didn’t work. IB just wasn’t an option on these cards with the drivers/firmware options.