Setting up 10gb in my Lab - Help

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

ok so I started this post over at hardforum.com but figured … why wouldn’t I put it there???

Ok so I’m new to this and I have a test lab at work.

I have 4 esxi boxes - will get to them later

a windows 10 pc (dell precision 5810)

Our guys were getting rid of a mess of hardware so I grabbed it and figured time to play around.

I stuck a mellanox card in the win10 box and drivers installed, albeit, MS drivers. but it seems to be good?

I also connected a cable to the switch and the light on the switch lit up but in win10 the NIC says cable unattached.

switch - Flextronics F-x430066 8 Port 4x SDR Infiniband

cable - Mellanox Mcc4l30-300 Microgigacn Latch 0.3 M Infiniband Cable

cards - HPE InfiniBand 4X DDR Conn-X PCI-E G2 Dual Port HCA (483514-B21)

so few questions…

  1. what is needed on win10 side other than what I have done?
  2. if I put another card in a secondary pc, and just attach both to the switch, just give static IP’s and that is it?

ill just start with them…thx

So over there I have pics. I cant put pics here so a bit harder…

Ill add it here by copying the text…

in Windows 10 dev manager - shows using Microsoft drivers

ipoib6x.sys - 4/25/2018 ver. 5.50.14643.0

C:>mst status -v

MST devices:

mt26418_pci_cr0 bus:dev.fn=01:00.0

mt26418_pciconf0 bus:dev.fn=01:00.0

C:>mlxfwmanager -d mt26418_pci_cr0 --query

Querying Mellanox devices firmware …

FATAL - Can’t find device id.

FATAL - Can’t find device id.

Device #1:

Device Type: N/A

Part Number: –

Description:

PSID:

PCI Device Name: mt26418_pci_cr0

Port1 MAC: N/A

Port1 GUID: N/A

Port2 MAC: N/A

Port2 GUID: N/A

Versions: Current Available

FW –

Status: Failed to open device

2

It is indeed a rare “oldies components” that you’re trying to put together…

Here is the current status with the components:

Starting with HP adapter “HP IB 4X DDR Conn-X PCI-e G2”

It’s a very old (~6 years) HP-branded “ConnectX” DDR adapter, that Mellanox declared as EOL (end of lite) many years ago. meaning, no HW & support available

In addition, there is no Mellanox WinOF driver for Windows10 that supports the old DDR HP/IB (ConnectX) adapter. The minimum adapter type required is ConnectX-3 (FDR)

Flextronics switch - again, very old IB switch not supported by Mellanox anymore

Bottom line:

I don’t see any option that you can “tailor” or setup a configuration where you have Win10 to work on IB over the Mellanox EOL components, simply because there is no compatible river & FW that support mutual workability over the old components

The same goes with Esxi servers.

3

Interesting I was able to post the original question but then cannot login so had to create a new account to do so…

so much has changed since I last posted this and I have it all listed on hardforum.com

Downloaded WinOF 5.50 and installed

long story short…

  1. found hp firmware for 2.8 and upgraded all of my cards
  2. then windows 10 found them fine
  3. went into card under system devices and changed the drivers there to point to HW driver location that WinOF placed there.
  4. Closed driver and went back in and chose ETH instead of IB for both ports
  5. then under network adapters, changed the drivers there to point to the eth driver location that WinOF placed

Windows 10 now sees the cards, 10gb and I can copy files between them and seems really fast

to do benchmarking, I was told to create a ramdrive on each machine, place files in one and copy through network to the other ramdrive. This would leave out sata, ssd, nvme drive performance differences as ram is extremely fast.

1gb network test for benchmarking numbers - 118.5MB/s

10gb network test for benchmarking numbers - 689 MB/s

so it is working!!!

only issue I am having now is simple… only the top ports can be used. no card will use the bottom ports. I even (expecting bad results) force flashed a firmware onto these cards

fw-25408-2_9_1000-MHGH29-XTC_A1.bin

which took them to 2.9.1000 and it took and cards seem to still be working. Just wish both ports would work.

I get the whole EOL and all… old tech sitting that still functions in a test lab is something to use and not throw away… ramdrive1gb