Is Firmware for Voltaire EX400 Available from Mellanox ?

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

Hello everyone.

I bought 2 of Mellanox MHGH28 XTC (branded Hewlett Packard) and two vanilla Voltaire 400EX infiniband HCA from e-bay for home use. All had the “cannot start, error 10” problem. I successfully flashed the HP version of v2.7 firmware to both of the MHGH28 and now Windows 7 64 bit version sees the cards and they work fine. Thanks for the thread that helped me do this. Still got the “error 10” problem with the Voltaire cards though.

The command “mst status” shows the Voltaire as MT25208, but if I query mt25208_pciconf0 with flint, it says:- “device type 25208 not supported any more: old device type”. The card is in the same PCIe x8 slot that worked with the MHGH28. Do I bin the Voltaires and buy more MHGH28 ? or can I communicate with them and get firmware from Mellanox since they bought Voltaire ? I note there is firmware starting with fw-25208_etc_etc on the Mellanox firmware download website.

Lastly, does the number “25208” just refer to the firmware that is loaded ? or does it identify the hardware / infiniband chip ?

Thanks in advance for any help.

Shaun

2

what’s the PSID number? (run ibv_devinfo and look for device ID)

3

Hello all.

Thanks for the help.

I will document more detail of my newbie journey later iand explicitly answer the questions in a more concise manner, but here is a brain dump so far.

Yes I had to use an earlier version of OFED to get a Windows 7 driver that would see both my HP (rebranded Mellanox MHGH28-XTC) and my Voltaire 400 because the latest V3.2 OFED driver worked with the Mellanox but not the Voltaire. Thanks for the tip on this.

But I still could not flash the firmware in the Voltaire using the latest version of Mellanox Firmware Tools (MFT) so i had to use an earlier version of that also. I could not find the earlier versions of MFT on the normal Mellanox web site, but I found them on the “Old.Mellanox” web site that has several earlier versions of MFT. Again, thanks for the tips.

In the end, I was not happy using old versions of OFED and MFT so I bought another Mellanox MHGH28-XTC for £29 and will sell the Voltaire. All my HP Mellanox cards are now flashed to 2.7 firmware (the HP version not vanilla Mellanox).

Next was getting the two MHGH28-XTC connected together with an Infiniband CX4 to CX4 cable I bought off e-bay from China for $20. I got ibping to work after I finally figured I had to run “ibping as server” on one machine then run ordinary ibping on the second. Then created a Windows share on one machine and tried to access it from the second using ethernet over Infiniband IPoIB. I gave each Infiniband card a fixed IP address 172.16.0.x and I disconnected both computers from my normal Gigabit network and sucessfully accessed the share on the other machine accross the Infiniband cable. SSD to SSD transfer was 220MB/sec which is about the max write rate of the SSD’s I was using. Gigabit gave 110MB/sec.

Next I discovered how to tell Windows 7 to prioritise the Infiniband network over the Gigabit network so I turned both networks back on. Not sure if Windows 7 would prioritise the faster Infiniband network over the slower Gigabit nextwork by default and so I configured Infiniband IPoIB higher just in case.

My next challenges are:-

Somehow connect the infiniband “network” (2 x PC connected with an infiniband cable), to my normal home gigabit network. Normal is on 192.168.0.x whilst infiniband is on 172.16.0.x. Why wont Win-7 allow me to set the Infiniband IPoIB as a “home” network ? Then I could just click on “merge networks” , right ? IPoIB shows as “public” network at present.

next, get better speed by using faster SSD and/or raid0, and trying Connected Mode which is supposed to be faster. (is it ?)

Get the 9024S Infiniband switch (£165 )that arrived today working with the new Infiniband set-up.

Get a newer MHGH29-XTC (still only £90 on e-bay) because the info says they can have one port doing 10GBE ethernet whilst the second port does Infinband. Then I can sell the two Intel X540-T 10GBE and replace them with duel purpose MHGH29-XTC’s at a third of the price.

Get it all working with Ubuntu as well as Windows.

Get it working as a SCSI setup target/initiator mode a la fibrechannel. (not i-SCSI) My fibrechannel cards are second hand 4GB ones so maybe I could get rid of these now as well.

Note. I have a 10GBE switch that has 7 ehthernet ports and one QSFP (SFP ? cant remember) port so I’d like to somehow connect this to the Infiniband network with a CX4 to SFP cable, is this possible even ?

Alternatively, which Infiniband switches have a “bridge” module built in ? and can I plug a bridge module into my second hand 9024 which seems to have a space in the middle to plug extra modules into it ?

Thanks again for the help so far. My original question is answerd. But is anyone else further up the “Infiniband for home use” learning curve that can say what hardware to seek on e-bay now ? then I can figure out the software setup later on . . .

e.g. any recommendations for a NAS software BSD or Linxux disty that has ZFS and is easy for Infiniband set up ? I am on FreeNas at present but I read Solaris variants are better for Infiniband, but wont work with infiniband cards that dont have memory built onto them (memfree cards like mine).

regards

Shaun

4

shaun Infrastructure & Networking - NVIDIA Developer Forums - If ibv_devinfo doesn’t show the PSID for some reason, another option is to use an older release of the Mellanox Firmware Tools. Something before the 3.x series.

Version 2.7.2 is still available on the old Mellanox website here:

http://old.mellanox.com/content/pages.php?pg=management_tools&menu_section=34 http://old.mellanox.com/content/pages.php?pg=management_tools&menu_section=34

5

Some things for you to know:

-1. You can flash your voltaire with the mellanox firmware, I can’t remember the switch, but if you pull up the flint manual it will tell you. I’ve done that with IBM adapters to make them into mellanox adapters. Something like “ignore PSID”.

  1. I’ve never used Infiniband on windows 7, but IPoIB is non-bridgeable, so that’s probably the issue you’re having on windows7. The only way around that is possibly doing a NAT connect using RRAS, but i’ve not gotten that to work either. (but i’m still a relative newbie to infiniband)

  2. MHGH29-XTC I’ve not heard of using one port and 10Gbe and the other infiniband. But If that’s possible, you’ll need some sort of adapter to go from CX4-SFP

  3. to use SRP, you would need to use the Openfabrics drivers, or a very old copy of the mellanox drivers and I don’t know if they even support SRP. (version 3.0)

  4. It’s not possible to plug a 10Gbe switch into your infiniband switch without some form of bridge, like a BridgeX.

  5. I don’t know about switches with modules, if such a switch exists, it would be a datacenter product with plug-in modules.

  6. Your 9024S does not have a module space in the middle, it’s the access panel to swap out the fan.

  7. As far as Infiniband targets, I uses ESOS which is linux-based. Why would you need zfs? you’re exporting the drives as targets to the initiators and then they format them with nfts. At least with ESOS, you have the option to use a virtual file and export that as a drive, but doing that is slower than just exporting the drive and using blockio.

6

in ibv_devinfo look for “board_id”

any luck with the newer MFT package?