Thanks for the help.
I will document more detail of my newbie journey later iand explicitly answer the questions in a more concise manner, but here is a brain dump so far.
Yes I had to use an earlier version of OFED to get a Windows 7 driver that would see both my HP (rebranded Mellanox MHGH28-XTC) and my Voltaire 400 because the latest V3.2 OFED driver worked with the Mellanox but not the Voltaire. Thanks for the tip on this.
But I still could not flash the firmware in the Voltaire using the latest version of Mellanox Firmware Tools (MFT) so i had to use an earlier version of that also. I could not find the earlier versions of MFT on the normal Mellanox web site, but I found them on the “Old.Mellanox” web site that has several earlier versions of MFT. Again, thanks for the tips.
In the end, I was not happy using old versions of OFED and MFT so I bought another Mellanox MHGH28-XTC for £29 and will sell the Voltaire. All my HP Mellanox cards are now flashed to 2.7 firmware (the HP version not vanilla Mellanox).
Next was getting the two MHGH28-XTC connected together with an Infiniband CX4 to CX4 cable I bought off e-bay from China for $20. I got ibping to work after I finally figured I had to run “ibping as server” on one machine then run ordinary ibping on the second. Then created a Windows share on one machine and tried to access it from the second using ethernet over Infiniband IPoIB. I gave each Infiniband card a fixed IP address 172.16.0.x and I disconnected both computers from my normal Gigabit network and sucessfully accessed the share on the other machine accross the Infiniband cable. SSD to SSD transfer was 220MB/sec which is about the max write rate of the SSD’s I was using. Gigabit gave 110MB/sec.
Next I discovered how to tell Windows 7 to prioritise the Infiniband network over the Gigabit network so I turned both networks back on. Not sure if Windows 7 would prioritise the faster Infiniband network over the slower Gigabit nextwork by default and so I configured Infiniband IPoIB higher just in case.
My next challenges are:-
Somehow connect the infiniband “network” (2 x PC connected with an infiniband cable), to my normal home gigabit network. Normal is on 192.168.0.x whilst infiniband is on 172.16.0.x. Why wont Win-7 allow me to set the Infiniband IPoIB as a “home” network ? Then I could just click on “merge networks” , right ? IPoIB shows as “public” network at present.
next, get better speed by using faster SSD and/or raid0, and trying Connected Mode which is supposed to be faster. (is it ?)
Get the 9024S Infiniband switch (£165 )that arrived today working with the new Infiniband set-up.
Get a newer MHGH29-XTC (still only £90 on e-bay) because the info says they can have one port doing 10GBE ethernet whilst the second port does Infinband. Then I can sell the two Intel X540-T 10GBE and replace them with duel purpose MHGH29-XTC’s at a third of the price.
Get it all working with Ubuntu as well as Windows.
Get it working as a SCSI setup target/initiator mode a la fibrechannel. (not i-SCSI) My fibrechannel cards are second hand 4GB ones so maybe I could get rid of these now as well.
Note. I have a 10GBE switch that has 7 ehthernet ports and one QSFP (SFP ? cant remember) port so I’d like to somehow connect this to the Infiniband network with a CX4 to SFP cable, is this possible even ?
Alternatively, which Infiniband switches have a “bridge” module built in ? and can I plug a bridge module into my second hand 9024 which seems to have a space in the middle to plug extra modules into it ?
Thanks again for the help so far. My original question is answerd. But is anyone else further up the “Infiniband for home use” learning curve that can say what hardware to seek on e-bay now ? then I can figure out the software setup later on . . .
e.g. any recommendations for a NAS software BSD or Linxux disty that has ZFS and is easy for Infiniband set up ? I am on FreeNas at present but I read Solaris variants are better for Infiniband, but wont work with infiniband cards that dont have memory built onto them (memfree cards like mine).