HPE InfiniBand 4X DDR Conn-X only uses top port

Infrastructure & Networking Adapters and Cables
1

2_9_1000fw
2_9_1000fw457×603 17.3 KB
disclaimer: I know that these cards are EOL… but still…what I’m asking is simple… when they were not EOL, they worked and I’m just trying to see why I am seeing this and if there is a work around.

for my home lab, I have cards

HPE InfiniBand 4X DDR Conn-X PCI-E G2 Dual Port HCA (483514-B21)

and have them in windows 10, setup as Ethernet and they work.

have them in esxi 6.7 and it works

have them setup in FreeNAS 11 and it works

my dilemma I just cant figure out…

I can only use the top port. if I use any of the bottom of the 2 ports on the card, I get no link lights.

originally these cards had HP firmware ranging from 2.5 to 2.7. I flashed with HP 2.8 on each card and then I was able to get them to work in each OS listed above. Having this issue where I can’t use the bottom port, I even tried force flashing with Mellanox fw 2.9.1000 and it took and they work!!!

still id love to have 2 cards in my machine with 4 ports for my home lab vs needing 4 cards to use just the top ports…

esxi, freenas and windows all show both ports… all say they are there and just down…

2

Hi Jeffrey,

I see the following from your description:

“Having this issue where I can’t use the bottom port, I even tried force flashing with Mellanox fw 2.9.1000 and it took and they work!!!”

I would like to confirm with you, once you tried with Mellanox FW, you were able to see link up on the bottom port as well. Is this correct?

Also, from the screenshot provided, Down and Disabled usually points to cable issue. What is the cable you are using? Did you try swapping the cable from the top port to bottom port? Also, to what are these cards connected? Is it back to back with HCA or there is a switch in between?

Thanks,

Namrata.

3

​unfortunately no…

that pic was just to show a 2.9.1000 firmware and also that it shows 2 ports. I can connect cables and never goes to up. I just cant figure out why.

4

Hi Jeffrey,

It would be great if you could let me know the following:

What is the cable you are using? Did you try swapping the cable from the top port to bottom port? Also, to what are these cards connected? Is it back to back with HCA or there is a switch in between?

Thanks,

Namrata.

5

What is the cable you are using?

cable I’m using…

IBN5800-3 Gore Infiniband 3M Cable

Did you try swapping the cable from the top port to bottom port?

Yes…

I took cable to both ports on one card and cant get a link light.

I took cable and went top port to top port on 2 machines and works and can ping away and access files without issue

I took cable and went top port on one and bottom port on another and no link light

I took cable and went bottom port on both and cant get a link light

it just seem that I cant get the bottom ports to work even though windows/esxi/freenas all see both ports but link never goes up… no matter what I try…

Also, to what are these cards connected? Is it back to back with HCA or there is a switch in between?

I was doing card to card aka pier to pier. I even put 4 cards in one machine and had 4 machines connected all through just top ports. tried bottoms in every configuration and never got anywhere. couldn’t get link lights…

I went out and bought an HP Procurve 6600cl off of ebay and now I’m going from top ports of nic to the switch and have all of that working.

I did not try a bottom port of any of the cards to the switch - but I don’t have my hopes up…

these cards are old… ConnectX-1 cards… I get that they are old… but work… and for the price and just a home user… works great so been learning how to flash firmware, how to set them up as Ethernet instead of infiniband… and it has been fun.

now these are HP branded cards to which HP only has firmware up to 2.8 and I found the model these translate too as MHGH29-XTC cards. I then found on Mellanox site, 2.9.1000 firmware and force flashed a few cards as they took and are working. still only top ports… its so weird…

now I put this post here to see if anyone has seen this issue before but its not like its killing me.

my next step is trying to find out how or if even possible to make a custom firmware for these cards to 2.9.1200 or higher to support rdma in windows for better speed performance… ive had no luck but have seen someone post on the forums that he would give a link to 2.9.1200 for these cards but he didn’t provide the link and I have messaged him and have not received a message. he did post in December so I know he is kinda active. I’m just being patient waiting for him…

6

Hi Jeffrey,

As you mentioned, these are EOL adapters, we do not support these anymore. Also, the cards are OEM branded and we, the Mellanox team follow hierarchy, wherein when OEM cards are involved, we re-direct to OEM because the firmware for these cards are customized by OEM and Mellanox is unaware of these customizations. It would be great if you could reach out to them as well.

Since these are very old cards and OEM based, I was unable to get the Release Notes for this firmware version to check on any known issues. However, I was able to get the RN for version 2.9.1000 which is for Mellanox branded card and I came across a known issue as follows:

  1. Autosensing after cable reconnection to different port protocol

Description:

If you disconnect a cable from an IB/Eth port and reconnect it to an Eth/IB port (different protocol), the link may not rise

Work around:

Restart the driver (openibd restart)

Since you mentioned you were experimenting on setting to Eth instead of Infiniband,may be you are running into this issue. It would be great if you could try this workaround.