IS5030 Speed Issues: Only DDR?

Infrastructure & Networking Switches and Gateways InfiniBand/VPI Switch Systems
1

Hi,

I’m having speed issues that I’ve isolated to my IS5030 switch.

System: I have two Infiniband switches: 1. A Mellanox IS50XX with 36 ports enabled and the FrabricIT internal subnet manager running, making it an IS5030, with latest firmware (IBM P/N: 98Y3756) 2. A Sun 36 port QDR Infiniband Switch, internal subnet manager. I also have 3 types of HCAs: Sun 375-3696 X4242A (rebranded Mellanox ConnectX-2 card), HP 544FLR-QDR (based on Mellanox ConnectX-3), and Intel/QLogic QLE7340. I haven’t figured out how to update the firmware for the Sun products. The HP HCAs have the latest firmware. The QLE7340’s don’t use firmware. Running CentOS 7 on all nodes.

I’ve tried the following tests:

  • Two HP HCAs, back-to-back, opensm: 40 Gb/s
  • Sun HCA, HP HCA, back-to-back, opensm: 40 Gb/s
  • Two QLE7340, back-to-back, opensm: 40 Gb/s
  • Sun and HP HCAs, Sun switch: 40 Gb/s
  • Sun and HP HCAs, IS5030 switch: 20 Gb/s
  • Two QLE7340, Sun switch: 10 Gb/s
  • Two QLE7340, IS5030 switch: 20 Gb/s

I had a long discussion with an Intel rep, and it turns out that these two Infiniband switches are not compatible with the QLE7340’s (some of the later Mellanox X series switches are, though), which is probably why they aren’t reaching QDR speeds. So let’s ignore those for now.

The really weird thing is that the Sun and HP HCAs should really be negotiating to 40 Gb/s with the IS5030, and I have no idea why they are not. Clearly the HCAs, software, and cables (I tested all of these) are fine since they work at 40 Gb/s back-to-back.

Any ideas?

2

I posted on servethehome, too, and learned some stuff from those guys.

I’ve done a lot of port configuration attempts with the webui and CLI. It says that the ports are capable of “2.5, 5 or 10Gbps”, and 10Gbps 4x is QDR, but then none of the ports negotiate to more than 5 4x = 20 Gbps. I attempted to force the port speed (both the switch and the hca port) using ibportstate on the node, but trying to force the switch port resulted in an error and trying to force the hca then forcing a renegotiate (pull cable, reinsert cable) caused the link to never come up again until after a reboot.

It turns out that there are DDR only versions of the IS50XX switches. https://cw.infinibandta.org/files/showcase_product/101213.173410.167.IS5030_35.pdf https://cw.infinibandta.org/files/showcase_product/101213.173410.167.IS5030_35.pdf I didn’t think there were because ALL of the IS50XX switches are advertised as QDR, including the one I bought. Mine is an IBM rebrand, so unfortunately doesn’t have the complete mellanox part number anywhere, but if yours has a “D” in the complete part number, then it’s a DDR switch. “Q” means QDR. I’m not sure if this applies to me or not yet.