The HP blade switch is not a VPI switch and therefore only work with IB. In one of my previous posts I recommended on on using connectx_port_config tool to configure the card to the right mode. There are 3 modes that the VPI NIC card could have:
IB Only - with this mode it can only work with IBM switches
Eth only - with this mode is can only work with Eth switches
Auto sense - with this mode it can work against either (or VPI switch) and will sense the port against.
For working with Eth. you will need an Eth blade. The connectx_port_config tool should set the mode of the card persistently.
Yeah, sounds like you’d better run the cards with IPoIB instead of native Ethernet mode, else the switches won’t talk to them.
For the speed difference, I kind of wonder if the SDP change toggled “connected mode” on.
An easy way to check is by doing an “ip addr” or “ifconfig” to show your ethernet/IPoIB settings.
If the adapters are not in connected mode, their mtu value will generally be around 2044 or similar (with mine anyway). When connected mode is enabled, the mtu shows up at about 65535 (ie 65k).
If that’s what the difference turns out to be (no idea :>), then connected mode can be enabled “properly” by changing a setting in /etc/rdma/ … something. Not remembering off the top of my head, but you should be able to find it pretty easily.