Not getting connection between 4036E switch and MT26428 adapter

Hi all,

Infiniband n00b here, trying to learn with some spare equipment I’ve inherited at work…

I have a Voltaire 4036E switch, which I have connected up to a Linux server that has a Mellanox MT26428 dual-port adapter via a 5m DAC cable. I factory-reset the switch, and have done very minimal config so far (pretty much just set the hostname.) I do not see any lights on the switchport or adapter, and the adapter status is DOWN. I have checked that the cable is seated properly on both ends.

I can see the adapter showing up in the server (in lspci output) :

0c:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)

The cable connection is into port 1 –

root@dockerhost01:~# ibstatus Infiniband device 'mlx4_0' port 1 status: default gid: fe80:0000:0000:0000:0002:c903:0010:bf51 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X) link_layer: InfiniBand Infiniband device 'mlx4_0' port 2 status: default gid: fe80:0000:0000:0000:0002:c903:0010:bf52 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X) link_layer: InfiniBand

On the switch side, I have the cable into port 18:

Test-IB-sw# cable-config show Port 1: Not present Port 2: Not present Port 3: Not present Port 4: Not present Port 5: Not present Port 6: Not present Port 7: Not present Port 8: Not present Port 9: Not present Port 10: Not present Port 11: Not present Port 12: Not present Port 13: Not present Port 14: Not present Port 15: Not present Port 16: Not present Port 17: Not present Port 18:Length 5m Vendor Name: Mellanox Code: QSFP+ Vendor PN: MCC4Q26C-005 Vendor Rev: B0 Vendor SN: AC501078867 Port 19: Not present Port 20: Not present Port 21: Not present Port 22: Not present Port 23: Not present Port 24: Not present Port 25: Not present Port 26: Not present Port 27: Not present Port 28: Not present Port 29: Not present Port 30: Not present Port 31: Not present Port 32: Not present Port 33: Not present Port 34: Not present Port eth1: Not present Port eth2: Not present

And this is what I see when I get the port status:

Test-IB-sw(utilities)# ibportstate 1 18 PortInfo: # Port info: Lid 1 port 18 LinkState:.......................Down PhysLinkState:...................PortConfigurationTraining LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedActive:.................undefined (7)

Do I have incompatible equipment connected? or if not, how best to continue troubleshooting?

Thanks,

Will

Hello Will

Do you have a subnet Manager running on the switch or the host? There needs to be at least one SM in the fabric to bring links active.

-Steve

Thanks, Steve, for your reply. Yes, the SM is running on the Voltaire

switch.

In further testing, I downed the connected server, swapped the IB card with

another one, and voila - the link came up! So, bad card. I just don’t know

how I could have figured that out. In any case, problem resolved.

Will