Hey,
I have a Single Mode LC link which needs to transport 40G Infiniband to an SX6036. That means 40GBase-LR transceivers are my only option and I cannot use 56G MTP SR or 40G SR transceivers.
But now for the issue:
I am using ConnectX-5 cards (Dual Port) with the latest firmware, 16.32.1010 and I just can’t get 40G IB to work.
I have tried different vendors and different programmings (fs.com, Finisar) and even bought genuine Mellanox LR4 transceivers (MC2210511-LR4) out of pure despair… But no chance.
MC2210511-LR4:
Infiniband device ‘mlx5_0’ port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0170
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
Infiniband device ‘mlx5_1’ port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0171
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
10G in IB Mode, 40G in Eth mode
However, this is not a general limitation.
If I plug in a DAC
Infiniband device ‘mlx5_0’ port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0170
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 100 Gb/sec (4X EDR)
link_layer: InfiniBand
Infiniband device ‘mlx5_1’ port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0171
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 100 Gb/sec (4X EDR)
link_layer: InfiniBand
I am seeing the same limitations with some 100G transceivers
Kaiam XQX5170 (100GBase-CWDM4) → No connection at all in IB mode, 100G in Ethernet
Infiniband device ‘mlx5_0’ port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0170
base lid: 0xffff
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
Infiniband device ‘mlx5_1’ port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0171
base lid: 0xffff
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
fs.com 100GBase-CWDM4 → 10G in IB Mode, 100G in Ethernet
Infiniband device ‘mlx5_0’ port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0170
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
Infiniband device ‘mlx5_1’ port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0171
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
Any ideas?
At least the genuine MC2210511-LR4 transceivers should work?