ConnectX-5 Infiniband + MC2210511-LR4 Transceivers = 10GBit/s SDR?!

Hey,

I have a Single Mode LC link which needs to transport 40G Infiniband to an SX6036. That means 40GBase-LR transceivers are my only option and I cannot use 56G MTP SR or 40G SR transceivers.

But now for the issue:

I am using ConnectX-5 cards (Dual Port) with the latest firmware, 16.32.1010 and I just can’t get 40G IB to work.

I have tried different vendors and different programmings (fs.com, Finisar) and even bought genuine Mellanox LR4 transceivers (MC2210511-LR4) out of pure despair… But no chance.

MC2210511-LR4:

Infiniband device ‘mlx5_0’ port 1 status:

default gid: fe80:0000:0000:0000:0c42:a103:006c:0170

base lid: 0xffff

sm lid: 0x0

state: 2: INIT

phys state: 5: LinkUp

rate: 10 Gb/sec (4X SDR)

link_layer: InfiniBand

Infiniband device ‘mlx5_1’ port 1 status:

default gid: fe80:0000:0000:0000:0c42:a103:006c:0171

base lid: 0xffff

sm lid: 0x0

state: 2: INIT

phys state: 5: LinkUp

rate: 10 Gb/sec (4X SDR)

link_layer: InfiniBand

10G in IB Mode, 40G in Eth mode

However, this is not a general limitation.

If I plug in a DAC

Infiniband device ‘mlx5_0’ port 1 status:

default gid: fe80:0000:0000:0000:0c42:a103:006c:0170

base lid: 0xffff

sm lid: 0x0

state: 2: INIT

phys state: 5: LinkUp

rate: 100 Gb/sec (4X EDR)

link_layer: InfiniBand

Infiniband device ‘mlx5_1’ port 1 status:

default gid: fe80:0000:0000:0000:0c42:a103:006c:0171

base lid: 0xffff

sm lid: 0x0

state: 2: INIT

phys state: 5: LinkUp

rate: 100 Gb/sec (4X EDR)

link_layer: InfiniBand

I am seeing the same limitations with some 100G transceivers

Kaiam XQX5170 (100GBase-CWDM4) → No connection at all in IB mode, 100G in Ethernet

Infiniband device ‘mlx5_0’ port 1 status:

default gid: fe80:0000:0000:0000:0c42:a103:006c:0170

base lid: 0xffff

sm lid: 0x0

state: 1: DOWN

phys state: 2: Polling

rate: 10 Gb/sec (4X SDR)

link_layer: InfiniBand

Infiniband device ‘mlx5_1’ port 1 status:

default gid: fe80:0000:0000:0000:0c42:a103:006c:0171

base lid: 0xffff

sm lid: 0x0

state: 1: DOWN

phys state: 2: Polling

rate: 10 Gb/sec (4X SDR)

link_layer: InfiniBand

fs.com 100GBase-CWDM4 → 10G in IB Mode, 100G in Ethernet

Infiniband device ‘mlx5_0’ port 1 status:

default gid: fe80:0000:0000:0000:0c42:a103:006c:0170

base lid: 0xffff

sm lid: 0x0

state: 2: INIT

phys state: 5: LinkUp

rate: 10 Gb/sec (4X SDR)

link_layer: InfiniBand

Infiniband device ‘mlx5_1’ port 1 status:

default gid: fe80:0000:0000:0000:0c42:a103:006c:0171

base lid: 0xffff

sm lid: 0x0

state: 2: INIT

phys state: 5: LinkUp

rate: 10 Gb/sec (4X SDR)

link_layer: InfiniBand

Any ideas?

At least the genuine MC2210511-LR4 transceivers should work?

I have checked and the MC2210511-LR4 does not have the Infiniband Support bit set - Is this intended?

Is there any way to get a 40G-LR4 Link with Infiniband on ConnectX-5?

Also, according to https://www.nvidia.com/en-us/networking/infiniband/optical-transceivers/, the MC2210511-LR4 should even be able to do FDR10…

I will double check on older CX-3 cards.