How to configure Mellanox card to accept breakout cable connection (4x10Gbps)?

1

Hello,

We installed 2 Mellanox NIC on serverA and 2 of them on serverB.

Mellanox NIC info:

Model: MCX354A

PSID: MT_1090110019

Driver: mlx4_en

version: 2.2-1.0.1 (Jun 13 2014)

firmware-version: 2.31.5050

ServerA-B:

-Dell poweredge R720xd

-CPU: 32 CPU each 2.9 GHz (Sandy Bridge): Numa0: 0-2-4-…30,

Numa1:1-3-…-31

-6 slot PCIe 8GT/s (16 lane)

-RAM: 128 GB DDR3

-OS: ubuntu 12.04 LTS (Kernel: 3.11.0.22)

-four Intel SFP+ cards with ixgbe driver

We are going to test breakout cable (40G QSFP+ to 4x10G SFP+), in the following configuration:

-1Mellanox NIC (FCBT) in serverA is connected to 4 intel NIC (SFP+) in server B, using a breakout cable.

But when we connect the breakout cable, just one 10G SFP+ is working from total of four, and 3x10G SFP+ is off (no light). We then generate the traffic, it is normal that just 10G traffic was passed.

I think that this is due to the fact that Mellanox NIC is configured for 40G.

My question is how can i configure Mellanox to recognize 4x10G configuration for breakout scheme?

We tested this issue using Cisco switch configured to 4x10G SFP+ both breakout cable and switch ports were active and working.

Thanks for your support

Mohammad

2

Thanks

3

HI Mohammad,

The split of the port from 40GbE to 4x10GbE is applicable only on Mellanox switch side.

It is not supported on Mellanox 40GbE adapters.

Thanks,

Ophir.

4

Hi,

I added a post discussing the breakout cable options when using Mellanox Hardware

I hope it will help,

Infrastructure & Networking - NVIDIA Developer Forums https://community.mellanox.com/s/article/mellanox-breakout-cables-40g--gt--4x10g-and-100g--gt--4x25g

Ophir.