56 GbE over a ConnectX-3 VPI adapter

,
#1

Hello,

I have a pair of ConnectX-3 VPI adapters (MCX354A-FBCT) in Ethernet mode and am attempting to test out 56 GbE connectivity by directly connecting the adapters to one another via an FDR cable (MC2207128). However, the cards only establish link at 40 GbE. In fact, ethtool at least does not seem to report that the card is 56 GbE-capable. I am using firmware 2.31.5050 and driver 2.2-1.0.0. For example, ethtool shows the following:

Supported link modes: 1000baseT/Full 1000baseKX/Full 10000baseT/Full 10000baseKX4/Full 10000baseKR/Full 40000baseKR4/Full 40000baseCR4/Full 40000baseSR4/Full 40000baseLR4/Full

Looking at the some of the ethtool source (drivers/net/ethernet/mellanox/mlx4/en_ethtool.c), I do not even see the #defines for reporting 56000 links. ibv_devinfo similarly reports the speed at 40 GBE.

Does something else need to be done to enable 56 GbE support on these cards?

Regards,

Thomas

#2

Hi Thomas,

In order to link at 56GbE in a back-2-back setup, you need to change the HCA ini file and re-burn the firmware.

  1. Download the .mlx file (fw code file) and ini file ( configuration file )from:

Creating a Customized Firmware Binary - Mellanox Technologies http://www.mellanox.com/page/custom_firmware_table → Firmware DISCLAIMER http://www.mellanox.com/downloads/firmware/disclaimer.php?download=/downloads/firmware/ConnectX3-rel-2_31_5050.tgz

  1. Extract the files

  2. Find the ini file that belongs to your HCA,

a. ibv_devinfo | grep board_id

b. grep <board_id> *

for example:

tar xzf ConnectX3-rel-2_31_5050.tgz

ibv_devinfo | grep board_id

board_id: MT_1090120019

grep MT_1090120019 *

MCX354A-FCB_A2-A5.ini:PSID = MT_1090120019

  1. create a copy of the ini file as a backup

  2. edit the ini file and add the following lines under the [IB] section:

port1_802_3ap_56kr4_enable = true

port1_802_3ap_56kr4_ability = true

port2_802_3ap_56kr4_enable = true

port2_802_3ap_56kr4_ability = true

  1. create a new firmware bin file:

l-supp-11:/tmp/2.31.5050 # mlxburn -fw fw-ConnectX3-rel.mlx -c MCX354A-FCB_A2-A5_56GbE.ini -wrimage fw-ConnectX3-rel-2_31_5050-MCX354A-FCB_A2-A5-56GbE_VPI.bin

-I- Generating image …

-I- Image generation completed successfully.

  1. backup the firmware currently loaded on the HCA:

mst start

flint -d mlx4_0 ri backup.bin

  1. burn the new firmware onto the card:

flint -d mlx4_0 -i fw-ConnectX3-rel-2_31_5050-MCX354A-FCB_A2-A5-56GbE_VPI.bin -y b

  1. reboot the server

You’re done. you can use the same .bin file for the other server if he has the same board_id

Please note that the fw release notes states:

• Added support for 56GbE in all devices supporting 40GbE

• Establishing 56GbE link with SwitchX® requires 56GbE enablement on the

switch

• Establishing 56GbE link on back to back setup requires additions to the INI.

• For further information, please contact Mellanox Support

#3

eddie.notz,

Great, thanks for the quick and detailed response! I have created the new firmware, but will wait until the end of the day when the servers are not in use to attempt burning it.

Regards,

Thomas

#4

eddie.notz,

I flashed the cards today and they did establish link at 56 Gbps. They defaulted to IB, but changing the port type to eth worked. I am now seeing ~ 50 Gbps using MPI over RoCE. Thanks again for your help!

Thomas

#5

Hi guys,

when using the last MLNX-OFED and firmware, there is no need to change the ini file (unless you have two servers back to back)

The adapter automatically discovers and changes the speed automatically to 56GbE.

See the flow over here:

HowTo Configure 56GbE Link on Mellanox Adapters and Switches https://community.mellanox.com/s/article/howto-configure-56gbe-link-on-mellanox-adapters-and-switches

Ophir.

#6

for back-2-back 56GbE connections between ConnectX-3 and ConnectX-3 Pro, the solution provided in this post is still relevent