MCX354A-FCBT / ConnectX-3 in Ethernet mode

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

Hi all

I’m pretty new to Mellanox adapters. Just got some ConnectX-3 VPI (MCX354A-FCBT) adapters and want to use the in my VMware homelab in 40G Ethernet mode. (no IB)

my plan is to connect the ESXi host to a Nexus switch with 40G QSFP ports.

is there some tutorial or best practice guide to accomplish this ?

  • which firmware to use

  • what drivers to load in ESXi 6

  • how to change ConnectX-3 ports to 40G Ethernet mode

  • what QSFP-cables to use (prefer Copper) / will this one work with a Nexus-switch ? (MC2207128-003 - FDR InfiniBand QSFP passive copper cable, 3m)

feedback is very much appreciated

regards,

2

Hi Mark,

For Connectx-3 and Esxi 6.0 please install the MLNX-OFED-ESX 3.15.5.5 version, for your reference please see the following download link:

http://www.mellanox.com/page/products_dyn?product_family=29&mtag=vmware_driver http://www.mellanox.com/page/products_dyn?product_family=29&mtag=vmware_driver

Please note that you will also need to upgrade the card firmware version to 2.36.5000 and above.

For this you would need to use our MFT tool package to query and burn new firmware if needed, this can be downloaded from here:

Mellanox Firmware Tools (MFT) Mellanox Firmware Tools (MFT)

For changing the speed on the host side please see the following article from VMware:

VMware Knowledge Base VMware Knowledge Base

if the connection is through a switch please also confirm that the switch port is configured to 40 G speed.

Thanks,

Karen.

3

Hi!

At first you must update latest MCX354A-FCBT firmware then change to ETH mode with firmware tools.

Then you can use your hca like general ethernet adapter on any Operating Systems that supported from Mellanox driver.

4

Hi karen

thanks for feedback !

just installed MFT and MST (mft-4.6.0.48-10EM-600.0.0.2494585.x86_64.vib and net-mst_4.6.0.48-1OEM.600.0.0.2494585.vib)

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

VIBs Installed: Mellanox_bootbank_mft_4.6.0.48-0

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

VIBs Installed: MEL_bootbank_net-mst_4.6.0.48-1OEM.600.0.0.2494585

but got some issue when using mst; it says mst is loaded but when issuing “mst status” it says it is not running ?

adapter is present and running, see below:

start mst by

[esxi-02:~] /opt/mellanox/bin/mst start

Module mst loaded successfully

[esxi-02:~] /opt/mellanox/bin/mst status -v

No MST devices were found or MST modules are not loaded.

You may need to run ‘mst start’ to load MST modules.

[esxi-02:~] /opt/mellanox/bin/mst start

Module mst is already loaded

[esxi-02:~] lspci | grep Mellanox

0000:01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] [vmnic2]

[esxi-02:~] esxcli software vib list | grep Mellanox

mft 4.6.0.48-0 Mellanox PartnerSupported 2017-05-21

[esxi-02:~] esxcli system module list | grep mlx4

nmlx4_core true true

nmlx4_en true true

nmlx4_rdma true true

5

works now; choose the wrong type. with type “natiive” instead “vmk” it does the trick.

mft-4.6.0.48-10EM-600.0.0.2768847.x86_64.vib

nmst-4.6.0.48-1OEM.600.0.0.2768847.x86_64.vib

running now with latest fw:

/opt/mellanox/bin/flint -d mt4099_pci_cr0 query

Image type: FS2

FW Version: 2.40.7000

FW Release Date: 22.3.2017

Product Version: 02.40.70.00

Rom Info: type=PXE version=3.4.746 devid=4099

Device ID: 4099

Also set “LINK_TYPE” to Ethernet by

/opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2