40Gb Ethernet in VMWare ESX 5.5?

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

I’m playing with some ConenctX-3 cards and seem to be able to connect two machines with 10Gb in ethernet mode just fine, but I saw a press release that said you can get 40Gbps out of these. Can anyone tell me how or tell me how to confirm what speeds I am getting? I currently see it as 10000 Full for speeds. Firmware has been updated to latest. This is using the drivers that came with 5.5.

net-mlx4-core 1.9.7.0-1vmw.550.0.0.1331820 VMware VMwareCertified 2014-01-26

net-mlx4-en 1.9.7.0-1vmw.550.0.0.1331820 VMware VMwareCertified 2014-01-26

Press release talking about 40Gbps.

NVIDIA Corporation - Home NVIDIA Corporation - Home

2

What is the card’s model number?

There are 10G only and 10/40G types.

3

Thanks for pointing that out! I hadn’t realized that there were so many variants of the card. So if I were to get a different card that was 40GbE capable, would ESX detect it correctly using the stock drivers? No other configuration needed?

Thanks,

Charles

4

The cards on both ends are:

[root@basement mnt]# ibv_devinfo

hca_id: mlx4_0

transport: InfiniBand (0)

fw_ver: 2.30.8000

node_guid: 0002:c903:009f:5a20

sys_image_guid: 0002:c903:009f:5a23

vendor_id: 0x02c9

vendor_part_id: 4099

hw_ver: 0x0

board_id: MT_1090110028

phys_port_cnt: 1

port: 1

state: PORT_DOWN (1)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: InfiniBand

5

Hi,

Could you please run the following and provide the output:

#esxcli network nic list

6

~ # esxcli network nic list

Name PCI Device Driver Link Speed Duplex MAC Address MTU Description

vmnic0 0000:001:00.0 igb Up 1000 Full 00:a0:d1:ec:a7:dc 1500 Intel Corporation 82576 Gigabit Network Connection

vmnic1 0000:001:00.1 igb Down 0 Half 00:a0:d1:ec:a7:dd 1500 Intel Corporation 82576 Gigabit Network Connection

vmnic1000202 0000:004:00.0 mlx4_en Down 0 Half 00:02:c9:9f:55:b1 1500 Mellanox Technologies MT27500 Family [ConnectX-3]

vmnic2 0000:004:00.0 mlx4_en Up 10000 Full 00:02:c9:9f:55:b0 9000 Mellanox Technologies MT27500 Family [ConnectX-3]

As an aside note, are the Mellanox EN drivers bundled in the ESX 5.5 install newer/better than the download links for 1.6.1?

7

Hello,

The card you have displayed MT_1090110028 is capable of 40gbps InfiniBand and 10gbps Ethernet. it is not capable of 40gbps Ethernet. Though we do have cards that are capable of 40gbps Ethernet.

ConnectX-3 VPI adapter card; dual-port QSFP; FDR10 IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6

PCI DevID: 4099

8

Closing the loop. I purchased a pair of the 40GB VPI with 40GbE on it and swapped cards. ESXi and CentOS immediately saw them as 40 Gbps connections. No updating needed.

Followup question.

The mlx4_en driver in ESX 5.5 stock reports as:

~ # ethtool -i vmnic2

driver: mlx4_en

version: 1.9.7.0 (Dec-03-2012)

firmware-version: 2.30.8000

bus-info: 0000:04:00.0

Is this newer than the one provided on the Mellanox and Vmware links

(Mellanox Technologies: VMware: Firmware - Driver Compatibility Matrix http://www.mellanox.com/page/vmware_matrix_hcl?mtag=vmware_driver 0? It appears to according to version number and date.

If so, should I continue to use or should I downgrade to 1.6.1-2? Can I enable all the features and whatnot with the stock 1.9.7 driver?