I’m playing with some ConenctX-3 cards and seem to be able to connect two machines with 10Gb in ethernet mode just fine, but I saw a press release that said you can get 40Gbps out of these. Can anyone tell me how or tell me how to confirm what speeds I am getting? I currently see it as 10000 Full for speeds. Firmware has been updated to latest. This is using the drivers that came with 5.5.
net-mlx4-core 184.108.40.206-1vmw.5220.127.116.111820 VMware VMwareCertified 2014-01-26
net-mlx4-en 18.104.22.168-1vmw.522.214.171.1241820 VMware VMwareCertified 2014-01-26
Press release talking about 40Gbps.
What is the card’s model number?
There are 10G only and 10/40G types.
Thanks for pointing that out! I hadn’t realized that there were so many variants of the card. So if I were to get a different card that was 40GbE capable, would ESX detect it correctly using the stock drivers? No other configuration needed?
The cards on both ends are:
[root@basement mnt]# ibv_devinfo
transport: InfiniBand (0)
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
Could you please run the following and provide the output:
#esxcli network nic list
~ # esxcli network nic list
Name PCI Device Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:001:00.0 igb Up 1000 Full 00:a0:d1:ec:a7:dc 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic1 0000:001:00.1 igb Down 0 Half 00:a0:d1:ec:a7:dd 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic1000202 0000:004:00.0 mlx4_en Down 0 Half 00:02:c9:9f:55:b1 1500 Mellanox Technologies MT27500 Family [ConnectX-3]
vmnic2 0000:004:00.0 mlx4_en Up 10000 Full 00:02:c9:9f:55:b0 9000 Mellanox Technologies MT27500 Family [ConnectX-3]
As an aside note, are the Mellanox EN drivers bundled in the ESX 5.5 install newer/better than the download links for 1.6.1?
The card you have displayed MT_1090110028 is capable of 40gbps InfiniBand and 10gbps Ethernet. it is not capable of 40gbps Ethernet. Though we do have cards that are capable of 40gbps Ethernet.
ConnectX-3 VPI adapter card; dual-port QSFP; FDR10 IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6
PCI DevID: 4099
Closing the loop. I purchased a pair of the 40GB VPI with 40GbE on it and swapped cards. ESXi and CentOS immediately saw them as 40 Gbps connections. No updating needed.
The mlx4_en driver in ESX 5.5 stock reports as:
~ # ethtool -i vmnic2
version: 126.96.36.199 (Dec-03-2012)
Is this newer than the one provided on the Mellanox and Vmware links
(Mellanox Technologies: VMware: Firmware - Driver Compatibility Matrix http://www.mellanox.com/page/vmware_matrix_hcl?mtag=vmware_driver 0? It appears to according to version number and date.
If so, should I continue to use or should I downgrade to 1.6.1-2? Can I enable all the features and whatnot with the stock 1.9.7 driver?