ConnectX-3 Pro connecting at 10g instead of 40g

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hello. This is my first attempt to use Mellanox products. I am trying to setup a direct link from a VMware 6.0u3 server and a Win2012r2 sever. Both are Dell R730 servers that came with the Mellanox card installed.

I plugged a passive cable from one server directly to the other and link went up with 10gbps. Traffic seems fine, but I need to make it connect at 40g. Here’s a link QSFP+ to QSFP+, 40Gbps, passive, Cisco compatible | DataStorageCables to the cable that I used:

It seems that Dell just stopped selling the card (I bought the servers 2 months ago) and I can’t find any direct references to this card from either Mellanox or Dell.

How can I identify the issue?

Here’s the description of the Dell part:

Dell Part: Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ PCIE Adapter Full Height, MCX33D, [540-BBOP]

Physical label on card: Model No: CX324A

Windows device name: ConnectX-3 PRO VPI (MT04103)

Firmware: 2.36.5080

PN: 0C8Y42 REV:E2

2

The cable was the issue. I got a cable from the list provided, did the loop test and the speed went up to 40g. It also increased the speed to 40g between the Win and Vmware servers.

This is the cable I used: link PROVANTAGE: Mellanox Technologies MC2210126-005 5M 40GE QSFP Passive Copper Cable 26AWG

Thank you.

3

Hi Mark,

Since there is no switch in the middle you will need to set the speed manually.

On the Vmware side there is an option to manually set the link speed , can you please change it and retest?

The command is : esxcli network nic set -n vmnic# -S speed

Configuring the speed and duplex of an ESXi or ESX host network adapter (1004089) | VMware KB VMware Knowledge Base

Thanks,

Karen.

4

I connected the cable in loopback at the Windows server and the connection was established at 10gb. The cable model isn’t listed in the document provided, so it’s likely not fully compatible. I have ordered a cable model from the list. I’ll report back when it is installed.

Thanks.

5

Hi Karen,

I applied the suggested step, but didn’t change the speed. VMware shows “Configured Speed: 40000 Mb” and “Actual Speed: 10000 Mb”. I disabled and re-enabled the NIC in the Windows server. Also unplugged and plugged back the cable. The speed stayed at 10 Gb. I couldn’t find an option to also change the speed to 40000 Mb in the Windows server.

Any more thought on this?

[root@VMhost3:~] esxcfg-nics -l | grep nml

vmnic1000402 0000:82:00.0 nmlx4_en Down 0Mbps Half 24:8a:07:6c:d8:41 1500 Mellanox Technologies MT27520 Family

vmnic4 0000:82:00.0 nmlx4_en Up 10000Mbps Full 24:8a:07:6c:d8:40 1500 Mellanox Technologies MT27520 Family

[root@VMhost3:~] esxcli network nic set -n vmnic4 -S 40000 -D full

[root@VMhost3:~] esxcfg-nics -l | grep nml

vmnic1000402 0000:82:00.0 nmlx4_en Down 0Mbps Half 24:8a:07:6c:d8:41 1500 Mellanox Technologies MT27520 Family

vmnic4 0000:82:00.0 nmlx4_en Up 10000Mbps Full 24:8a:07:6c:d8:40 1500 Mellanox Technologies MT27520 Family

748×553 18.2 KB

6

Hi Mark,

From your output i suspect that this cable does not support 40GbE speed.

I see that your card is dual port, can you connect the cable in loopback (from port #1 to port #2) to the card and see if you are able to get 40 GbE link rate?

For your refernce please also see a list of tested 40GbE cable with your card:

http://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_40_7000-release_notes.pdf http://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_40_7000-release_notes.pdf

Thanks,

Karen.