ESXi 6.5U1 40,56Gb Ethernet Performance problems


I’m test lossy TCP/IP test on ESXi 6.5U1.

Here is a sample link.

Speed testing 40G Ethernet in the Homelab | Erik Bussink Speed testing 40G Ethernet in the Homelab | Erik Bussink

I’m also test with same ConnectX-3 MCX354A-FCBT with latest firmware.

All bunch of MCX354A-FCBT with vSwitch MTU 9000 connect to SX6036G Gateway Switch systems with port MTU 9000

Q01. Any switch mode (include VPI-single-mode or Ethernet mode 40GbE port configuration) in SX6036G 40GbE port configuration shows me a terrible performance level.

But If change port mode to 10GbE mode shows me a normal 10GbE ethernet performance.

Do you have a any solution?

Q02. latest MCX354-FCBT firmware can’t link-up properly with SX6036G’s 56GbE port mode.

Do you have any a solution?

Assuming you are using Mellanox-Native Esxi driver then as far as I recall - Esxi6.5 does not support Connectx-3 adapter (only Cx-4/5)

Nevertheless, from your description, it looks though your are running a VM to VM test

I would suggest that you first “dismount” the vswitch / virtual adapter mode and try a simple “bare-metal” 40G performance test from an Esxi_server-to-Esxi_server, meaning from a physical adapter (CX-3) to another CX-3 adapter

Ensure you have identical MTU size on both switch & Exsi server (MTU 9000)

Also suggesting you approach Mellanox website to read more about the Esxi6.5 va. Mellanox adapter compatibility matrix

I test seperate VM on local SSD & ESXi 6.5 host.

Here is briefs

ESX host 01 - iPerf VM 01 - SX6036G - ESX host 02 - iPerf VM 02

All switch & vSwitch, vKernel adapter, nmlx_en adapter set to 9k MTU.

And ESXi 6.5 update 1 inbox driver include nmlx4_core that support Mellanox ConnectX-3, Pro and belows.

Why did you comment that based your personal memorandum?

In another thread on this community said CX-3 supported with ESXi 6.5 inbox driver.

I’ll test Arista 40Gb switch with same configuration then I’ll update result.

Best Regard,

Jae-Hoon Choi

Update 01. Direct connection between 2 of CX-3s test

This test shows almost 19Gb/s grade performance

Update 02. SX6036G Global Pause On switched test

This test also shows almost 19Gb/s grade performance

Update 03. SX6036G all port 10GbE switched test

This test also shows almost 10Gb/s grade performance

Update 04. ESXi inbox Ethernet driver packet drop bug - ESXi host iPerf test results…:(

It must have a packet drop & error bug in Mellanox inbox ESXi drivers.

How do you think about below?

Update 05. Windows, Linux Host test

All Windows, Linux Host show me a greate performance on 40GbE configurations that include CX-3 HCAs & SX6036G switches.

Best Regard,

Jae-Hoon Choi

Update 01.

I can build a 56Gb Ethernet TCP/IP network with ConnectX-3 (firmware 2.42.5000) & SX6036G (firmware 3.6.4006), now.

packet drop was not appeared that I don’t use PFC configuration with ConnectX-3 ESXi 6.5 update 1.

40, 56GbE Link-up require small delay to establish requested speed.

But performance level is terrible like below.

In 56GbE mode, I’m test iPerf test between physical ESXi host then I met a almost 6.11Gb/s performance only.

* 56GbE iPerf client - physical ESXi 6.5 update 1 host 01 with MTU 9000

* 56GbE iPerf server - physical ESXi 6.5 update 1 host 02 with MTU 9000

Is there any configuration shold be need on SX6036G?

* This is a traditional TCP/IP Network, not RDMA Network.

* When 56Gb ethernet show me a 56GbE throughput, I’ll switch to another test like iSER driver and so on.


Jae-Hoon Choi

I found what’s the problem myself.

All of 40GbE or above architecture on the planet consists of 4 lane configuration.

That’s a 1st problem, 2nd is Mellanox CX-3 chipset’s TSO bugs.

If you disable TSO offload then you’ll meet a improve of performance, but also meet a high CPU usages.

I gave up the performance TCP/IP tuning with 40GbE or above.


Jae-Hoon Choi