MNPA19-XTR Connectx-2 / Windows 10 - Slow LAN download speed

Infrastructure & Networking Adapters and Cables
1

Hello all,

I’m new to the mellanox community and using 10gb network equipment in general, however, I’m experiencing some transfer speed issues using two MNPA19-XTR Connectx-2 cards and I’m looking for some help.

System #1

Windows 10 Pro Build 1903

Intel i7-4790k

16gb Ram

Using a 3gb Ramdisk for testing

System #2

Windows Server 2016 Standard Build 1607

Xeon E5-2630 v2

16gb Ram

Using a 3gb Ramdisk for testing

Network Equipment:

Mikrotik CRS305-1G-4S+IN SFP+ 10gb Switch

Fiber connected Finisar FTLX8571D3BCL transceivers or Cisco SFP-H10GB-CU3M SFP+ DAC’s

2x Mellanox MNPA19-XTR Connectx-2 SFP+ network cards.

The specifics of the issue I’m experiencing is that downloading a 2.5gb test file shared on the Windows server 2016 Ramdisk to the Windows 10 Ramdisk only yields an average transfer speed of 45MB/s. The same file, uploaded from my Windows 10 Ramdisk to the Windows server 2016 Ramdisk yields a transfer speed of 632MB/s.

I’ve tried various matching driver versions, MTU settings, Large Send offload on or off, Send / Receive buffer sizes, fiber or copper cabling, different switch ports but the download speed problem persists on Windows 10. I would like to share that I’ve tested the same Ramdisk file transfers using Windows 7 (network card driver version 4.80) and the 2016 server, the download / upload speed are exactly the same and as expected, an average of 630MB/s either way.

Is there a known performance issue with Windows 10 and the Connectx-2 cards? Is there anyone with a suggestion on how I could fix it?

I appreciate any help you all can offer me, I’m getting a bit frustrated.

2

Since you are using a 5y old Mellanox adapter ConnectX-2, which is now EOL (end-of-life), then first thing you need to do is to line up with the “compatible” components.

The only combination of that I found that I think will still work properly with ConnectX-2 is:

O.S. - Win7/Win8.1/Win2008/Win2012(R2)

Driver: WinOF v4.80.5000 (can be downloaded from Mellanox website / Archive)

Adapter: ConnectX-2 fw v2.9.1000

can be found in https://www.mellanox.com/page/firmware_table_ConnectX2IB

More on that you can find in Mellanox website link: http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_Release_Notes_Rev_4.80.pdf

As you can probably see - Win10 & Win2016 will not work properly with ConnectX-2 fw & mellanox driver

Now, assuming that you managed to put together the suggested combination - next would be doing "optimization & performance-fine-tuning"on the system (Driver, fw parameters), as indicated in cahpter 3.8 of the WinOF v4.80 user manual: http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_User_Manual_v4.80.pdf

If all tuning is done properly then run : “Ntttcp” test between 2x CX-2 adapters to ensure you get network-bandwidth around ~9Gb/s

if this is achieved this performance, then you are good to start you other tests

Hope this helps