Windows 10 Connect-X 3 client capped at about 10Gbit, expected 56Gbit

Hello

I’ll start with a few bullets

  1. I have a few systems with ConnectX-3 adapters connected to SX1012 switches. They’re running in 56Gbit to a DDN MediaScaler.
  2. A new client have been added, a windows 10 client. The first windows 10 client in this setup.
  3. Switch port config and also NIC config in windows is the same as one of the windows 7 hosts. Well, with the exception that the windows 10 client have a newer driver and hence some other/renamed options.
  4. This windows 10 client is capped at, what looks like, 10Gbit in the system. I’ve run multiple and parallel iPerf tests and have done comparisons with the other systems. Can’t get it to go faster and if anything this host should be faster than the win7 client since it’s newer and more modern.

To me it feels like there’s something I have missed in the somewhat newer driver in Windows 10. Have any one seen something similar?

Switch port config of NEW windows 10 host, m5-clipster02:

show interfaces ethernet 1/7

Eth1/7

Admin state: Enabled

Operational state: Up

Description: m5-clipster02_GPFS

Mac address: e4:1d:2d:39:52:f4

MTU: 9216 bytes(Maximum packet size 9238 bytes)

Fec: auto

Flow-control: receive on send on

Actual speed: 56 Gbps

Width reduction mode: disabled

Switchport mode: access

MAC learning mode: Enabled

Last clearing of “show interface” counters : 02:03:04

60 seconds ingress rate: 740016 bits/sec, 92502 bytes/sec, 659 packets/sec

60 seconds egress rate: 127499808 bits/sec, 15937476 bytes/sec, 1932 packets/sec

Rx

66792621 packets

66792141 unicast packets

267 multicast packets

213 broadcast packets

580396963241 bytes

0 error packets

0 discard packets

Tx

26885122 packets

26879548 unicast packets

4675 multicast packets

899 broadcast packets

21021660379 bytes

0 discard packets

show interfaces ethernet 1/9 transceiver

Port 1/9 state

identifier : QSFP+

cable/ module type : Passive copper, unequalized

ethernet speed and type: 56GigE

vendor : Mellanox

cable length : 4m

part number : MC2207126-004

revision : A3

serial number : MT1444VS02832

Switch port config of OLD windows 7 host, clipster_gpfs:

show interfaces ethernet 1/9

Eth1/9

Admin state: Enabled

Operational state: Up

Description: Clipster_GPFS

Mac address: e4:1d:2d:39:52:cf

MTU: 9216 bytes(Maximum packet size 9238 bytes)

Fec: auto

Flow-control: receive on send on

Actual speed: 56 Gbps

Width reduction mode: disabled

Switchport mode: access

MAC learning mode: Enabled

Last clearing of “show interface” counters : 02:04:42

60 seconds ingress rate: 496 bits/sec, 62 bytes/sec, 1 packets/sec

60 seconds egress rate: 864 bits/sec, 108 bytes/sec, 1 packets/sec

Rx

140469415 packets

140468154 unicast packets

314 multicast packets

947 broadcast packets

1144807131028 bytes

0 error packets

0 discard packets

Tx

54283116 packets

54277639 unicast packets

4760 multicast packets

717 broadcast packets

207471985950 bytes

0 discard packets

show interfaces ethernet 1/7 transceiver

Port 1/7 state

identifier : QSFP+

cable/ module type : Passive copper, unequalized

ethernet speed and type: 56GigE

vendor : Mellanox

cable length : 4m

part number : MC2207126-004

revision : A3

serial number : MT1436VS02630

Cheers and thanks.

there are several factors that may result a low iperf “network performance” within the windows 10 client

  1. what is the PCIe slot type that the CX-3 adapter is plugged in? is it a “strong” Gen3, or is it an old Gen2 ? calculate first the maximum performance that you can get from HW perspective.
  2. if it is Gen3 and higher, then next I wound suggest that you run fw, driver & hw “performance fine-tuning” as instructed in Mellanox WinOF release-note (can be downloaded from our Mellanox website). this may significantly improve your network performance
  3. run: # Get-NetadapterRss - ensure that the cores are well distributed over the relevant CPU/s-NUMA/s
  4. run “NTttcp” performance tool and not “iperf”, since iperf is “Linux” oriented test tool (working synchronous) while “NTttcp” is a windows oriented tool (working Asynchronous). If you must use iperf, then use iperf2 which enables multithreading…
  5. ensure that you have identical fw version & WinOF driver versions on the 2x test-nodes

hope this helps