Fluctuating speeds on 40gb Mellanox

You seem to “mix up” couple of test components in your test-description so I’m not sure I fully understand what is the exact test you’re trying to run?

is it a network performance or a “W/R RAID10 array storage-copy” test you’re running ?

is it over RDMA or is it TCP/UDP? what is the test-tool you’re running?

Assuming you are not using a switch in between but using compatible Mellanox fiber or copper cable connecting “Back-to-Back” between the ConnectX-3 adapters of PC vs. server, then here is what I suggest:

  • Strat with performance fine-tuning of both Win10 & Win2019 as per Mellanox best practice. Use the guidance of WinOF User-Manual,

https://docs.mellanox.com/display/WINOFv55052000/Introduction

  • Next, run RDMA test (nd_read_bw) test between the CX-3 adapters of PC vs. Win2019 server to ensure first you achieve optimum network performance ~40Gb/s between PC & Server. More on RDMA test you’ll find in the User Manual
  • Run “NTttcp” test to ensure you have TCP/UDP optimum network performance ~35-36Gb/s (usually lower then RDMA performance)

https://www.interfacett.com/blogs/performance-testing-and-monitoring-using-free-tool-ntttcp-from-microsoft/

  • if all tests above result are good & expected network performance - you should be able now to get optimum copy performance between the initiator (PC) to the the Storage RAID 10 target. Use “IOmeter_for_Windows” tool "
  • http://www.iometer.org/doc/downloads.html
  • Notice the difference between “network performance” & “copy performance”. copy performance depends on the storage RAID subsystem proper configuration

Hope this helps