Enviroment: connectx3 with last firmware and last drivers. Sfp+ dac cable direct connected to same card.Host A is Windows 10 last upgrade, in a pcie 2.0 4x (5gx4), host B is freenas 11.5 last update(mtu9000, txrxbuffers 4096 on win). From B to A with Iperf2 i see 9.5gb speed with -P 10, from A to B,same iperf2 setting, i see 1,5/1,7 gb…if i install freenas on host A( using same hardware to check power) iperf2 gives me 9,5gb, so i think problem is windows10. Is win10 not capable to use the capacities of mellanox card?
When coming to measure “network performance” between 2x Mellanox ConnectX-3 adapters on Windows platforms – my recommendation is that you use NTttcp tool rather than iperf2 tool
That is because NTttcp is more of a “windows TCP/UDP oriented” tool and iperf2 is a “Linux TCP/UDP oriented tool”
It is not clear whether you intend using RDMA / SMBDirect storage protocol to access the FreeSAN shared file-system? Or you will just use TCP/UDP traffic?
In close, the problem is probably not in Win10. I think you need first to implement the “performance tuning” on driver / fw / HW parameters
Suggesting:
-
Ensure you’ve installed on the Win10 clients the latest WinOF driver v5.50 & latest fw v2.42.5000,
-
Make sure you use Mellanox certified cables in the E2E direct connection between the Win10 Clients
-
before installing FreeSAN on the clients, start with “performance fine-tuning” of all clients, as per Mellanox best practice. Use the guidance of WinOF User-Manual,
https://docs.mellanox.com/display/WINOFv55052000/Introduction
- Run “NTttcp” test to ensure you have TCP/UDP optimum “network performance” between 2x Clients’ ConnectX-3. If for example a you expect 20Gb/s speed then the optimum results that NTttcp will provide is ~18Gb/s (NTttcp is working in the kernel-space, thus, slower)
- In addition - since you use a relevant slow PCIe Gen2.0 then you will not be able to accede ~25Gb/s bandwidth, as you’ll find in the “ Calculating pcie limiration” article
https://community.mellanox.com/s/article/understanding-pcie-configuration-for-maximum-performance
- MTU=9000 is fine as it provides better network performance
Thanks for reply Avi,
but pci is ok
Maximum PCIe Bandwidth = 5G * 16 * (1 - 1/5) - 1G = 80G * 0.8 - 1G = ~63Gb/s
i’m using pci 2.0 x4 (15,75gbps for a 10Gbps card is ok)
if on hardware used for win10, i install freebsd or ubuntu or freenas, i see simmetrical 9,76gbps on both directions.
I saw and know ntttcp, but on nix os and win os the only tool capable is iperf, i will try a test pc with win os for ntttpc, but the results will tell me only about win performance….freenas is bsd similar, and it is less complicated to switch the os client, rather than nas server os
Mtu is 9014 on both peers, and cabling is certified….Windows 10, or Windows 8,1i used for a test, gives me 1,4/1,6Gbps on results.
And strange thing is if i try a smb share, from win10 to freenas, i see 2,8/3,4Gbps speed, due to sata3 limit…so tested a ramdisk too with same limit results.
I will use the enviroment only for smb shares and plex streams only, so 300MB of speed for share connections is good enough, but i use to investigate why i’m seeing these stranges results.
i’ve seen manual but for win10 there are few documentation,please could you be more specific on pointing me in the right direction?
thanks for support and have a wonderful day