I recently installed a Mellanox ConnectX-4 Lx 25GbE NIC in my desktop and also in my server. Because of performance issues, I’ve connected them directly with a DAC cable. The DAC has a 25gbps link.
For some reason, I can not get more than 7.3gbps when testing. I’ve repeated the below test many times and end op with the same result.
C:\Program Files\Mellanox\MLNX_WinOF2\Performance Tools>nd_send_bw -a -C 10.0.1.4
#qp #bytes #iterations MR [Mmps] Gb/s CPU Util.
0 1 100000 3.841 0.03 100.00
0 2 100000 3.954 0.06 100.00
0 4 100000 3.978 0.13 62.06
0 8 100000 3.963 0.25 100.00
0 16 100000 3.947 0.51 61.58
0 32 100000 3.953 1.01 100.00
0 64 100000 3.938 2.02 61.44
0 128 100000 3.856 3.95 100.00
0 256 100000 2.699 5.53 100.00
0 512 100000 1.536 6.29 95.99
0 1024 100000 0.825 6.76 100.00
0 2048 100000 0.428 7.02 100.00
0 4096 100000 0.218 7.16 98.98
0 8192 100000 0.110 7.23 100.00
0 16384 100000 0.055 7.27 100.00
0 32768 100000 0.028 7.28 100.00
0 65536 100000 0.014 7.29 100.00
0 131072 100000 0.007 7.30 100.00
I noticed that the Cpu Util in the output “100%”. What is that figure from? My CPUs on both ends are at 10% max when I’m doing a transfer. I’m wondering if that’s the issue?
I’ve written the following Powershell script to adjust the driver parameters:
cls
$location = "HKLM:\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\*"
$nics = gci $location | where-object { $_.GetValue('DriverDesc') -like 'Mellanox*' }
# Window performance tweaks according to Nvidia documentation
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" -Name "SackOpts" -Value 0 -Type DWORD
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\AFD\Parameters" -Name "FastSendDatagramThreshold" -Value 64000 -Type DWORD
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\Ndis\Parameters" -Name "RssBaseCpu" -Value 1 -Type DWORD
netsh int tcp set global rss = enabled
foreach ($n in $nics) {
$n.PSPath
#server
#Set-ItemProperty -Path $n.PSPath -Name '*MaxRssProcessors' -Value "32" -Type String
#all
Set-ItemProperty -Path $n.PSPath -Name '*NumRSSQueues' -Value "8" -Type String
Set-ItemProperty -Path $n.PSPath -Name '*ReceiveBuffers' -Value "1024" -Type String
Set-ItemProperty -Path $n.PSPath -Name '*TransmitBuffers' -Value "2048" -Type String
Set-ItemProperty -Path $n.PSPath -Name '*NetworkDirect' -Value "1" -Type String
Set-ItemProperty -Path $n.PSPath -Name 'TxIntModerationProfile' -Value "2" -Type String
Set-ItemProperty -Path $n.PSPath -Name 'RxIntModerationProfile' -Value "2" -Type String
#Testing
Set-ItemProperty -Path $n.PSPath -Name 'AsyncReceiveIndicate' -Value "1" -Type String
Set-ItemProperty -Path $n.PSPath -Name 'RfdReservationFactor' -Value "10000" -Type String
# Research and no diff (set back to default)
Set-ItemProperty -Path $n.PSPath -Name '*JumboPacket' -Value "1512" -Type String
Set-ItemProperty -Path $n.PSPath -Name '*MaxRssProcessors' -Value "24" -Type String
# Researched and helps
Set-ItemProperty -Path $n.PSPath -Name '*FlowControl' -Value "3" -Type String
}
$adapters = Get-NetAdapter | where -Property "InterfaceDescription" -like "Mellanox*"
foreach ($a in $adapters) {
Disable-NetAdapter -Name $a.Name -Confirm:$false
Enable-NetAdapter -Name $a.Name -Confirm:$false
}
None of the above tweaks seem to affect anything, except for Flow Control, which drops performance when disabled.
According to the driver Info, my cards are connected to a PCI-E 8.0GT/s x4 port which should be good for 3.938 GB/s.
When copying and pasting using using SMB, the speed starts out at 1.75GB/s, fluctuates a little, and then it drops down to 0, waits for a few seconds, then jumps back up to 1.75. I’m not sure why it keeps disconnecting. Speeds are good, then it drops, so I end up with just 7gbps because it keeps dropping the connection in the middle of the transfers.