I have a setup with a Linux server connected directly via a 40 Gb QSFP cable to a desktop PC running Windows 10 that has a Mellanox Connect-X 3 pro network card. I am sending multicast UDP packets on the Linux server and receiving them on the desktop PC.
The problem is that the transmit throughput to the desktop is inconsistent across multiple tests. What I mean by this is if we define a test as send 13 Gbps from the Linux server and receive the packets on the desktop PC, that test will run perfectly around 80 to 95 % of the time. However, sometimes the desktop PC only gets something like 12.5 Gbps instead of 13 Gbps.
If the transmission is running at full rate, it continues to run at full rate regardless of the test duration (e.g. whether the test is run for 5 seconds or a minute). However, if the test starts and is failing (only achieves the lower data rate of say 12.5 Gbps) then it achieves this rate for the duration of the test, regardless of how long it runs. That is, once the UDP transmission rate is 12.5 Gbps instead of 13 Gbps it does not recover or fluctuate, it is rock solid at the erroneous data rate and visa versa for when the test achieves the correct data rate. (At higher test data rates, the erroneous rate does fluctuate as shown below)
I have used iperf, my own multicast benchmark app and the actual program for which I need this to work and they all exhibit this strange behavior.
What I have tried:
- Windows Server 2019 instead of Windows 10: Same behavior
- Play with all the network card driver settings: Same behavior
- Install the latest driver and the oldest driver that can still install: Same behavior
- Install Linux on the desktop PC instead of Windows: I can reliably get close to 40 Gbps UDP multicast with both iperf and my multicast benchmark app with practically 0 % packet loss.
- Log the receive socket buffer size once packets go missing: The rx buffer does not appear to be close to full
- Only join the multicast group and do no reading of the sockets and watch the data rate on Task Manager/Performance/Network graph. Even in this case you can see the lower data rate happens, leading me to believe it’s an issue with the network card / driver / Windows rather than with how fast these test apps are reading data off the sockets.
- Windows sockets trace logging, I don’t know exactly what to do with these files but I couldn’t see much difference between the traces obtained from a test with the erroneous rate versus a test with the valid rate.
- Firewall is turned off
- UDP Unicast rather than multicast reliably gets close to 40 Gbps reception on the Windows desktop so it’s probably specifically related to multicast.
Here’s a screenshot of how it looks in Task Manager while running iperf. I probably ran the test about 30 times or so before getting the erroneous data rate. You can see from the CPU usage as well that at the erroneous data rate less CPU was used / allocated.
Task manager view of the data rates across multiple iperf tests
This was the iperf invocation on the Linux server:
!#bin/bash
drh=4G
time_seconds=5
buffer=24M
iperf -c 224.0.0.37 -u -B 10.236.0.2 -i 1 -b $drh -w $buffer -l 8958 -t $time_seconds -p 5001 &
iperf -c 224.0.0.37 -u -B 10.236.0.2 -i 1 -b $drh -w $buffer -l 8958 -t $time_seconds -p 5002 &
iperf -c 224.0.0.37 -u -B 10.236.0.2 -i 1 -b $drh -w $buffer -l 8958 -t $time_seconds -p 5003 &
iperf -c 224.0.0.37 -u -B 10.236.0.2 -i 1 -b $drh -w $buffer -l 8958 -t $time_seconds -p 5004 &
And this on the Windows desktop, run in 4 consoles with ports 5001 through to 5004:
iperf-2.0.14a-win.exe -s -B 224.0.0.37%%10.236.0.3 -u -l 8958 -i 1 -w 12M -p 5001
In the erroneous condition there is packet loss across all 4 ports.
If I run an app which only opens sockets, joins the multicast group and does nothing else I get something like this (while sending multicast data at around 34.5 Gbps). Again, note the CPU correlation, which makes it look like some kind of load balancing / thermal throttling issue. But why does it not recover from this condition?
Task Manager when multicast group is joined but sockets are not read: AXoQRYg0vLx7.png Delete the space I can only put one link in the post (Same site as the first image)… .
On Windows with iperf I get about 5 % loss at 21 Gbps and 0 % at 17 Gbps. So I would expect that 13 Gbps should work reliably. The maximum level is worse than Linux in any case as mentioned above (40 Gbps at practically 0 % loss).
My questions are:
- Does it look like this problem is with the network card driver or Windows? Where should further questions be directed? (I’m probably going to post / link on the Nvida forum as well)
- Why does the data rate not recover? I’d find it more acceptable if it eventually recovered without closing / reopening sockets.
- Is reliable 15 Gbps UDP multicast transmission possible on Windows or should I just forget it and use Linux?
- Is there anything else I should have tried?