I have the following setup (simplified):
Client (ConnectX 5 100g) ← 100g fiber → Switch (Mikrotik CRS510) ← 100g DAC → Router (ConnectX 4 2x 100g) ← 25g fiber–> Internet
Running a speed test on the router yields ~22g download/upload to the internet.
Running iperf from client to router yields 70-90g.
Running a speed test on the client to internet gets ~22g download but just 400m upload.
Running iperf from client to internet shows the same slow upload.
The router has a dual port ConnectX 4. One trunk port with multiple vlans to the switch, and one plain to the internet. I’ve tested both with VyOS and with a Live CD Debian 12. Also tested with different clients, all same result. With the Live CD I tested with very simple setup (NAT + allow all outbound / established). Also tried moving WAN to the switch as a VLAN on the same interface as LAN vlans. Also tried connecting client directly to the router without any switch in between.
Doing download tests I get visible CPU load for handling the 22g, but doing upload the CPU (7700X) is almost idle.
I tried setting/disabling different offloads, so far no idea what else to test. MTU on all interfaces is 1500. Upgraded to latest ConnectX firmware etc.
Some test results
tcpdump from routers LAN interface during upload test: sudo tcpdump -i eth1 -s 0 host 77.109.175.63
(77.109.175.63 is the remote speedtest server)
No Time Src Dst Pro Length Info
...
26 0.070335 10.2.2.1 77.109.175.63 TCP 34822 51434 → 8080 [ACK] Seq=1841857 Ack=1 Win=65535 Len=34752 TSval=1561212148 TSecr=599954409
27 0.070503 10.2.2.1 77.109.175.63 TCP 1518 51434 → 8080 [ACK] Seq=1876609 Ack=1 Win=65535 Len=1448 TSval=1561212148 TSecr=599954409
28 0.070741 10.2.2.1 77.109.175.63 TCP 24686 51434 → 8080 [ACK] Seq=1878057 Ack=1 Win=65535 Len=24616 TSval=1561212148 TSecr=599954409
29 0.070804 10.2.2.1 77.109.175.63 TCP 17446 51434 → 8080 [ACK] Seq=1902673 Ack=1 Win=65535 Len=17376 TSval=1561212148 TSecr=599954409
30 0.071079 10.2.2.1 77.109.175.63 TCP 29030 51434 → 8080 [ACK] Seq=1920049 Ack=1 Win=65535 Len=28960 TSval=1561212148 TSecr=599954409
31 0.071129 10.2.2.1 77.109.175.63 TCP 7310 51434 → 8080 [ACK] Seq=1949009 Ack=1 Win=65535 Len=7240 TSval=1561212148 TSecr=599954409
32 0.071244 10.2.2.1 77.109.175.63 TCP 17446 51434 → 8080 [ACK] Seq=1956249 Ack=1 Win=65535 Len=17376 TSval=1561212148 TSecr=599954409
33 0.071255 10.2.2.1 77.109.175.63 TCP 10206 51434 → 8080 [ACK] Seq=1973625 Ack=1 Win=65535 Len=10136 TSval=1561212148 TSecr=599954409
34 0.071595 10.2.2.1 77.109.175.63 TCP 37718 51434 → 8080 [ACK] Seq=1983761 Ack=1 Win=65535 Len=37648 TSval=1561212148 TSecr=599954409
35 0.071648 10.2.2.1 77.109.175.63 TCP 11654 51434 → 8080 [ACK] Seq=2021409 Ack=1 Win=65535 Len=11584 TSval=1561212148 TSecr=599954409
36 0.071783 10.2.2.1 77.109.175.63 TCP 39166 51434 → 8080 [ACK] Seq=2032993 Ack=1 Win=65535 Len=39096 TSval=1561212148 TSecr=599954409
37 0.071952 10.2.2.1 77.109.175.63 TCP 21790 51434 → 8080 [ACK] Seq=2072089 Ack=1 Win=65535 Len=21720 TSval=1561212148 TSecr=599954409
38 0.072286 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=1876609 Win=16319 Len=0 TSval=599954411 TSecr=1561212148
39 0.072324 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=1878057 Win=16342 Len=0 TSval=599954411 TSecr=1561212148
40 0.072491 10.2.2.1 77.109.175.63 TCP 65230 51434 → 8080 [ACK] Seq=2093809 Ack=1 Win=65535 Len=65160 TSval=1561212148 TSecr=599954409
41 0.072494 10.2.2.1 77.109.175.63 TCP 10206 51434 → 8080 [ACK] Seq=2158969 Ack=1 Win=65535 Len=10136 TSval=1561212148 TSecr=599954409
42 0.072635 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=1902673 Win=16494 Len=0 TSval=599954412 TSecr=1561212148
43 0.072661 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=1920049 Win=16531 Len=0 TSval=599954412 TSecr=1561212148
44 0.072967 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=1949009 Win=16470 Len=0 TSval=599954412 TSecr=1561212148
45 0.072982 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=1956249 Win=16432 Len=0 TSval=599954412 TSecr=1561212148
46 0.073036 10.2.2.1 77.109.175.63 TCP 65230 51434 → 8080 [ACK] Seq=2169105 Ack=1 Win=65535 Len=65160 TSval=1561212148 TSecr=599954409
47 0.073043 10.2.2.1 77.109.175.63 TCP 4414 51434 → 8080 [ACK] Seq=2234265 Ack=1 Win=65535 Len=4344 TSval=1561212148 TSecr=599954409
48 0.073098 10.2.2.1 77.109.175.63 TCP 2966 51434 → 8080 [ACK] Seq=2238609 Ack=1 Win=65535 Len=2896 TSval=1561212148 TSecr=599954409
49 0.073113 10.2.2.1 77.109.175.63 TCP 7310 51434 → 8080 [ACK] Seq=2241505 Ack=1 Win=65535 Len=7240 TSval=1561212148 TSecr=599954409
50 0.073179 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=1983761 Win=16478 Len=0 TSval=599954412 TSecr=1561212148
51 0.073272 10.2.2.1 77.109.175.63 TCP 17446 51434 → 8080 [ACK] Seq=2248745 Ack=1 Win=65535 Len=17376 TSval=1561212148 TSecr=599954409
52 0.073283 10.2.2.1 77.109.175.63 TCP 5862 51434 → 8080 [ACK] Seq=2266121 Ack=1 Win=65535 Len=5792 TSval=1561212148 TSecr=599954409
53 0.073377 10.2.2.1 77.109.175.63 TCP 18894 51434 → 8080 [ACK] Seq=2271913 Ack=1 Win=65535 Len=18824 TSval=1561212148 TSecr=599954409
54 0.073536 10.2.2.1 77.109.175.63 TCP 20342 51434 → 8080 [ACK] Seq=2290737 Ack=1 Win=65535 Len=20272 TSval=1561212148 TSecr=599954409
55 0.073577 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=2032993 Win=16365 Len=0 TSval=599954413 TSecr=1561212148
56 0.073589 10.2.2.1 77.109.175.63 TCP 4414 51434 → 8080 [ACK] Seq=2311009 Ack=1 Win=65535 Len=4344 TSval=1561212148 TSecr=599954409
57 0.073686 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=2072089 Win=16418 Len=0 TSval=599954413 TSecr=1561212148
58 0.073808 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=2093809 Win=16509 Len=0 TSval=599954413 TSecr=1561212148
59 0.073820 10.2.2.1 77.109.175.63 TCP 36270 51434 → 8080 [ACK] Seq=2315353 Ack=1 Win=65535 Len=36200 TSval=1561212148 TSecr=599954409
60 0.073832 10.2.2.1 77.109.175.63 TCP 10206 51434 → 8080 [ACK] Seq=2351553 Ack=1 Win=65535 Len=10136 TSval=1561212148 TSecr=599954409
61 0.073916 10.2.2.1 77.109.175.63 TCP 11654 51434 → 8080 [ACK] Seq=2361689 Ack=1 Win=65535 Len=11584 TSval=1561212148 TSecr=599954409
62 0.074039 10.2.2.1 77.109.175.63 TCP 10206 51434 → 8080 [ACK] Seq=2373273 Ack=1 Win=65535 Len=10136 TSval=1561212148 TSecr=599954409
63 0.074053 10.2.2.1 77.109.175.63 TCP 1518 51434 → 8080 [ACK] Seq=2383409 Ack=1 Win=65535 Len=1448 TSval=1561212148 TSecr=599954409
64 0.074376 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=2116977 Win=16501 Len=0 TSval=599954413 TSecr=1561212148
65 0.074437 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=2169105 Win=16094 Len=0 TSval=599954413 TSecr=1561212148
66 0.074658 10.2.2.1 77.109.175.63 TCP 65230 51434 → 8080 [ACK] Seq=2384857 Ack=1 Win=65535 Len=65160 TSval=1561212148 TSecr=599954409
67 0.074659 10.2.2.1 77.109.175.63 TCP 5862 51434 → 8080 [ACK] Seq=2450017 Ack=1 Win=65535 Len=5792 TSval=1561212148 TSecr=599954409
68 0.074715 10.2.2.1 77.109.175.63 TCP 5862 51434 → 8080 [ACK] Seq=2455809 Ack=1 Win=65535 Len=5792 TSval=1561212148 TSecr=599954409
69 0.074964 77.109.175.63 10.2.2.1 TCP 70 8080 → 51434 [ACK] Seq=1 Ack=2187929 Win=16524 Len=0 TSval=599954414 TSecr=1561212148
....
tcpdump from client interface during upload test: sudo tcpdump -i en14 -s 0 host 77.109.175.63
4 0.000039 10.2.2.1 77.109.175.63 TCP 1514 54124 → 8080 [ACK] Seq=14481 Ack=1 Win=65535 Len=1448 TSval=2255708013 TSecr=602381411
5 0.000039 10.2.2.1 77.109.175.63 TCP 1514 54124 → 8080 [ACK] Seq=15929 Ack=1 Win=65535 Len=1448 TSval=2255708013 TSecr=602381411
6 0.000040 10.2.2.1 77.109.175.63 TCP 1514 54124 → 8080 [ACK] Seq=17377 Ack=1 Win=65535 Len=1448 TSval=2255708013 TSecr=602381411
7 0.000041 10.2.2.1 77.109.175.63 TCP 1514 54124 → 8080 [ACK] Seq=18825 Ack=1 Win=65535 Len=1448 TSval=2255708013 TSecr=602381411
8 0.000041 10.2.2.1 77.109.175.63 TCP 1514 54124 → 8080 [ACK] Seq=20273 Ack=1 Win=65535 Len=1448 TSval=2255708013 TSecr=602381411
9 0.000042 10.2.2.1 77.109.175.63 TCP 1514 54124 → 8080 [ACK] Seq=21721 Ack=1 Win=65535 Len=1448 TSval=2255708013 TSecr=602381411
10 0.000042 10.2.2.1 77.109.175.63 TCP 1514 54124 → 8080 [ACK] Seq=23169 Ack=1 Win=65535 Len=1448 TSval=2255708013 TSecr=602381411
11 0.000043 10.2.2.1 77.109.175.63 TCP 1514 54124 → 8080 [ACK] Seq=24617 Ack=1 Win=65535 Len=1448 TSval=2255708013 TSecr=602381411
12 0.000076 10.2.2.1 77.109.175.63 TCP 1514 54120 → 8080 [ACK] Seq=15929 Ack=1 Win=65535 Len=1448 TSval=2802166125 TSecr=602381411
13 0.000077 10.2.2.1 77.109.175.63 TCP 1514 54120 → 8080 [ACK] Seq=17377 Ack=1 Win=65535 Len=1448 TSval=2802166125 TSecr=602381411
14 0.000077 10.2.2.1 77.109.175.63 TCP 1514 54120 → 8080 [ACK] Seq=18825 Ack=1 Win=65535 Len=1448 TSval=2802166125 TSecr=602381411
15 0.000078 10.2.2.1 77.109.175.63 TCP 1514 54120 → 8080 [ACK] Seq=20273 Ack=1 Win=65535 Len=1448 TSval=2802166125 TSecr=602381411
16 0.000079 10.2.2.1 77.109.175.63 TCP 1514 54120 → 8080 [ACK] Seq=21721 Ack=1 Win=65535 Len=1448 TSval=2802166125 TSecr=602381411
17 0.000079 10.2.2.1 77.109.175.63 TCP 1514 54120 → 8080 [ACK] Seq=23169 Ack=1 Win=65535 Len=1448 TSval=2802166125 TSecr=602381411
18 0.000080 10.2.2.1 77.109.175.63 TCP 1514 54120 → 8080 [ACK] Seq=24617 Ack=1 Win=65535 Len=1448 TSval=2802166125 TSecr=602381411
19 0.000081 10.2.2.1 77.109.175.63 TCP 1514 54120 → 8080 [ACK] Seq=26065 Ack=1 Win=65535 Len=1448 TSval=2802166125 TSecr=602381411
20 0.000182 77.109.175.63 10.2.2.1 TCP 66 8080 → 54123 [ACK] Seq=1 Ack=1 Win=24530 Len=0 TSval=602381411 TSecr=678372975
21 0.000222 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=8689 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
22 0.000223 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=10137 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
23 0.000224 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=11585 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
24 0.000225 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=13033 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
25 0.000225 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=14481 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
26 0.000226 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=15929 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
27 0.000227 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=17377 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
28 0.000227 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=18825 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
29 0.000228 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=20273 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
30 0.000229 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=21721 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
31 0.000229 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=23169 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
32 0.000230 10.2.2.1 77.109.175.63 TCP 1514 54123 → 8080 [ACK] Seq=24617 Ack=1 Win=65535 Len=1448 TSval=678372977 TSecr=602381411
33 0.000263 77.109.175.63 10.2.2.1 TCP 66 8080 → 54123 [ACK] Seq=1 Ack=7241 Win=24576 Len=0 TSval=602381411 TSecr=678372975
Interesting bit here is the difference in package sizes?