Hi:
I am a user of mellanox ethernet card and have two type cards ConnectX-4 LX and ConnectX-3 Pro. And they all running on Redhat Linux.
I get a problem of udp reordering between fragmented and non-fragmented UDP packets. when the packages larger than MTU mixed with packages smaller than MTU, ConnectX-4 LX using mlx5 on receive side’s udp flow will seperate into 2 flows. So I use the below command to solve it.
ethtool -N ethx rx-flow-hash udp4 sd
the command change 4-tuple hash for UDP to 2-tuple and work perfect .Udp traffic between 2 ConnectX-4 LX is ordered no matter fragment or not.
BUT when I use the ConnectX-3 Pro use mlx4 as sender side, the ConnectX-4 LX using mlx5 as receiver become out-of-order. It seem that the command “ethtool -N ethx rx-flow-hash udp4 sd” has no effort for the udp packages sended by mlx4.
Now I got below situation for udp traffic :
CX4 —> CX4 good
CX4 —> other brand card good
other brand card → CX4 good
CX3 —> other brand card good
CX3 —> CX4 OUT-OF-ORDER!
I think the sender side should be no problem, because other brand card can get right package order. Maybe in the module mlx5 some special function is on effort when sender is a mlx4?
So maybe someone can help me. very very think you.
I use the attach c program to test out-of-order:
Usage:
Compile on Linux:
gcc udp_server_seq.c -o udp_server_seq
gcc udp_client_seq.c -o udp_client_seq
Run:
server side:
./udp_server_seq | tee a.txt
client side:
./udp_client_seq serverip 8000
Check:
./ooo2.py a.txt or open a.txt
and the out-of-order ratio is 20% of my test.
udp_server_seq.c (1.74 KB)
udp_client_seq.c (2.38 KB)
ooo2.py (584 Bytes)