I think the transmission algorithms you set using mlnx_qos include vendor, strict, and ets.
The following URL explains strict and ets, but there is no information about vendor.
Could you please let me know if there are any documents that describe the transmission algorithm vendor?
(The adapter used is connectx6 Dx.)
・NVIDIA MLNX_OFED Documentation v24.10-0.7.0.0
Regarding the transmission algorithm, we conducted the following additional research.
[Settings]
・RDMA (priority 0: dscp0) ⇒ Use TC1, transmission algorithm: vendor
・TCP (priority 5: dscp46) ⇒Use TC5, transmission algorithm: vendor
If you directly connect the sending PC and the receiving PC and simultaneously perform RDMA transfer with qperf and TCP transfer with iperf from the sending PC, you will get the following results.
*Link speed is 40Gbps. RDMA and TCP alone can send at about 39Gbps each.
・RDMA: 17.25Gbps
・TCP: 22.4Gbps
If you use UDP instead of TCP, the results are as follows, with RDMA and UDP having the same bandwidth (RDMA is a UDP protocol, so it appears that the same protocol has the same bandwidth).
・RDMA: 19.8Gbps
・UDP: 19.52Gbps
Also, if you set the transmission algorithm in the above [Settings] to ets and set the BW to 50% for both TC1 and TC5, the results are as follows.
・RDMA: 15.8Gbps
・TCP: 22.9Gbps
Question
Do you know why TCP has a higher bandwidth when setting the transmission algorithm vendor and ets?
Are there any processes that give priority to TCP?
※ The same result will be obtained if you just switch the RDMA and TCP priorities.
・Change RDMA priority from 0 to 5
・Change TCP priority from 5 to 0
Hi Satakahi,
Thank you for posting your query on NVIDIA community!
“Vendor” is similar as “strict”, the only difference between “vendor” and “strict” is on the priority:
When setting multiple TCs’ transmission algorithm to be “strict”, there will be absolute (strict) priority determined by TC number among these TCs, e.g. TC 7 is the highest priority, TC 0 is lowest.
The default transmission algorithm for all TCs is “vendor”, where all TCs share the same priority, e.g. there is no priority difference between TC 7 and TC 0.
Here is the driver source code on how TC group is built for priorities, it shows difference between vendor/strict/ets:
drivers/net/ethernet/mellanox/mlx5/core/en_dcbnl.c::mlx5e_build_tc_group()
Thanks,
Namrata.
Thank you for your reply.
I understand that there is no difference in priority for “vendor”.
However, I don’t know why TCP seems to be given priority when set to “vendor” as posted on 5/20. Could you please tell me what you think about this?