TCP BW: large iperf difference between identical IB hosts

We have 4 test servers with ConnectX-6 VPI adapter card, connected via HDR IB (200Gb/s) to a QM8700 HDR switch. This is part of a fabric that also has FDR HCAs and switches.
I’m running iperf3 -s on these servers and use one of the older FDR hosts as client with iperf3 -c IP.
Results in 3.8 Gb/s, 35.5 Gb/s, 36.4 Gb/s, 26.5 Gb/s`, respectively.
In particular the first two values a baffling, because server 1 and server 2 have exactly identical hardware and identical system installations (servers 3 and 4 have more CPU and more RAM, server 4 is running Debian 12, the others Rocky 8.9, all of which might account for slight variations of the result).
However, the factor 9 between server 1 and 2…

Which metric/ configuration option should I look at?

  • Firmware of the HCAs on server 1 and 2 is identical (20.38.1002)
  • ib_read_bw -F -R -a -z between the four servers and the said client gives a roughly constant result: no differences here

A reboot of server 1 leved this out.

Now running iperf3 -c <IP> -P 15 -t 30 I achieve 25Gb/s from my client to servers 1/2/3, and 36 Gb/s to server 4 - perhaps a bonus of Debian? ;-) ;-)

Hello t.roth,

Thank you for posting your inquiry on the NVIDIA Developer Forum - Infrastructure and Networking - Section.

For TCP performance testing, we only recommend to use iperf(2) as you can start it with multiple parallels.

Please review the following URL for more information → ESPCommunity

Thank you and regards,
~NVIDIA Networking Technical Support

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.