100Gbit slow on Epyc HW.

we recognize slow speed on an AMD H11SSL-i single socked Boards with the latest Bios 1.0C

Setup

Two servers connected over a Mellanox SN2100 switch.MSN2100

Bios settings

C-State disabled

IO-SVR disabled

SMV enabled

Perforamce mode enabled

On this setup, we run an Ubuntu Kernel 4.15.0-37.40,

but we recognize this problem also with the 4.19.1 mainline kernel

and an old ubuntu 4.4 Kernel.

We use to try two different kinds of nic

First the Mellanox MCX556A-ECAT

Second the Mellanox MCX456A-ECAT

The problem is with an iperf3 parallel benchmark test

we only reach 60GBit on the MCX556A-ECAT and 50GBit on the MCX456A-ECAT.

With an Intel Broadwell (SuperMicro X10SRi-F Board), the iperf3 test shows results near 90 GBit what is ok with Spectre and Meltdown mediation.

Has anyone an Idea what could be the reason?

Hello,

I would like to suggest reviewing the following community page:

https://community.mellanox.com/s/article/performance-tuning-for-mellanox-adapters

Also, would like to suggest using iperf2 instead of iperf3 and use more threads by adding the -P flag.

If the performance will still be lower, please open a case with Mellanox support for further investigation by sending an email to support@mellanox.com mailto:support@mellanox.com .

Best Regards,

Viki