MCX354A won't do 56Gbps

Infrastructure & Networking Adapters And Cables
#1

I’ve got a network with several beefy servers and MCX354A NICs on SX6036, all setup for 56GBE. (confirmed by switch link and NIC, cable are MC2207130-002).

However I try though, I can’t get past ~40.5gbps.

MTU is set to 9000 (9014 in switch).

switch
switch970×488 156 KB
nic

speed
speed740×327 8.63 KB

I’ve tried iperf2 & multiple instances of iperf3, the CPUs hardly break a sweat but I can’t get past this speed barrier it seems. According to MLNX_TUNE all looks good, this profile was set;

‘mlnx_tune -p HIGH_THROUGHPUT’

CPU is EPYC 7402P, 4x 32GB 3200MHz.

CPU is set to performance power mode;

root@user:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

performance

#2

Hello Ruben,

Thank you for posting your inquiry on the NVIDIA Networking Community.

Based on the information provided, on the adapter and switch side everything is set correctly.

As this is AMD EPYC platform, you need to implement the recommendations from AMD to achieve full performance.

You can use the following Guides based on your OS or environment you are running → https://developer.amd.com/resources/epyc-resources/epyc-tuning-guides/

Thank you and regards,

~NVIDIA Networking Technical Support