Mellanox MCX4131A-GCAT 1x50G SR-IOV over KVM DPDK testpmd performance issues

Attach the MCX4131A-GCAT 1x50G NIC to the X86 platform, setup SR-IOV to enable the DPDK testpmd testing at GuestOS.

From the testing, found that the 64B can’t achieve the same performance as baremetal environment (30% gap).

Similar results at the ARM platform.

Was it because of the NIC didn’t support the 64B processing at SR-IOV mode?

Is there any special settings should try?

Already set the inte_iommu=on, iommu=pt at Host OS.

Host Kernel: Linux 3.10.0-957.el7.x86_64

QEMU/KVM Hypervisor:

virsh # version

Compiled against library: libvirt 4.5.0

Using library: libvirt 4.5.0

Using API: QEMU 4.5.0

Running hypervisor: QEMU 1.5.3

mlxconfig -d 65:00.0 set NUM_OF_VFS=1 SRIOV_EN=1 CQE_COMPRESSION=1

lspci at Host OS:

65:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

65:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]

Guest OS xml cfg:

PCI passthrough:

Guest OS: Linux vm 3.10.0-957.el7.x86_64

GCC @Guest OS: gcc version 7.3.1 20180303

DPDK @Guest OS: V19.08

testpmd: ./x86_64-native-linuxapp-gcc/app/testpmd -l 1-17 -w 00:06.0 – -i --nb-cores=4 --burst=64 --rss-ip --txq=4 --rxq=4 --mbcache=512 --rxd=4096 --txd=4096

Hi Jerry,

Please refer to the following performance report by Mellanox for DPDK 19.08

http://fast.dpdk.org/doc/perf/DPDK_19_08_Mellanox_NIC_performance_report.pdf

For DPDK 19.05

http://fast.dpdk.org/doc/perf/DPDK_19_05_Mellanox_NIC_performance_report.pdf

There are few examples for ConnectX-4/5.

Thanks,

Samer

Hi, Samer

Thanks for your quick response.

I didn’t see the MCX4131A-GCAT 1x50G SR-IOV over KVM from all the published Mellanox NIC performance reports. Only saw the baremetal test results with 2 NICs at 2x40G mode.

Would like to confirm with Mellanox, does this specific NIC not support 64B well at 1x50G SR-IOV mode?

Thanks,

Jerry

Hi Jerry,

Please review the below Release notes for all supported features by mlx5

http://doc.dpdk.org/guides/rel_notes/release_19_08.html#api-changes

Also in the user manual “SR-IOV is supported”

http://doc.dpdk.org/guides-19.08/nics/mlx5.html

Regarding the NIC review the documentation.

Thanks,

Samer

Hi, Samer

Thanks for providing the inputs.

Yes. This NIC card does support the SR-IOV feature.

My question here is I have tried all the performance optimizations the user manual has listed.

32.10. Performance tuning

http://doc.dpdk.org/guides-19.08/nics/mlx5.html

But the SR-IOV performance throughput (64B) dropped 30% comparing with baremetal performance.

Is there any more settings I could try?

Thanks,

Jerry

Hi Jerry,

I see that you have a valid support contract with Mellanox

Please open a support case for deep debug at support@mellanox.com.

Thanks,

Samer

Hi, Samer

Thanks a lot!

Thanks,

Jerry