I'm getting low throughput with MPLS tagged packets. Is there something that can be tuned to improve performance?

In this test setup I have a couple of VMs running iperf tests and the traffic between them is passed through another two VMs/virtual routers. All are running on seperate KVM hosts.

The virtual routers have two SR-IOV VFs each. The hosts all have ConnectX-5 25Gbps adapters.

VM1<–>Virtual router 1<–>Virtual router 2<–>VM2

With no MPLS I’m getting decent throughput through this topology:

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-10.0 sec 3.35 GBytes 2.88 Gbits/sec

[ 8] 0.0-10.0 sec 3.76 GBytes 3.23 Gbits/sec

[ 4] 0.0-10.0 sec 3.84 GBytes 3.30 Gbits/sec

[ 7] 0.0-10.0 sec 3.94 GBytes 3.38 Gbits/sec

[ 5] 0.0-10.0 sec 1.34 GBytes 1.15 Gbits/sec

[ 6] 0.0-10.0 sec 3.94 GBytes 3.38 Gbits/sec

[ 9] 0.0-10.0 sec 3.25 GBytes 2.79 Gbits/sec

[ 10] 0.0-10.0 sec 3.82 GBytes 3.29 Gbits/sec

[SUM] 0.0-10.0 sec 27.2 GBytes 23.4 Gbits/sec

But if the the packets between the virtual routers are MPLS tagged, the throughput is only about 10-20%:

[ ID] Interval Transfer Bandwidth

[ 16] 0.0-10.0 sec 313 MBytes 263 Mbits/sec

[ 4] 0.0-10.0 sec 433 MBytes 363 Mbits/sec

[ 3] 0.0-10.0 sec 413 MBytes 346 Mbits/sec

[ 9] 0.0-10.0 sec 341 MBytes 286 Mbits/sec

[ 7] 0.0-10.0 sec 415 MBytes 348 Mbits/sec

[ 6] 0.0-10.0 sec 418 MBytes 351 Mbits/sec

[ 5] 0.0-10.0 sec 419 MBytes 351 Mbits/sec

[ 10] 0.0-10.0 sec 431 MBytes 361 Mbits/sec

[SUM] 0.0-10.0 sec 3.11 GBytes 2.67 Gbits/sec

I have tried running different software for the virtual routers. Both Ubuntu 20.04 and Nokia vSR. In both cases the performance seems to be the same.

What Mellanox driver version are you using?

If you are using Mellanox driver, what is the throughput with Inbox driver? (from native OS)

What is the Kernel version?

What is the current FW of the ConnectX5 25Gbs HCA card?

What type of switch(s) these KVM servers connect to?

Are you using TCP/UDP?

What is your Packet layout?

What is the MTU?

What is the packet size(s)?

Note: In our MLNX_EN version 5.x we added the following feature:

https://docs.mellanox.com/display/MLNXEN501000/Changes+and+New+Features

MPLS TrafficAdded support for reporting TSO and CSUM offload capabilities for MPLS tagged traffic and, allowed the kernel stack to use these offloads.

you can validate offloads parameters via the ethtool -k

Sophie.

We have tried this with the inbox driver that comes with Ubuntu. Version 5.0. We also tried a newer Mellanox driver (5.1-1.0.4)

The Kernel version on the host is 5.4.73. On the Ubuntu guest it is 5.4.0.

The firmware version is firmware version: 16.28.2006

All the tests are run from machines with 1500 MTU using TCP. The packet size is 1470.

I don’t see the offload parameters mentioned in ethtool. Here is the output of ethtool -k on one of the hosts:

Features for ens10f0np0:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: off [fixed]

tx-checksum-ip-generic: on

tx-checksum-ipv6: off [fixed]

tx-checksum-fcoe-crc: off [fixed]

tx-checksum-sctp: off [fixed]

scatter-gather: on

tx-scatter-gather: on

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

tx-tcp-segmentation: on

tx-tcp-ecn-segmentation: off [fixed]

tx-tcp-mangleid-segmentation: off

tx-tcp6-segmentation: on

udp-fragmentation-offload: off

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: on

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: on

tx-gre-csum-segmentation: on

tx-ipxip4-segmentation: on

tx-ipxip6-segmentation: on

tx-udp_tnl-segmentation: on

tx-udp_tnl-csum-segmentation: on

tx-gso-partial: on

tx-sctp-segmentation: off [fixed]

tx-esp-segmentation: off [fixed]

tx-udp-segmentation: on

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off

rx-all: off

tx-vlan-stag-hw-insert: on

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: on [fixed]

l2-fwd-offload: off [fixed]

hw-tc-offload: off

esp-hw-offload: off [fixed]

esp-tx-csum-hw-offload: off [fixed]

rx-udp_tunnel-port-offload: on

tls-hw-tx-offload: off [fixed]

tls-hw-rx-offload: off [fixed]

rx-gro-hw: off [fixed]

tls-hw-record: off [fixed]

I forgot to mention. The switches in this scenario are HPE (Mellanox) SN2100M