In this test setup I have a couple of VMs running iperf tests and the traffic between them is passed through another two VMs/virtual routers. All are running on seperate KVM hosts.
The virtual routers have two SR-IOV VFs each. The hosts all have ConnectX-5 25Gbps adapters.
VM1<–>Virtual router 1<–>Virtual router 2<–>VM2
With no MPLS I’m getting decent throughput through this topology:
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.35 GBytes 2.88 Gbits/sec
[ 8] 0.0-10.0 sec 3.76 GBytes 3.23 Gbits/sec
[ 4] 0.0-10.0 sec 3.84 GBytes 3.30 Gbits/sec
[ 7] 0.0-10.0 sec 3.94 GBytes 3.38 Gbits/sec
[ 5] 0.0-10.0 sec 1.34 GBytes 1.15 Gbits/sec
[ 6] 0.0-10.0 sec 3.94 GBytes 3.38 Gbits/sec
[ 9] 0.0-10.0 sec 3.25 GBytes 2.79 Gbits/sec
[ 10] 0.0-10.0 sec 3.82 GBytes 3.29 Gbits/sec
[SUM] 0.0-10.0 sec 27.2 GBytes 23.4 Gbits/sec
But if the the packets between the virtual routers are MPLS tagged, the throughput is only about 10-20%:
[ ID] Interval Transfer Bandwidth
[ 16] 0.0-10.0 sec 313 MBytes 263 Mbits/sec
[ 4] 0.0-10.0 sec 433 MBytes 363 Mbits/sec
[ 3] 0.0-10.0 sec 413 MBytes 346 Mbits/sec
[ 9] 0.0-10.0 sec 341 MBytes 286 Mbits/sec
[ 7] 0.0-10.0 sec 415 MBytes 348 Mbits/sec
[ 6] 0.0-10.0 sec 418 MBytes 351 Mbits/sec
[ 5] 0.0-10.0 sec 419 MBytes 351 Mbits/sec
[ 10] 0.0-10.0 sec 431 MBytes 361 Mbits/sec
[SUM] 0.0-10.0 sec 3.11 GBytes 2.67 Gbits/sec
I have tried running different software for the virtual routers. Both Ubuntu 20.04 and Nokia vSR. In both cases the performance seems to be the same.