Hello,
I am using the BlueField-2 DPU to perform an experiment to use OVS bridges to send packets to a DOCA Flow app in VNF mode which forwards packets to another OVS bridge. I’d expect this to give me a bandwidth at line-rate (100Gbps), but I am only getting at about 80+Gbps. I wonder what’s the reason behind the bandwidth degradation?
The setup: I have two hosts each equipped with a BlueField-2 SmartNIC and the uplink port p0 of the two DPUs are connected to a switch. Then I run pktgen on the two hosts to test the network bandwidth. The following picture shows the topology:
The VNF mode DOCA Flow app is not performing any additional actions; it simply creates two pipes that forward traffic from one port to the other.
DOCA version on both SmartNICs:
[main] release build
DOCA SDK Version (Compilation): 3.2.0118
DOCA Runtime Version (Runtime): 3.2.0118
ARM OS:
5.15.0-1081-bluefield #83-Ubuntu SMP Fri Oct 17 10:11:15 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
BlueField:
$ sudo mlxfwmanager --query
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: BlueField2
Part Number: MBF2M516C-CECO_Ax
Description: BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL
PSID: MT_0000000733
PCI Device Name: /dev/mst/mt41686_pciconf0
Base GUID: (removed for privacy)
Base MAC: (removed for privacy)
Versions: Current Available
FW 24.46.1006 N/A
PXE 3.8.0100 N/A
UEFI 14.39.0013 N/A
UEFI Virtio blk 22.4.0014 N/A
UEFI Virtio net 21.4.0013 N/A
The OVS bridges are configured the same way on the both SmartNICs:
$ sudo ovs-vsctl show
Bridge br0
Port br0
Interface br0
type: internal
Port pf0hpf
Interface pf0hpf
Port en3f0pf0sf0
Interface en3f0pf0sf0
Bridge br1
Port p0
Interface p0
Port en3f0pf0sf1
Interface en3f0pf0sf1
Port br1
Interface br1
type: internal
ovs_version: "3.2.0044"
$ sudo ovs-vsctl get Open_vSwitch . other_config:hw-offload
"true"
The same DOCA Flow app is run with the same command on both SmartNICs:
$ sudo ./simple_fwd -- -a aux/2,dv_flow_en=2 -a aux/3,dv_flow_en=2 -l 60
Then, on the hosts, I run pktgen with the same command:
$ sudo ./pktgen -l 16-31 -n 8 -a 0000:81:00.0 -- -T -j --rxd=2048 --txd=2048 -m "[17:18].0"
On Host 1, in pktgen:
set 0 size 1500
set 0 proto tcp
set 0 dst mac (Host 2's MAC)
set 0 dst ip (Host 2's IP)
set 0 src ip (Host 1's IP)/32
set 0 rate 100
start 0
Then, pktgen on Host 1 reports it is sending traffic at about 98Gbps. On Host 2, pktgen reports it is receiving traffic at about 83Gbps.
For comparison, I removed my DOCA Flow app from the datapath by connecting pf0hpf and p0 directly using an OVS bridge on both SmartNICs and rerun the experiment with the same steps, and now on Host 1/2 pktgen sends/receives traffic at 99.9Gbps.
$ sudo ovs-vsctl show
Bridge br0
Port br0
Interface br0
type: internal
Port pf0hpf
Interface pf0hpf
Port p0
Interface p0
ovs_version: "3.2.0044"
For another comparison, I implemented a DOCA Flow app in switch mode and got rid of the OVS bridges completely, directly bridging pf0hpf and p0 representors using my DOCA Flow app. This app also simply forwards traffic from one port to the other without doing any additional actions. I rerun the experiment with the same steps and pktgen on Host 1/2 also can send/receive traffic at 99.9Gbps.
This indicates that there’s something going on then I combine OVS bridges with a VNF mode DOCA Flow app. I believe this “bump-in-the-wire” is a standard practice, so my question is: is this bandwidth drop expected? If so, what’s the reason behind this? Is it because the packets are cycling back-and-forth between the eSwitch and the hardware accelerators? If not so, did I make any misconfigurations? What can I do to debug this issue?
Any suggestions would be greatly appreciated! And I am more than happy to provide further information or source code.
Thank you very much!