BlueField-3 DPU: DOCA Flow + DOCA Eth Rx Fails to Receive RoCEv2 UD Packets via SF

Infrastructure & Networking BlueField
1

Dear NVIDIA Forum/Support Team,

I am developing a receiver application on a BlueField-3 DPU using DOCA Flow for hardware offload and DOCA Eth Rx for packet reception. The application runs on a Scalable Function (SF) port, specifically en3f0pf0sf3.

We are encountering a critical issue where RoCEv2 Unreliable Datagram(UD) packets are lost when they are routed through the internal Open vSwitch (OVS) bridge on the DPU, while standard IP traffic (ICMP, TCP, UDP) is successfully received.

We suspect this issue is related to the specific interaction between RoCEv2 UD traffic, OVS forwarding, and the DOCA Flow hardware offload path to the SF.

Environment & Configuration

  • DPU Hardware: BlueField-3 (BF3) DPU (P/N: 900-9D3B6-00CV-A_Ax)
  • Firmware Version: 32.46.1006
  • OS/Kernel: Ubuntu 22.04.5 LTS (Kernel: 5.15.0-1074-bluefield)
  • Used DOCA libs: DOCA Flow (Flow setup) + DOCA Eth Rx (Reception)
  • RoCEv2 Type: UD Send

Network topology & OVS Configuration🌐 Network Topology & OVS Setup

Traffic is routed internally on the DPU using the br0 OVS bridge:

External Host -> p0(Physical port) -> br0(OVS bridge) -> en3f0pf0sf3(Scalable Function) -> My DOCA Eth Rx App

OVS configratuion

$ ovs-vsctl show
dc0aa193-0097-4026-aa9e-61000ad6fdbe
    Bridge br0
        Port br0
            Interface br0
                type: internal
        Port p0
            Interface p0
        Port en3f0pf0sf3
            Interface en3f0pf0sf3

Status

  • Working case

When the application receives packets directly from p0 (bypassing the bridge/SF), all traffic types (ICMP, TCP, UDP, RoCEv2 UD) are received successfully.

  • Fail case

When receiving from en3f0pf0sf3 via the p0 -> br0 -> sf path, ICMP, TCP, and UDP are received, but RoCEv2 UD packets are consistently lost.

Question

Is the loss of RoCEv2 packets when using DOCA Eth Rx on a Scalable Function (SF) port an intended limitation (by design)?

If not, what specific configurations are required?

Thank you for your time and assistance in resolving this issue.

2

Hello user8937,

Thanks for reaching out. From your description, I think this can be expected because with SF and RoCE, the RoCE traffic is offloaded and wont be handled by the Ethernet driver to DOCA Eth Rx (Ethernet RxQ).

From the RDMA over Converged Ethernet (RoCE) documentation:
https://docs.nvidia.com/doca/archive/2-9-4/rdma-over-converged-ethernet-(roce)/index.html

“RoCE traffic is not shown in the associated Ethernet device’s counters since it is offloaded by the hardware and does not go through Ethernet network driver.”

and:

“The mlx5_ib driver holds a reference to the net device … However, RoCE traffic does not go through the mlx5_core driver; it is completely offloaded by the hardware.”

For SF to receive RoCE traffic, you will probably need to work with DOCA RDMA (instead of DOCA Eth Rx).

Alternatively you could configure RoCEv2 traffic to behave normal. Eg.

https://docs.nvidia.com/doca/archive/2-10-0/virtual+switch+on+bluefield/index.html

“Configure RoCEv2 packets to behave normally.
RoCEv2 packets follow UDP port 4791 and a different source port in each direction of the connection.
RoCE traffic is not supported by CT.
In order to run RoCE from the host add the following line …
ovs-ofctl add-flow ctBr table=0,udp,tp_dst=4791,action=normal
This rule allows RoCEv2 UDP packets to skip connection tracking rules.”

Hope this helps.