I am implementing a Packet Spraying (Load Balancing) solution on BlueField-3 (DOCA 3.3) by matching the RoCEv2 BTH PSN and modifying the UDP Source Port. I use custom header (flex parse) to match PSN.
While the per-packet PSN matching and action (UDP sport modification) work as expected, I have encountered a critical issue where the ICRC becomes invalid (0xFFFFFFFF), leading to packet drops at the receiver.
Key Observations:
Native Path Works: When I remove the doca_flow_custom_header logic and only modify the UDP Source Port using standard matches, the ICRC remains valid and RDMA communication is successful.
Flex Parser Conflict: As soon as I introduce a doca_flow_custom_header_graph_arc to parse the BTH PSN (redirecting UDP to FLEX), the ICRC fails.
Action-Independent: Even if I define the Custom Header for Match-only and perform zero actions on the custom fields , the ICRC is still corrupted to all fs.
Hardware Steering (HWS): The issue persists even if the Pipe and Entry do not explicitly use the custom action. It appears that the mere declaration of a Flex Parser node for RoCE traffic disables the hardware’s native ICRC preservation/re-calculation logic.
Technical Environment:
Hardware: BlueField-3 DPU
DOCA Version: 3.3.0109
Protocol: RoCEv2 (UDP Port 4791)
Goal: Match BTH PSN → Modify UDP Source Port → Maintain valid ICRC.
Questions:
Is there a way to enable ICRC bypass/preservation when using the Flex Parser on RoCEv2 traffic?
Does the BF3 hardware support a “Read-Only” Flex Parser mode that does not break the native RoCE protocol offload path?
Is there a recommended way to match BTH fields without triggering the “Custom Header” logic that seems to invalidate the RoCE hardware context?
How to disable ICRC check of BF3? It seems that there is no rx_icrc_chk_dis field.
Is DPL a better choice to develop a per-packet load balancing method?
With DOCA Flow on BF3 there is an important distinction between thenative RoCE pathand theFlex Parser / custom headerpath. On the native path (UDP/4791 + RoCE BTH), the RoCE engine still owns the packet: it tracks QP/PSN and recomputes/checks ICRC in hardware. That’s why “plain” rules that just touch standard headers (IP/UDP) keep ICRC valid after a UDP sport change.
Once you introduce adoca_flow_custom_header/ Flex Parser arc for the RoCE header, the packet is parsed as UDP + “custom header” instead of as native RoCE. Flex Parser is designed for user-defined protocols (seeDOCA FlowandDOCA Target Architecture), and it doesnotpreserve protocol-specific offloads like RoCE ICRC. So after you modify UDP sport on a packet that has gone through a custom header graph, there is no RoCE engine left to recalculate ICRC – the ICRC field ends up invalid (0xFFFFFFFF) and the remote RNIC drops the packet.
To your specific questions:
1/ ICRC bypass/preservation with Flex Parser: Not currently supported. Once RoCE traffic is steered through a Flex Parser/custom header graph, RoCE ICRC offload is no longer guaranteed.
2/ “Read-only” Flex Parser for RoCE: Not exposed; any custom parse graph for the RoCE header effectively replaces the native RoCE parsing path.
3/ Match BTH fields without breaking RoCE context: With DOCA Flow there is no supported way today to match on RoCE BTH fields (e.g., PSN) and still keep native RoCE offload and ICRC. For PSN-based load balancing you would need to implement it above RoCE (in the RDMA layer / application) or in another programmable data-plane framework, not via DOCA Flow custom header on live RoCE packets.
4/ Disabling ICRC on BF3: There is no public, supported knob in current BF3/CX7 releases to turn off RoCE ICRC checking; disabling ICRC is not supported for RoCE interoperability.
5/ DPL for per-packet load balancing: For more advanced, protocol-aware load-balancing logic, DOCA Target Architecture / DPL is indeed the intended framework. It lets you define custom parsers and actions in P4, but it’s a different programming model from DOCA Flow and there is no ready-made RoCE-PSN load-balancer example today.