Product / Severity
-
Product: DOCA-OVS (OVS-DOCA), ConnectX-7
-
Severity: High (hardware offload non-functional for the entire deployment role)
Environment
| Item | Value |
|---|---|
| NIC | ConnectX-7, MCX755106AS-HEA_Ax, PSID MT_0000000834 |
| Firmware | 28.47.2682 |
| DOCA | doca-host 3.3.0109 (RELEASE 26.01-based-3.3.6) |
| OVS | ovs-vswitchd (Open vSwitch) 3.3.0040, datapath_type=doca, DPDK 25.11.0+doca2601 |
| OS / kernel | Rocky Linux 9.7, kernel 5.14.0-611.55.1.el9_7.x86_64 |
| Bonding | Two CX7 ports (38:00.0 / 38:00.1) in kernel LACP bond (mode 802.3ad), eswitch switchdev, hardware LAG; OVS port bond0 is type=doca attached to the bond |
| Topology | OpenStack/OVN gateway node. All traffic is wire-to-wire hairpin on the same LAG: RX on VLAN 81 (external) → route → Geneve encap (with one OVN TLV option, class=0x102, type=0x80, 4-byte data) → push VLAN 71 (underlay) → TX out the same bond. Reverse direction: Geneve+option decap → push VLAN 81 → TX same bond |
| other_config | doca-init=true, hw-offload=true, pmd-cpu-mask=0xf0 (4 PMDs), vlan-limit=0 |
| FLEX_PARSER_PROFILE_ENABLE | reproduced with both 0 and 8 (cold reboot between) |
Problem statement
For a plain routed flow with no conntrack and no NAT (single flat megaflow:
pop_vlan → set_mac → dec_ttl → tnl_push(geneve+option) → push_vlan → output),
OVS reports the flow as fully offloaded (offloaded:yes, dp:doca), and
ovs-appctl doca/dump-offloads shows a complete HW rule including the
62-byte precompiled raw_encap header and the final forward action.
However, every single data packet still arrives at the PMD. Packet captures
via ovs-appctl dpif-doca/dump-packets on show that each packet reaching the
PMD is wrapped in an internal VXLAN encapsulation with VNI=0xdead, carries
non-zero flow metadata written by the HW pipeline (e.g. meta=0x00002000), and
is immediately logged as post-hw-recover by netdev_offload_doca. I.e. the
packet hit the HW rules (metadata is set), but HW did not complete
forwarding and punted it to SW, which then executes the full action list
(~1000–1100 cycles/packet measured).
Effective result: OVS-DOCA on this node behaves as software forwarding for
100% of traffic. Throughput is identical to a kernel-OVS gateway on the same
path (~17.3 Gbps, limited elsewhere), with roughly 2x more TCP retransmissions
in the encap direction.
Evidence
1. HW rule is complete (doca/dump-offloads, offload-trace enabled)
flow create 0 user_id ... priority 1 group 0x0 transfer
pattern port_id id is 0 physdev=1 / tag index 3 spec 0x0 /
eth src is 72:3a:c0:51:04:65 dst is fa:16:3e:04:af:83 type is 0x8100 has_vlan is 1 /
vlan inner_type 0x800 tci spec 0x51 tci mask 0xefff /
ipv4 src spec 10.199.81.245 mask /30 dst is 192.168.150.243 proto 6 ttl is 0x40 / tcp
actions count / of_pop_vlan / set_mac_src fa:16:3e:74:03:65 / set_mac_dst fa:16:3e:92:13:43 /
set_ipv4_ttl 63 / raw_encap index 0 (62B: eth+ipv4+udp(6081)+geneve(vni,1 TLV opt)+...) /
represented_port ethdev_port_id 0
Reverse (decap) rule equally complete: geneve+geneve-opt match → raw_decap →
of_push_vlan(81) → represented_port 0.
2. Every packet is punted via VNI=0xdead and recovered in SW
dpif_doca(pmd-c05)|INFO|port_no=3: in_port=3, recirc_id=0, meta=0x00002000, size=1568.
outer: eth(zeros)/ipv4(src=0.0.0.0,dst=0.0.0.0,ttl=0)/udp(dst=4789)/vxlan(VNI=0xdead)
inner: original frame (VLAN 81, 10.199.81.245 -> 192.168.150.243, TCP 5201), 1518B
dpif_doca(pmd-c05)|INFO|post-hw-recover: port_no=3: ... size=1518. (original frame)
~450k post-hw-recover log lines accumulate per few minutes of a single
iperf3 stream. Both encap and decap directions punt.
3. PMD perf counters — HW classification works, forwarding does not
During a 60 s single-stream run (dpif-doca/pmd-perf-show, core carrying the flow):
Rx packets: 51,328,414 (989 cycles/pkt)
Datapath passes: 51,328,744 (1.00/pkt)
- PHWOL hits: 0
- EMC hits: 16,662 (0.0%)
- Megaflow hits: 2,538,597 (4.9%)
-> ~95% of packets resolved via HW flow mark (skip SW classifier)
Tx packets: 24,365 (TX side completed without PMD Tx involvement)
4. Also reproduced / already ruled out
-
FLEX_PARSER_PROFILE_ENABLE0 and 8 (cold reboot): identical punt behavior. -
Not CT-related: the flat flow above has no ct() at all. (Separately, OVN FIP
NAT chains — ct(zone13,nat)→ct(commit,zone14,nat)→ct(zone13,nat) — never get
HW CT connections, consistent with documented CT-CT-NAT unsupported.) -
Not
check_pkt_len-related: OVNgateway_mtuwas removed for the test
router, the megaflow contains no check_pkt_len, and the rule does get
inserted into HW — insertion is not the problem, execution is. -
Rules are hit: punted packets carry HW-written metadata (
meta=0x2000/0x3000),
which a HW-miss packet would not have. -
Port capabilities report
tx_geneve_tso_offload=false,
tx_out_ip_csum_offload=true,tx_out_udp_csum_offload=false.
Questions
-
Is wire-to-wire (hairpin) forwarding with
raw_encap/raw_decapof
Geneve (carrying one TLV option) expected to be executed fully in eswitch
HW on ConnectX-7 in OVS-DOCA 3.3, when the underlying port is a kernel
LACP bond (hardware LAG) and both inner directions involve VLAN
push/pop on the wire side? -
What exactly does the VNI=0xdead recovery path indicate — which action in
our rule fails HW execution and triggers the punt? Is there a counter or
debug knob that reports the punt reason? -
Is this a known limitation or a known issue in DOCA 3.3.0109
(possibly related to Issue 4943253)? Is it fixed in DOCA 3.4 or later? -
If this combination (LAG + VLAN underlay + Geneve w/ TLV option + hairpin)
is outside the supported offload matrix for gateway-style nodes, please
point us to the authoritative support matrix statement.
Attachments (available on request)
-
Full
ovs-appctl doca/dump-offloadsoutput (offload-trace enabled) -
ovs-vswitchd.logsegment with dump-packets + post-hw-recover entries -
ovs-appctl dpif-doca/pmd-perf-showfull output for all 4 PMDs -
ovs-vsctl show,ovs-vsctl list Open_vSwitch, bond/interface configs -
mlxconfig -d 38:00.0 qfull query,mlxfwmanager --query -
OVN NB/SB relevant logical router / LRP configuration