HBN 3.3.0 silently drops decapsulated VXLAN frames from non-HBN Linux VTEP peer despite EVPN routes being learned — is this supported?

Infrastructure & Networking BlueField
2

Update — additional testing with OVS-DOCA + switchdev + DPDK (NVIDIA-native peer)

To rule out the possibility that this is a kernel-VXLAN-specific issue, I rebuilt the Linux peer using a fully NVIDIA-native stack:

  • ConnectX-6 Dx PF in switchdev mode (devlink dev eswitch set pci/... mode switchdev)
  • openvswitch-doca 2.8.0 (0091_24.07_based_3.3.0) with datapath_type=netdev
  • MLNX_DPDK 22.11.2407.1.0 with DOCA 2.8.0082 initialized (hw-offload=true, doca-init=true)
  • OVS bridge with:
    • VF representors (_pf0vf0..3)
    • PF as trunk port carrying VLANs 1600, 2000, 102
    • Internal ports for mgmt / underlay / tenant SVI
    • VXLAN tunnel port (type=vxlan, key=10002, local_ip=<peer-underlay>, remote_ip=flow)

OVS reports offloaded flows correctly (offloaded packets: 41.02% (130237/317514)) and the DPDK PMD is running on a pinned core.

Same failure mode reproduces

Ping from the tenant SVI to the endpoint behind DPU1 still fails 100%. Wire captures on the peer’s underlay confirm VXLAN packets leave the peer.

But critically — on HBN’s p0_if (inside container), tcpdump -i p0_if -nn -e -c 10 vlan 102 sees:

  • No VLAN 102 traffic from our peer at all
  • Only ambient VLAN 3952-3957 broadcasts from other tenants
  • BGP TCP on VLAN 2000 (underlay) works fine

The switchdev-mode peer’s traffic reaches the same fate as the kernel-VXLAN peer’s traffic.

New evidence — MAC mobility FDB corruption on HBN

The most interesting finding. Inside HBN container:

$ bridge fdb show | grep <peer-svi-mac>
<peer-svi-mac> dev vxlan48 vlan 102 extern_learn master br_default
<peer-svi-mac> dev vxlan48 dst <DPU2-underlay> src_vni 10002 self extern_learn

$ bridge fdb show | grep <endpoint-mac-behind-DPU1>
<endpoint-mac> dev vxlan48 vlan 102 extern_learn master br_default
<endpoint-mac> dev vxlan48 dst <DPU2-underlay> src_vni 10002 self extern_learn

Two things wrong here:

  1. The Linux peer’s SVI MAC is learned as being behind DPU2 via VXLAN — but DPU2 doesn’t own that MAC. The peer is a direct EVPN neighbor of BOTH DPUs. DPU1 has cached DPU2’s re-advertisement of the same MAC instead of its own directly-received route from the peer.
  2. **DPU1’s OWN local endpoint MAC (behind pf0hpf_if) is also learned as being behind DPU2 via VXLAN.**This is fundamentally wrong — the endpoint is physically attached to DPU1 via the host PF representor. HBN’s data-plane FDB has been poisoned by DPU2’s EVPN re-advertisement.

When our peer sends VLAN 102 traffic destined to that endpoint MAC:

  • DPU1’s uplink receives it
  • DPU1’s FDB says “that MAC lives via VXLAN to DPU2”
  • DPU1 attempts to re-encapsulate/forward to DPU2 (or drops)
  • Traffic never reaches the local pf0hpf_if → endpoint dies

Additional tests to rule out MAC-related issues

  • Manually set our internal port’s MAC to match the previously-working bond MAC (e8:eb:d3:fd:d1:2cknown to work when the peer used a kernel VLAN sub-interface on a bond) — no improvement
  • Verified switchdev mode is fully engaged — dpif/show reports netdev@ovs-netdev, PMD threads active, VF representors bound
  • DPU→peer ping also fails — HBN cannot ping the peer’s SVI either, so this is bidirectional
  • BGP EVPN sessions remain Established throughout all tests, prefixes flowing

What now works, what doesn’t (recap after all testing)

Scenario Result
DPU ↔ DPU L2 VNI 10002 Works (baseline)
Peer via kernel VXLAN + kernel bridge + FRR HBN drops decapsulated frames
Peer via native VLAN 102 through TOR trunk (kernel bond VLAN sub-interface, old MAC) Previously worked
Peer via native VLAN 102 through TOR trunk (OVS internal port, new MAC) HBN drops
Peer via OVS-DOCA + DPDK + switchdev + VXLAN Same drops, plus MAC mobility corruption visible
Any direction (peer→DPU or DPU→peer) once peer switched to switchdev Fails

Additional questions

Extending the original questions with what we’ve learned:

  1. Does HBN’s data-plane FDB have a mechanism to prefer directly-received EVPN Type-2 routes over re-advertisements from other VTEPs? The fact that a locally-attached endpoint’s MAC gets overwritten to point at a remote VTEP looks like a fundamental issue.
  2. Is there a documented interop matrix listing which “kinds” of VTEP peers HBN accepts? So far every non-HBN peer implementation we’ve tried (kernel VXLAN, OVS-DPDK-DOCA in switchdev) reproduces the same drops.
  3. Are there specific extended communities or EVPN attributes that HBN’s FDB installation logic requires from a Type-2 advertisement? Our peer advertises RT:65001:10002 ET:8. DPU-to-DPU advertisements carry the same. No Router MAC is advertised in either direction (this is L2 EVPN, not symmetric IRB).
  4. How can we clear a “corrupt” FDB entry on HBN without a full pod restart? supervisorctl restart nl2doca and bridge fdb del do not seem to persist correct state — DPU2’s EVPN re-advertisement immediately re-poisons the entry.

Happy to share full hbn-support archives, sos reports, and pcaps of the failing scenarios with anyone from the engineering team.