HBN 3.3.0 silently drops decapsulated VXLAN frames from non-HBN Linux VTEP peer despite EVPN routes being learned — is this supported?

Question: Is HBN 3.3.0 expected to interoperate with a standard Linux kernel VXLAN + FRR EVPN peer? If yes, what am I missing in the config? If no, is there any documentation stating this limitation?

Setup

  • DPU: 2× BlueField-3
  • HBN version: 3.3.0 (DOCA 3.3, Ubuntu 22.04 BFB)
  • Peer: One standard x86 Linux node running:
    • Kernel VXLAN (netlink, nolearning, EVPN mode)
    • VLAN-aware bridge (vlan_filtering 1)
    • FRR 8.5.3 with bgpd, zebra, staticd
    • Peer is NOT running HBN, just plain Linux
  • All three nodes on the same underlay /24, iBGP AS 65001 full mesh
  • L2 VNI 10002 mapped to VLAN 102 on all three peers

What works

  1. DPU ↔ DPU L2 VNI 10002 traffic — fully functional, hardware-accelerated
  2. BGP EVPN sessions Established between all three peers (2 DPUs + 1 Linux peer)
  3. EVPN Type-2 and Type-3 routes exchanged correctly. From FRR on the Linux peer:
$ sudo vtysh -c "show bgp l2vpn evpn summary"
Neighbor        V   AS   MsgRcvd  MsgSent   State/PfxRcd
<DPU1>          4 65001  ...      ...       Established/4
<DPU2>          4 65001  ...      ...       Established/0
  1. Linux peer’s kernel FDB is correctly populated by FRR from EVPN:
$ bridge fdb show dev vxlan102 | grep extern
<MAC-behind-DPU1> vlan 102 extern_learn master br0
<MAC-behind-DPU1> dst <DPU1-underlay-IP> self extern_learn
  1. Linux peer’s ARP cache is populated by EVPN Type-2 (ARP suppression):
$ ip neigh show dev vlan102
<endpoint-IP> lladdr <MAC-behind-DPU1> extern_learn NOARP proto zebra

What fails

Ping from Linux peer’s SVI to an endpoint behind DPU1 fails 100%.

Wire capture on Linux peer’s underlay — packet is properly encapsulated unicast (thanks to EVPN Type-2 giving MAC→VTEP mapping):

IP <linux-peer>.59873 > <DPU1>.4789: VXLAN, flags [I] (0x08), vni 10002
    IP <linux-peer-svi> > <endpoint-behind-dpu1>: ICMP echo request

Inside HBN container — tcpdump on p0_if and br_default:

p0_if      P    IP <linux-peer>.59873 > <DPU1>.4789: VXLAN vni 10002, ICMP ...
br_default In   IP <linux-peer>.59873 > <DPU1>.4789: VXLAN vni 10002, ICMP ...

Packet arrives at the uplink AND enters the kernel bridge.

Inside HBN container — tcpdump on pf0hpf_if:

(no output — 0 packets)

The decapsulated frame never reaches the host PF representor. It is silently dropped somewhere between br_defaultand pf0hpf_if.

For comparison, DPU1 → local host via pf0hpf_if works normally:

$ tcpdump -i pf0hpf_if -nn arp
ARP Request who-has <endpoint-IP> tell <HBN-SVI-IP>
ARP Reply <endpoint-IP> is-at <MAC-behind-DPU1>

So the pf0hpf_if path is not broken — only traffic decapsulated from the non-HBN VTEP is dropped.

HBN config (relevant excerpts)

nv set bridge domain br_default type vlan-aware
nv set bridge domain br_default vlan 102 vni 10002
nv set interface p0_if bridge domain br_default
nv set interface pf0hpf_if bridge domain br_default access 102
nv set nve vxlan enable on
nv set nve vxlan mac-learning off
nv set nve vxlan source address <DPU1-underlay-IP>
nv set nve vxlan arp-nd-suppress off
nv set evpn enable on
nv set router bgp enable on
nv set router bgp autonomous-system 65001
nv set vrf default router bgp peer-group fabric remote-as 65001
nv set vrf default router bgp peer-group fabric address-family l2vpn-evpn enable on
nv set vrf default router bgp neighbor <DPU2-underlay-IP> peer-group fabric
nv set vrf default router bgp neighbor <linux-peer-underlay-IP> peer-group fabric

Diagnostic evidence

HBN bridge fdb show shows the Linux peer as a static HER destination:

00:00:00:00:00:00 dev vxlan48 dst <linux-peer> src_vni 10002 self permanent

HBN software-tables/17 (nl2docad ASIC bridge table) contains the Linux peer’s SVI MAC:

"dst mac":"<linux-peer-svi-mac>"

So the MAC IS programmed into the ASIC bridge table. But the packet still doesn’t egress pf0hpf_if.

vxlan48 bridge port flags look normal:

71: vxlan48 ... master br_default state forwarding
    learning on flood on mcast_flood on bcast_flood on
    neigh_suppress off

nl2doca daemon is running:

$ supervisorctl status nl2doca
nl2doca   RUNNING   pid <PID>, uptime <T>

Things tried (none worked)

  1. supervisorctl restart nl2doca
  2. Static FDB entry bridge fdb add <linux-peer-svi-mac> dev vxlan48 dst <linux-peer> src_vni 10002 self permanent
  3. Stopping nl2doca entirely (to force software forwarding) — still no packet on pf0hpf_if
  4. Full teardown and rebuild of VLAN 102 config on HBN
  5. Toggling arp-nd-suppress, mac-learning, bridge port flags
  6. Verified peer firewall is not blocking (no drops on inbound path)
  7. Verified no MTU mismatch (all interfaces MTU 9216)

Release notes review

I read the HBN 3.4.0 release notes carefully. None of the following bugs match my symptom:

  • 3865633 (open) — describes drops of host-generated UDP:4789 WITHOUT valid VXLAN encap. My packets ARE properly encapsulated and originate from an external peer.
  • 4214631 (fixed in 3.0.0) — host-side originated 4789 drops in L3-EVPN. Wrong direction.
  • 4264397 (fixed in 3.0.0) — IPv6 NA-with-unicast-MAC not punted. Different protocol (I’m using IPv4).
  • 2821785 (open) — hardware MAC learning limitation. But that’s just a performance note, not a silent drop.
  • 4991261 (3.4.0 known) — stale VNI after L3 VRF change. Not applicable, my VNI is stable.

I could not find any statement in the HBN documentation about whether non-HBN Linux VTEPs are supported peers.

Questions

  1. Is HBN 3.3.0 expected to accept and forward VXLAN traffic from a non-HBN (plain Linux kernel + FRR) VTEP peer?
  2. If yes — what am I missing in the HBN or FRR config?
  3. If no — is this documented anywhere I should have found?
  4. Would upgrading to HBN 3.4.0 change this behavior? (No 3.4.0 bug fix in the release notes seems to target this scenario.)
  5. Is there any punt-path debug I can enable on nl2docad to see WHY the decapsulated frame is being dropped between br_default and pf0hpf_if?

Thanks in advance for any insight. Happy to share full sos reports, pcaps, and hbn-support archives with anyone from the engineering team who wants to look at this.

Update — additional testing with OVS-DOCA + switchdev + DPDK (NVIDIA-native peer)

To rule out the possibility that this is a kernel-VXLAN-specific issue, I rebuilt the Linux peer using a fully NVIDIA-native stack:

  • ConnectX-6 Dx PF in switchdev mode (devlink dev eswitch set pci/... mode switchdev)
  • openvswitch-doca 2.8.0 (0091_24.07_based_3.3.0) with datapath_type=netdev
  • MLNX_DPDK 22.11.2407.1.0 with DOCA 2.8.0082 initialized (hw-offload=true, doca-init=true)
  • OVS bridge with:
    • VF representors (_pf0vf0..3)
    • PF as trunk port carrying VLANs 1600, 2000, 102
    • Internal ports for mgmt / underlay / tenant SVI
    • VXLAN tunnel port (type=vxlan, key=10002, local_ip=<peer-underlay>, remote_ip=flow)

OVS reports offloaded flows correctly (offloaded packets: 41.02% (130237/317514)) and the DPDK PMD is running on a pinned core.

Same failure mode reproduces

Ping from the tenant SVI to the endpoint behind DPU1 still fails 100%. Wire captures on the peer’s underlay confirm VXLAN packets leave the peer.

But critically — on HBN’s p0_if (inside container), tcpdump -i p0_if -nn -e -c 10 vlan 102 sees:

  • No VLAN 102 traffic from our peer at all
  • Only ambient VLAN 3952-3957 broadcasts from other tenants
  • BGP TCP on VLAN 2000 (underlay) works fine

The switchdev-mode peer’s traffic reaches the same fate as the kernel-VXLAN peer’s traffic.

New evidence — MAC mobility FDB corruption on HBN

The most interesting finding. Inside HBN container:

$ bridge fdb show | grep <peer-svi-mac>
<peer-svi-mac> dev vxlan48 vlan 102 extern_learn master br_default
<peer-svi-mac> dev vxlan48 dst <DPU2-underlay> src_vni 10002 self extern_learn

$ bridge fdb show | grep <endpoint-mac-behind-DPU1>
<endpoint-mac> dev vxlan48 vlan 102 extern_learn master br_default
<endpoint-mac> dev vxlan48 dst <DPU2-underlay> src_vni 10002 self extern_learn

Two things wrong here:

  1. The Linux peer’s SVI MAC is learned as being behind DPU2 via VXLAN — but DPU2 doesn’t own that MAC. The peer is a direct EVPN neighbor of BOTH DPUs. DPU1 has cached DPU2’s re-advertisement of the same MAC instead of its own directly-received route from the peer.
  2. **DPU1’s OWN local endpoint MAC (behind pf0hpf_if) is also learned as being behind DPU2 via VXLAN.**This is fundamentally wrong — the endpoint is physically attached to DPU1 via the host PF representor. HBN’s data-plane FDB has been poisoned by DPU2’s EVPN re-advertisement.

When our peer sends VLAN 102 traffic destined to that endpoint MAC:

  • DPU1’s uplink receives it
  • DPU1’s FDB says “that MAC lives via VXLAN to DPU2”
  • DPU1 attempts to re-encapsulate/forward to DPU2 (or drops)
  • Traffic never reaches the local pf0hpf_if → endpoint dies

Additional tests to rule out MAC-related issues

  • Manually set our internal port’s MAC to match the previously-working bond MAC (e8:eb:d3:fd:d1:2cknown to work when the peer used a kernel VLAN sub-interface on a bond) — no improvement
  • Verified switchdev mode is fully engageddpif/show reports netdev@ovs-netdev, PMD threads active, VF representors bound
  • DPU→peer ping also fails — HBN cannot ping the peer’s SVI either, so this is bidirectional
  • BGP EVPN sessions remain Established throughout all tests, prefixes flowing

What now works, what doesn’t (recap after all testing)

Scenario Result
DPU ↔ DPU L2 VNI 10002 Works (baseline)
Peer via kernel VXLAN + kernel bridge + FRR HBN drops decapsulated frames
Peer via native VLAN 102 through TOR trunk (kernel bond VLAN sub-interface, old MAC) Previously worked
Peer via native VLAN 102 through TOR trunk (OVS internal port, new MAC) HBN drops
Peer via OVS-DOCA + DPDK + switchdev + VXLAN Same drops, plus MAC mobility corruption visible
Any direction (peer→DPU or DPU→peer) once peer switched to switchdev Fails

Additional questions

Extending the original questions with what we’ve learned:

  1. Does HBN’s data-plane FDB have a mechanism to prefer directly-received EVPN Type-2 routes over re-advertisements from other VTEPs? The fact that a locally-attached endpoint’s MAC gets overwritten to point at a remote VTEP looks like a fundamental issue.
  2. Is there a documented interop matrix listing which “kinds” of VTEP peers HBN accepts? So far every non-HBN peer implementation we’ve tried (kernel VXLAN, OVS-DPDK-DOCA in switchdev) reproduces the same drops.
  3. Are there specific extended communities or EVPN attributes that HBN’s FDB installation logic requires from a Type-2 advertisement? Our peer advertises RT:65001:10002 ET:8. DPU-to-DPU advertisements carry the same. No Router MAC is advertised in either direction (this is L2 EVPN, not symmetric IRB).
  4. How can we clear a “corrupt” FDB entry on HBN without a full pod restart? supervisorctl restart nl2doca and bridge fdb del do not seem to persist correct state — DPU2’s EVPN re-advertisement immediately re-poisons the entry.

Happy to share full hbn-support archives, sos reports, and pcaps of the failing scenarios with anyone from the engineering team.