Question: Is HBN 3.3.0 expected to interoperate with a standard Linux kernel VXLAN + FRR EVPN peer? If yes, what am I missing in the config? If no, is there any documentation stating this limitation?
Setup
- DPU: 2× BlueField-3
- HBN version: 3.3.0 (DOCA 3.3, Ubuntu 22.04 BFB)
- Peer: One standard x86 Linux node running:
- Kernel VXLAN (netlink,
nolearning, EVPN mode) - VLAN-aware bridge (
vlan_filtering 1) - FRR 8.5.3 with
bgpd,zebra,staticd - Peer is NOT running HBN, just plain Linux
- Kernel VXLAN (netlink,
- All three nodes on the same underlay /24, iBGP AS 65001 full mesh
- L2 VNI 10002 mapped to VLAN 102 on all three peers
What works
- DPU ↔ DPU L2 VNI 10002 traffic — fully functional, hardware-accelerated
- BGP EVPN sessions Established between all three peers (2 DPUs + 1 Linux peer)
- EVPN Type-2 and Type-3 routes exchanged correctly. From FRR on the Linux peer:
$ sudo vtysh -c "show bgp l2vpn evpn summary"
Neighbor V AS MsgRcvd MsgSent State/PfxRcd
<DPU1> 4 65001 ... ... Established/4
<DPU2> 4 65001 ... ... Established/0
- Linux peer’s kernel FDB is correctly populated by FRR from EVPN:
$ bridge fdb show dev vxlan102 | grep extern
<MAC-behind-DPU1> vlan 102 extern_learn master br0
<MAC-behind-DPU1> dst <DPU1-underlay-IP> self extern_learn
- Linux peer’s ARP cache is populated by EVPN Type-2 (ARP suppression):
$ ip neigh show dev vlan102
<endpoint-IP> lladdr <MAC-behind-DPU1> extern_learn NOARP proto zebra
What fails
Ping from Linux peer’s SVI to an endpoint behind DPU1 fails 100%.
Wire capture on Linux peer’s underlay — packet is properly encapsulated unicast (thanks to EVPN Type-2 giving MAC→VTEP mapping):
IP <linux-peer>.59873 > <DPU1>.4789: VXLAN, flags [I] (0x08), vni 10002
IP <linux-peer-svi> > <endpoint-behind-dpu1>: ICMP echo request
Inside HBN container — tcpdump on p0_if and br_default:
p0_if P IP <linux-peer>.59873 > <DPU1>.4789: VXLAN vni 10002, ICMP ...
br_default In IP <linux-peer>.59873 > <DPU1>.4789: VXLAN vni 10002, ICMP ...
Packet arrives at the uplink AND enters the kernel bridge.
Inside HBN container — tcpdump on pf0hpf_if:
(no output — 0 packets)
The decapsulated frame never reaches the host PF representor. It is silently dropped somewhere between br_defaultand pf0hpf_if.
For comparison, DPU1 → local host via pf0hpf_if works normally:
$ tcpdump -i pf0hpf_if -nn arp
ARP Request who-has <endpoint-IP> tell <HBN-SVI-IP>
ARP Reply <endpoint-IP> is-at <MAC-behind-DPU1>
So the pf0hpf_if path is not broken — only traffic decapsulated from the non-HBN VTEP is dropped.
HBN config (relevant excerpts)
nv set bridge domain br_default type vlan-aware
nv set bridge domain br_default vlan 102 vni 10002
nv set interface p0_if bridge domain br_default
nv set interface pf0hpf_if bridge domain br_default access 102
nv set nve vxlan enable on
nv set nve vxlan mac-learning off
nv set nve vxlan source address <DPU1-underlay-IP>
nv set nve vxlan arp-nd-suppress off
nv set evpn enable on
nv set router bgp enable on
nv set router bgp autonomous-system 65001
nv set vrf default router bgp peer-group fabric remote-as 65001
nv set vrf default router bgp peer-group fabric address-family l2vpn-evpn enable on
nv set vrf default router bgp neighbor <DPU2-underlay-IP> peer-group fabric
nv set vrf default router bgp neighbor <linux-peer-underlay-IP> peer-group fabric
Diagnostic evidence
HBN bridge fdb show shows the Linux peer as a static HER destination:
00:00:00:00:00:00 dev vxlan48 dst <linux-peer> src_vni 10002 self permanent
HBN software-tables/17 (nl2docad ASIC bridge table) contains the Linux peer’s SVI MAC:
"dst mac":"<linux-peer-svi-mac>"
So the MAC IS programmed into the ASIC bridge table. But the packet still doesn’t egress pf0hpf_if.
vxlan48 bridge port flags look normal:
71: vxlan48 ... master br_default state forwarding
learning on flood on mcast_flood on bcast_flood on
neigh_suppress off
nl2doca daemon is running:
$ supervisorctl status nl2doca
nl2doca RUNNING pid <PID>, uptime <T>
Things tried (none worked)
supervisorctl restart nl2doca- Static FDB entry
bridge fdb add <linux-peer-svi-mac> dev vxlan48 dst <linux-peer> src_vni 10002 self permanent - Stopping
nl2docaentirely (to force software forwarding) — still no packet onpf0hpf_if - Full teardown and rebuild of VLAN 102 config on HBN
- Toggling
arp-nd-suppress,mac-learning, bridge port flags - Verified peer firewall is not blocking (no drops on inbound path)
- Verified no MTU mismatch (all interfaces MTU 9216)
Release notes review
I read the HBN 3.4.0 release notes carefully. None of the following bugs match my symptom:
- 3865633 (open) — describes drops of host-generated UDP:4789 WITHOUT valid VXLAN encap. My packets ARE properly encapsulated and originate from an external peer.
- 4214631 (fixed in 3.0.0) — host-side originated 4789 drops in L3-EVPN. Wrong direction.
- 4264397 (fixed in 3.0.0) — IPv6 NA-with-unicast-MAC not punted. Different protocol (I’m using IPv4).
- 2821785 (open) — hardware MAC learning limitation. But that’s just a performance note, not a silent drop.
- 4991261 (3.4.0 known) — stale VNI after L3 VRF change. Not applicable, my VNI is stable.
I could not find any statement in the HBN documentation about whether non-HBN Linux VTEPs are supported peers.
Questions
- Is HBN 3.3.0 expected to accept and forward VXLAN traffic from a non-HBN (plain Linux kernel + FRR) VTEP peer?
- If yes — what am I missing in the HBN or FRR config?
- If no — is this documented anywhere I should have found?
- Would upgrading to HBN 3.4.0 change this behavior? (No 3.4.0 bug fix in the release notes seems to target this scenario.)
- Is there any punt-path debug I can enable on
nl2docadto see WHY the decapsulated frame is being dropped betweenbr_defaultandpf0hpf_if?
Thanks in advance for any insight. Happy to share full sos reports, pcaps, and hbn-support archives with anyone from the engineering team who wants to look at this.