Environment / topology (two independent endpoints joined by one leaf switch):
- Endpoint A: GB200 NVL72 compute tray, one of its two BlueField-3 DPUs per
host — this is the DPU showing the vxlan48 TX drops. (The host’s other
DPU is intentionally left without admin-VPC config, per NICo/infra-
controller’s is_primary_dpu logic — not usable as a same-host comparison.)- bf-bundle-3.2.2-125_26.02_ubuntu-22.04_prod
- DOCA HBN container: nvcr.io/nvidia/doca/doca_hbn:3.2.2-doca3.2.2
- Kernel: 5.15.0-1086-bluefield
- Middle: Cumulus SN5600 leaf switch, unnumbered eBGP underlay
- Endpoint B: a separate x86 server with its own ConnectX-7 NIC, running
plain Linux + FRR 8.4.4 (not a DOCA HBN container) — manually configured
as an FRR/BGP EVPN “border gateway” peering into Endpoint A’s admin VPC
over the same SN5600 underlay, to test L3-EVPN reachability into the
DPU’s tenant network. - Note: this site’s automated NICo provisioning is currently stuck
pre-discovery for unrelated reasons, so Endpoint A’s admin-VPC/EVPN
config (FNN/L3-EVPN mode) was applied by hand via NVUE (nv set/
nv config apply), matching what NICo’s agent would normally push.
Symptom:
The DPU’s L3VNI VXLAN interface (external/collect-metadata mode, shared with
an L2VNI on the same VLAN-trunk bridge port) shows catastrophic TX packet loss,
and — importantly — this is a locally visible drop, not a silent one:
# ip -s link show vxlan48
TX: bytes packets errors dropped carrier collsns
900 15 0 1300083 0 0
errors stays at 0 but dropped climbs steadily with every failed attempt
(observed growing from 1295880 → 1299779 → 1300083 across successive tests),
accumulated over several weeks of uptime — only a handful of packets ever
succeed. So this specific interface’s own kernel/driver is actively rejecting
these packets at transmit time, not silently losing them downstream. This
happens despite every layer of the EVPN control plane being verified correct:
show ip route vrf <vrf> <prefix>: route present, distance/metric correct,
no rejected/offload-failure flagip route get <dst> vrf <vrf>: resolves via the correct nexthop/onlink SVIip neigh show: nexthop MAC (RMAC) correctly programmed via
extern_learn proto zebrabridge fdb show dev vxlan48: remote VTEP FDB entry correctly present
(dst src_vni self extern_learn)- tcpdump on the sending side’s IRB SVI (
vlan<x>_l3) confirms the SYN/ARP
packet is correctly generated with the right destination MAC — but it
never arrives at the remote VTEP (0 packets captured there).
Isolated root-cause test (note: this test surfaced a DIFFERENT, and
possibly unrelated, bug — see the discrepancy noted below):
Created a brand-new, completely independent point-to-point VXLAN interface
between the same two loopback IPs (unrelated to any EVPN/VNI config,
plain fixed-VNI mode, not “external”):
ip link add test-vxlan type vxlan id <id> dstport <port> \
local <ip-a> remote <ip-b>
With default settings (udpcsum enabled), pings across this tunnel fail
in one direction. Unlike vxlan48, this failure shows zero local TX
drops/errors on either end — the packet is accepted for transmission
with no complaint, it just never gets a reply. Rebuilding the SAME tunnel
with noudpcsum on both ends fixes it completely — 20/20 and then 15/15
follow-up pings at 0% loss, consistently reproducible. This looks like a
UDP checksum offload bug in this NIC/driver combination: a bad checksum
gets silently discarded by the receiving side’s hardware, invisible to
any local counter on the sender.
However, applying the same fix to the REAL vxlan48 EVPN interface (deleted
and recreated with noudpcsum, preserving VLAN-trunk/bridge-slave config,
FDB/EVPN VNI binding correctly re-verified via show evpn vni after an FRR
reload) did NOT restore connectivity — 5/5 test requests through the real
vpc_61325/nico-pxe path still timed out immediately afterward, while the
isolated test tunnel remained 100% healthy at the same time. (For
completeness: before touching checksum settings at all, we’d already
tried restarting nl2doca alone — 5/5 fail — and a forced full
nv config apply + nl2doca restart — 8/8 fail — neither of which
changed anything; those two rounds are unrelated to the checksum
experiment, just ruling out other angles first.)
So I now have two distinct failure signatures that may or may not share a
root cause:
- vxlan48 (external/multi-VNI EVPN mode): LOCAL TX-drop counter climbs
with every failure — the drop is visible and happens at/before
transmit, on this box. - The isolated plain-VXLAN tunnel (before the checksum fix): zero local
drops/errors — transmit “succeeds” locally, the packet is simply never
answered, consistent with the remote side discarding a bad checksum.
Given vxlan48 keeps failing even after switching it to noudpcsum, and its
failure signature (local TX drops) differs from what the checksum bug
looked like in isolation, I suspect there are two separate issues here:
the checksum offload bug (confirmed, fixable with noudpcsum) and a
second, so-far-unidentified issue specific to the “external”/multi-VNI
EVPN vxlan construct that causes this interface to reject packets at
transmit time.
Question 1: I found HBN 3.4.0 release notes list Issue 4369016
(“Datapath flow rules may be missing for ingress IPv4 traffic during VXLAN
L3 native tenant encapsulated testing…OVS logs display DOCA hardware
steering warnings”) under Known Issues — this matches our symptom almost
exactly (correct control-plane state, silent dataplane failure). Is this
the same issue? Is there a fix or workaround, or a version where it’s
resolved?
Question 2: Is there a way to disable UDP checksum computation for a
zebra/NVUE-managed EVPN VXLAN interface (the kind created via nv config apply with nve: vxlan: enable: on, not manually with ip link add)?
This device gets recreated by NVUE/zebra, so a manual ip link flag
doesn’t stick across any reapply.
Happy to provide full packet captures, nv show output, and FRR
running-config on request.
For reference, the same class of “VXLAN + checksum offload” bug has been
reported on completely unrelated hardware/stacks — e.g.
UDP: bad checksum on VXLAN interface · Issue #1279 · flannel-io/flannel · GitHub (“UDP: bad checksum on VXLAN
interface”), where ethtool -K flannel.1 tx-checksum-ip-generic off was
the confirmed workaround. This suggests the checksum portion of our issue
may not be BlueField-specific, but the secondary “external”/multi-VNI EVPN
forwarding failure (which persists even with checksum disabled) does seem
specific to this platform/HBN version.