BlueField-3 DPU mode: eSwitch never delivers vport traffic to representors

Summary

On a BlueField-3 B3210 in DPU mode, no traffic is ever delivered from a vport to its representor netdev, in either direction, for every vport. OVS therefore never sees a packet, and the host-to-Arm datapath does not work at all.

The eSwitch itself is demonstrably healthy. A skip_sw flower rule installs in hardware and counts exactly the packets that the representor fails to receive. The fault appears to be the FDB-miss to representor slow path specifically, not the switch.

Card: BlueField-3 B3210, P/N 900-9D3B6-00CC-A_Ax, PSID MT_0000001024

Mode: DPU mode (BF_MODE DPU mode)

Bundle: bf-bundle-3.4.0-92_26.04_ubuntu-24.04_64k_prod (DOCA 3.4.0, 64k default image)

Kernel: 6.8.0-1022-bluefield-64k — getconf PAGESIZE = 65536

MLNX OFED: MLNX_OFED_LINUX-26.04-0.8.5

NIC firmware: 32.49.1014

OVS: 3.4.0040, ovs-kernel (OVS_DOCA=“no”, dpdk_initialized false)

eSwitch: mode switchdev, inline-mode none, encap-mode basic

Steering: flow_steering_mode smfs, esw_port_metadata true

Thermal: ASIC idle at 67–69 °C

This is a fresh BFB install running the vendor-default configuration. The default ovsbr1 (p0 + pf0hpf + en3f0pf0sf0 + internal) is exactly as the bundle created it. Nothing was added, removed or reconfigured before the test below.

Minimal reproducer

Runs entirely on the Arm. No host involvement, no PCIe, no cables, and no changes to OVS.

sudo ip addr add 10.10.0.2/24 dev ovsbr1

sudo ip link set ovsbr1 up

sudo ip netns add sftest

sudo ip link set enp3s0f0s0 netns sftest # the SF’s own netdev

sudo ip netns exec sftest ip link set enp3s0f0s0 up

sudo ip netns exec sftest ip addr add 10.10.0.3/24 dev enp3s0f0s0

sudo ip netns exec sftest ping -c3 10.10.0.2

Result:

3 packets transmitted, 0 received, +3 errors, 100% packet loss

# ethtool -S en3f0pf0sf0

rx_packets: 0 ← nothing ever reaches the representor

tx_packets: 47

vport_rx_packets: 78 ← but the eSwitch counted the SF’s frames

tcpdump -nei en3f0pf0sf0 captures 0 packets. tcpdump on a netdev captures at RX, before any OVS or TC processing, so this is not OVS discarding them — the representor genuinely never receives. The same holds in reverse: the representor’s tx_packets climbs while the SF’s rx_packets stays at 0.

The eSwitch hardware is fine

On the same install, minutes apart:

sudo tc filter add dev en3f0pf0sf0 ingress protocol ip prio 3 flower skip_sw \

src_ip 10.10.0.3 action drop

sudo tc filter show dev en3f0pf0sf0 ingress

in_hw in_hw_count 1

# send exactly 30 packets from the SF, then:

sudo tc -s filter show dev en3f0pf0sf0 ingress

Sent hardware 2940 bytes 30 pkt

30 sent, 30 matched and counted in hardware. Traffic from the vport reaches the FDB and hardware steering rules match it correctly. Only the default miss path to the representor fails.

Already ruled out

  • OVS. Removing the representor from ovsbr1 entirely and giving it a plain IP changes nothing.

  • The host and PCIe. The reproducer never touches them.

  • hw-offload. Fails identically with true and with false.

  • ovs-doca vs ovs-kernel. OVS_DOCA=“no”, dpdk_initialized false, no DOCA/DPDK libraries linked, empty datapath_type. The kernel datapath is correct and is what runs.

  • Page size. Originally on the 4K image; reflashed to the 64k default, identical result.

  • Soft state. Reproduces on a fresh install minutes after first boot, and survives a full AC power cycle and an Arm-side openibd restart (representor MACs changed, so the eSwitch genuinely re-initialised).

  • The second eSwitch. 0000:03:00.1 reproduces identically with its own SF pair, so this is not damage to one instance.

  • Firmware knobs. All five INTERNAL_CPU_* values are consistent, with ECPF_ESWITCH_MANAGER and ECPF_PAGE_SUPPLIER both ECPF(1).

  • TC rules and traps. No filters on any interface; every devlink -s trap show counter is zero. No drop, error or discard counter is non-zero anywhere — on the SF, the representor or the uplink.

  • Thermal. Reproduces at 67–69 °C.

  • Version. DOCA 3.4.0 is the current release, and 32.49.1014 is the firmware it ships.

devlink port show reports every port with the expected flavour — pf0hpf as pcipf controller 1 external true, and the SF as pcisf with state active opstate attached.

Questions

  1. Is there a known issue in DOCA 3.4.0 or firmware 32.49.1014 where the FDB-miss to representor slow path is not programmed? I found nothing matching in the 3.4.0 known-issues list.

  2. Is there a diagnostic that dumps the FDB miss rules directly, so I can confirm whether the slow-path rule exists at all?

  3. flow_steering_mode is smfs. I could not test dmfs for comparison — changing it requires the eSwitch in legacy mode, and in DPU mode the host PF vport is permanently attached, so it always reports “E-Switch is busy”, even after deleting the SF. Is there a supported way to switch steering mode on a BlueField-3 in DPU mode?

The uplinks p0 and p1 have no cables fitted and show NO-CARRIER. The reproducer never involves an uplink, but I am flagging it in case the FDB miss path depends on uplink carrier in a way I have not anticipated.

Happy to run any diagnostic on request — the card is on a bench and I can reflash it freely.

Thank you

Pre-empting a likely answer, since I found an earlier thread with a similar-sounding symptom — host PF pinging pf0hpf, “Destination Host Unreachable” — which was resolved as two addresses in the same subnet on one Linux host hitting a route-table limit. That is not what is happening here, for three reasons.

1. The two addresses are in separate network namespaces.

In the reproducer the SF netdev sits in the sftest namespace and the representor (or ovsbr1) is in the default namespace. Separate namespaces mean separate route tables — a netns was used precisely so that a single host route table could not be the explanation.

2. The same failure occurs between two genuinely different machines.

The original host-to-Arm test had 10.10.0.1/24 on enp1s0f0np0 on the x86 host and 10.10.0.2/24 on ovsbr1 on the Arm. Those are different systems with different kernels. No single-host route table is involved, and the result is identical.

3. The evidence is below the IP layer entirely.

  • tcpdump -nei en3f0pf0sf0 captures 0 packets. tcpdump captures at netdev RX, before any routing decision.

  • The representor’s rx_packets stays 0 while vport_rx_packets climbs — the eSwitch counted the frames and did not deliver them.

  • A flower skip_sw rule on the representor’s ingress counted 30 of 30 packets in hardware.

If this were a routing or ARP problem the frames would still have arrived at the representor and been visible to tcpdump; the host would simply not have known what to do with them. They never arrive at all.

The hardware rule counting the exact packets that the representor never receives is the core of the report: the vport is live, the FDB is live, and steering programming works. Only the miss path to the representor fails.

Happy to run any diagnostic that would distinguish the two explanations — the card is on a bench and I can reflash it freely.

Hello,

There is no known bug that matches the symptom described here. The mlx5 debugfs exposes the software steering domain, including tables, matchers and rules:

/sys/kernel/debug/mlx5//steering/fdb/<domain_handle>

https://networking-docs.nvidia.com/doca/archive/3-4-0/ovs-kernel-hardware-acceleration#Troubleshooting-SMFS

The dumps can be processed with the steering dump parser:

https://github.com/Mellanox/mlx_steering_dump

Another place you can check is the miss-path drop counter:

/sys/class/net/<rep>/rep_config/miss_rl_dropped_packets
/sys/class/net/<rep>/rep_config/miss_rl_dropped_bytes

Regarding question 3 on switch flow_steering_mode, it’s not in-place once you are already in switchdev. The steering mode must be set before moving to switchdev, and cannot be changed after switchdev is entered.

2026-08-26 · fw 32.49.1014 · bf-bundle-3.4.0-92_26.04_ubuntu-24.04_64k_prod · kernel 6.8.0-1022-bluefield-64k

Thank you — all three pointers were actionable. Two are now closed with measurements and the third found something. Results below, in your order.

1. Miss-path drop counters — not the cause

miss_rl_dropped_packets stays at zero through a live reproduction, on every representor, on both PFs:

miss_rl_cfg rate: 0[packets/s] burst: 0[packets] (disabled)

miss_rl_dropped_packets 0 ← read AFTER 796 frames entered the vport

miss_rl_dropped_bytes 0

representor rx_packets 0

representor vport_rx_packets 796

So the frames are not being rate-limited away. Consistent with everything else we have measured: no counter anywhere reports a drop.

2. Steering dump — attached

/sys/kernel/debug/mlx5/0000:03:00.0/steering/fdb/dmn_00000000d7a135a6 dumps cleanly (16,674 bytes, 245 records) and parses with mlx_steering_dump. Raw and parsed output are attached rather than summarized, but for orientation:

35 rules, 4 tables, 7 matchers

actions 24x VPORT 0x0 3x VPORT 0xffff (uplink)

5x VPORT 0x8020 2x FT (chained) 1x MODIFY_HDR

matches 58x metadata_reg_c_0 18x source_qp 2x dmac

Nothing looks absent to us, but we are not the right readers of this file. If there is a specific table or matcher you would like isolated, say which and we will pull it.

3. flow_steering_mode before switchdev — this is the useful one

You are right that it cannot be changed in place, and that corrected a wrong conclusion on our side: we had recorded dmfs as untestable, because devlink dev eswitch set … mode legacy returned busy. That was true only of the instance with our SF attached. On the second, untouched instance the full sequence works:

devlink dev eswitch set pci/0000:03:00.1 mode legacy ok

devlink dev param set pci/0000:03:00.1 name flow_steering_mode value dmfs ok

devlink dev eswitch set pci/0000:03:00.1 mode switchdev ok, dmfs held

We also found why it never persisted. /usr/sbin/mlnx_bf_configure forces smfs on every boot unless IPsec full offload is enabled:

steering_mode=`get_steering_mode ${dev}`

if [ “${steering_mode}” == “dmfs” ]; then

set_eswitch_mode ${dev} legacy

set_steering_mode ${dev} smfs

fi

if [ “${IPSEC_FULL_OFFLOAD}” == “yes” ]; then

… set_steering_mode ${dev} dmfs

fi

So we are running the SMFS path because that is the bundle default, and IPSEC_FULL_OFFLOAD=“yes” in /etc/mellanox/mlnx-bf.conf is the supported route to a persistent dmfs. We have now run it. Results in the next section.

Is `IPSEC_FULL_OFFLOAD` the route you would have chosen, or is there a way to select dmfs for a test that does not also enable IPsec full offload?

Result under dmfs: identical failure

IPSEC_FULL_OFFLOAD=“yes”, rebooted, both PFs came up dmfs (confirmed with devlink dev param show) on an otherwise untouched vendor-default boot. The reproducer was the first thing run:

representor rx_packets 0 → 0

representor vport_rx 37 ← frames entered

miss_rl_dropped_packets 0

ping 100% loss

So the fault is not in the SMFS miss-path implementation. It reproduces on both steering implementations, on both eSwitch instances, on a fresh vendor-default install, on current firmware.

One divergence, which we would value your reading of

The skip_sw flower rule we have been using to show the FDB is alive behaves differently in the two modes:

Column 1 Column 2 Column 3
smfs dmfs
rule installs in_hw in_hw_count 1 in_hw in_hw_count 1
packets matched in hardware 30 of 30 0, after 124 frames entered the vport
representor rx_packets 0 0

We polled to 30 seconds because the rule reports used_hw_stats delayed; the counter never left zero.

We cannot tell from here whether that means the rule matched no packets — i.e. vport traffic is not reaching the FDB at all under dmfs — or simply that hardware stats are not populated for this rule type under dmfs. The two readings point in very different directions and we would rather ask than guess.

The primary result is not ambiguous either way: the representor receives nothing in either mode.

Two further negatives since the last post

Unicast, not just ARP. Every earlier test began with ping, hence with an ARP broadcast, and the dump contains a dmac 01:00:00:00:00:00 rule — so broadcast handling was a live hypothesis. With static neighbours installed in both directions, so ICMP leaves as pure unicast: 100% loss, representor rx_packets delta 0. Not broadcast-specific.

The frames do not reach the host either. Since every rule ends in VPORT 0x0, we checked whether traffic is landing on the host PF instead of the Arm. Host-side enp1s0f0np0, read before and after an Arm-side reproduction: rx_packets 0, rx_vport_unicast_packets 0, unchanged.

The reproducer, unchanged

Entirely on the Arm. No host, no PCIe, no OVS, no cables.

sudo ip addr add 10.10.0.4/24 dev en3f0pf0sf0

sudo ip link set en3f0pf0sf0 up

sudo ip netns add sftest

sudo ip link set enp3s0f0s0 netns sftest

sudo ip netns exec sftest ip link set enp3s0f0s0 up

sudo ip netns exec sftest ip addr add 10.10.0.3/24 dev enp3s0f0s0

sudo ip netns exec sftest ping -c3 10.10.0.4

100% loss. tcpdump -nei en3f0pf0sf0 captures zero. The representor’s rx_packets stays 0 while its vport_rx_packets climbs. Identical on both eSwitch instances, on a fresh vendor-default install.

And the hardware is provably alive on that same interface:

tc filter add dev en3f0pf0sf0 ingress protocol ip prio 3 flower skip_sw \

src_ip 10.10.0.3 action drop → in_hw in_hw_count 1

(send exactly 30 packets) → Sent hardware 2940 bytes 30 pkt

30 sent, 30 matched and counted in hardware, while the representor netdev received zero. The FDB works. The FDB-miss → representor path does not.

Withdrawing this report — I cannot reproduce it

I need to correct my own thread. The problem I reported does not reproduce on
this card, and the reproducer I posted now passes.
I am withdrawing the report
rather than following up on it, and I apologise for the noise.

Run verbatim today — the exact commands from my first post, same addresses, same
vendor-default ovsbr1:

sudo ip addr add 10.10.0.2/24 dev ovsbr1
sudo ip link set ovsbr1 up
sudo ip netns add sftest
sudo ip link set enp3s0f0s0 netns sftest
sudo ip netns exec sftest ip link set enp3s0f0s0 up
sudo ip netns exec sftest ip addr add 10.10.0.3/24 dev enp3s0f0s0
sudo ip netns exec sftest ping -c3 10.10.0.2
3 packets transmitted, 3 received, 0% packet loss
rtt min/avg/max/mdev = 0.135/11.339/33.744/15.842 ms

# ethtool -S en3f0pf0sf0
     rx_packets: 835 -> 842        <-- I reported 0
     vport_rx_packets: 9510496 -> 9510505

What else I retested, and it all works

Same card, same day, bundle-default configuration:

vport class result
SF (en3f0pf0sf0) works
VF (c1pf0vf0, host VF on enp1s0f0np0) works, 0% loss both directions
host PF (pf0hpf) works

The VF class was never tested when I filed the report; SF and PF were, and both
now pass. I could not find any configuration in which a representor fails to
receive.

One of my own tests WAS wrong, and one was not — the difference matters

I had two variants of this reproducer and I conflated them:

  • An internal five-line variant put the address on the representor
    (ip addr add 10.10.0.4/24 dev en3f0pf0sf0) while that port was a member of
    ovsbr1. That fails, correctly and by design — an OVS-enslaved port is not
    answered by its own local stack. That one was my error.
  • The variant I posted here put the address on the bridge
    (dev ovsbr1). That is a correct test. It is the one that has changed
    behaviour, and I cannot explain why.

So I am not claiming I merely misconfigured the test I showed you. I am claiming
I cannot reproduce it, which is a weaker and more honest statement.

Confirmed again on 2026-08-29, on a different boot

I re-ran the same reproducer on the day of writing, on a later boot of the
card — because this whole message is about a result that changed once already,
and I did not want to report a second change by accident:

3 packets transmitted, 3 received, 0% packet loss
rtt min/avg/max/mdev = 0.133/4.604/13.519/6.303 ms

# ethtool -S en3f0pf0sf0
     rx_packets:       216 -> 224     (delta 8)     <- the post reported 0
     tx_packets:       1922 -> 1932   (delta 10)
     vport_rx_packets: 216 -> 226     (delta 10)

# tcpdump -nei en3f0pf0sf0
     10 frames captured                             <- the post reported 0

Same commands, same addresses, same card. It passes, and rx_packets tracks
vport_rx_packets rather than staying at zero.

The absolute counter values are lower than in the run above (216 against
835) because the Arm was rebooted between the two runs and the counters reset
with it — the deltas are what matter, and both are non-zero where the original
report had rx_packets flat at 0.

The card was not idle: SR-IOV VFs were active on the host PF and carrying
traffic for an unrelated workload, and the DPU had been up ~1h45m. It was
restored afterwards — ovsbr1 back DOWN with no address, enp3s0f0s0 back in
the root namespace, no namespaces left, both bridges as the bundle created
them.

flow_steering_mode reads dmfs, where the original post recorded smfs.
That difference is still present, still unexplained, and still the most likely
of the four candidates below.

What changed between then and now — none of it established as the cause

  • flow_steering_mode was smfs when I filed; it reads dmfs now
    (devlink dev param show pci/0000:03:00.0 name flow_steering_mode, and
    /sys/class/net/p0/compat/devlink/steering_mode).
  • The card was reflashed to the 64k bundle and has been power-cycled and
    rebooted many times since.
  • OVS was reconfigured in between: p0/p1 were removed from
    ovsbr1/ovsbr2 and added back during unrelated work, and the bridges were
    brought down and up.
  • The uplinks now have a cable fitted (a single 100G loopback, port 1 to port
    2); they were NO-CARRIER when I filed. I flagged the missing cable in the
    original post as a long shot, and I still have no evidence it is relevant —
    but it is a difference and I would rather list it than not.

I have since tested the cable/carrier difference as far as it can be tested
without unplugging, and it is the weakest of the four:

  • ip link set p1 down does not remove p0’s carrier on this loopback —
    mlx5 keeps the PHY up on an administrative down, so carrier cannot be removed
    from software. Reproducing the original NO-CARRIER state needs a firmware
    port disable or a physical unplug.
  • More to the point, pf0hpf and en3f0pf0sf0 each report carrier=1
    independently of the uplink. ovsbr1 therefore had carrier from its
    representor members when I filed, even with p0 dead — so the obvious
    mechanism, “the bridge internal port never came up and so never answered”,
    does not hold.

That does not rule the cable out, but it removes the reason to suspect it.

Of the remaining three, flow_steering_mode is the one I would look at first,
and it is also partly testable — it is a runtime parameter, not a property of
the reflash. I have not tried to bisect the others, because I no longer have a
failing state to bisect against.

My question 3 is also moot

I asked about a skip_sw flower counter divergence between smfs and dmfs
30 of 30 packets counted in hardware under smfs, 0 after 124 frames under
dmfs. That does not reproduce either. The card is running dmfs today and the
same control counts 30 of 30 in hardware:

tc filter add dev <rep> ingress protocol ip prio 3 flower skip_sw \
    src_ip 10.10.0.3 action drop      ->  in_hw in_hw_count 1
(send exactly 30 packets)             ->  Sent hardware 2940 bytes 30 pkt

So please disregard that question as well.

What I would still value, if it is cheap to answer

Only one thing, and it is the loose end I cannot close myself: in my original
data the representor’s rx_packets was 0 while vport_rx_packets climbed.
Today rx_packets moves in step with vport_rx_packets in every configuration
I can construct — including the ones where the ping fails for the OVS reason
above. I have no explanation for a representor counting vport_rx while
rx_packets stays flat, and I would like to know whether that combination is
something the driver can produce legitimately, or whether it indicates the
earlier readings were taken wrongly.

If it is not cheap to answer, please just close the thread. There is nothing
here for NVIDIA to fix, and I would rather say so plainly than leave an open
report against a card that works.

Thanks to anyone who spent time on it.


Environment as of this post

Card         BlueField-3 B3210, PSID MT_0000001024
Bundle       bf-bundle-3.4.0-92_26.04_ubuntu-24.04_64k_prod
Kernel       6.8.0-1022-bluefield-64k
NIC firmware 32.49.1014
OVS          3.4.0040 (ovs-kernel)
eSwitch      mode switchdev, inline-mode none, encap-mode basic
             flow_steering_mode dmfs