Intermittent Packet Loss During VRRP MAC Reconfiguration on ConnectX-6 DX VF (SR-IOV, DPDK 23.11.3, vSRX VM)

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Problem Description

When any VRRP group transitions its state (Master ↔ Backup), which involves adding or removing the corresponding Virtual MAC address on the VF, 1–2 ICMP packet drops are consistently observed on other VRRP groups’ virtual IPs that remain in a stable state.

The packet loss occurs only on the Mellanox ConnectX-6 Dx SR-IOV VF path.
When the VF is configured in promiscuous mode, the issue does not occur.

Environment

  • Virtualization: SR-IOV enabled — VFs directly assigned to VM (no representor)

  • Guest OS: FreeBSD 15 / Junos OS (vSRX)

  • Application Stack: DPDK 23.11.3 (used by vSRX data plane)

  • Feature in Use: VRRP (~30 groups) using Virtual MACs on VF interface

  • VF Configuration: ALLMULTICAST enabled; promiscuous mode disabled

Hardware and Software Configuration

  • NIC: Mellanox BlueField-2 ConnectX-6 Dx (MT42822)

  • Interface: ens3f1np1 (PF)

  • VFs: 4 active (VF 0–3) — trust mode on, spoof check off

  • Driver: MLX5 25.01-0.6.0

  • Firmware: 24.39.8002

  • Host Kernel: 5.4.0-216-generic

# ethtool -i ens3f1np1
driver: mlx5_core
version: 25.01-0.6.0
firmware-version: 24.39.8002 (MT_0000000540)
bus-info: 0000:19:00.1
supports-statistics: yes
supports-priv-flags: yes


# ip link show ens3f1np1
13: ens3f1np1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 ...
    vf 0  link/ether aa:bb:cc:19:01:01 ..., spoof checking off, trust on
    vf 1  link/ether aa:bb:cc:19:01:02 ..., spoof checking off, trust on
    vf 2  link/ether aa:bb:cc:19:01:03 ..., spoof checking off, trust on
    vf 3  link/ether aa:bb:cc:19:01:04 ..., spoof checking off, trust on

Observation

  • ICMP packet loss correlates exactly with VRRP state transitions on any group.

  • Loss is visible on unrelated VRRP groups’ VIPs.

  • ethtool -S <pf_name> | grep rx_steered shows incrementing error counters during MAC add/remove operations.

  • Enabling VF promiscuous mode fully mitigates the issue.

Request

Please investigate whether this issue is related to VF mac add/delete filtering behavior in the MLX5 driver or firmware when VRRP virtual MACs are dynamically updated on VFs (non-promiscuous mode).

Kindly confirm if this is a known issue or if a fix/workaround is available in later OFED or firmware releases. Just note that DPDK 23.11.x enabling promiscous mode resolve this issue- no 1-2 packets drops during VRRP failover.