CX6 ASAP² VF-LAG Transmits Through Only One PF

Infrastructure & Networking Software And Drivers Mellanox OFED
1

We are observing an intermittent throughput drop with the following setup:

  • NIC: ConnectX-6
  • Driver: OFED-internal-25.04-0.6.1
  • Firmware: 22.45.1020
  • Networking: OVS-DPDK with SR-IOV and ASAP² hardware offload in VF-LAG mode
  • Bond: bond0, configured in 802.3ad mode and connected to an S-MLAG switch pair
  • PF lag_port_select_mode: queue_affinity
  • Topology: Two hosts, each running one VM with an SR-IOV VF passed through to the guest

We run TCP iperf between the two VMs with 30 parallel connections (-P 30). After the test starts, aggregate throughput sometimes drops from approximately 180 Gbps to 90 Gbps. During the drop, the transmit packet rate reported by mlnx_perf for tx_packets_phy on one PF falls to zero, while the other PF continues transmitting.

No PF link-down events or bond slave state changes are reported in dmesg or /var/log/messages.

Increasing the number of parallel TCP connections to 100 (-P 100) generally allows the aggregate transmit throughput to remain around 180 Gbps, without the drop to 90 Gbps observed with -P 30.

Is this a known issue with VF-LAG in queue_affinity mode? What diagnostics would help determine why traffic stops using one physical port?