We compared two setups with the same overlay (OpenStack/OVN, Geneve, Security Groups), VM → VM across two nodes:
- OVS-TC offload: ConnectX-6 Lx MCX631102AN-ADA, fw 26.43.8016, Ubuntu 22.04 kernel 6.8 inbox mlx5_core, switchdev + VF-LAG, OVS 3.2.3
hw-offload=true. All flows verified indump-flows type=offloaded(Geneve encap/decap + ct + recirc). Guests on vDPA (mlx5_vdpa/vhost_vdpa) and SR-IOV VF. - OVS-kernel: kernel datapath, Broadcom NIC, kernel 5.15, virtio/tap.
CPU and throughput on the offload nodes are as expected. Latency is not:
| OVS-kernel | OVS-TC offload | |
|---|---|---|
| First packet of a new flow | ≤ 2 ms, no loss | 27–38 ms, 1 packet lost |
| ping idle avg / p99 | 0.136 / 0.194 ms | 0.198 / 0.231 ms (SR-IOV end) · 0.235 / 0.280 ms (vDPA end) |
| qperf tcp_lat | 38.1 µs | 30.4 µs (SR-IOV end) · 35.9 µs (vDPA end) |
Question: is higher latency than OVS-kernel — a ~30 ms setup cost with one dropped packet per new flow, and a higher steady-state RTT unless the endpoint is SR-IOV — the expected behaviour of TC-flower offload on ConnectX-6 Lx, or does it indicate a misconfiguration? If expected, what tuning (OVS n-offload-threads / tc-policy / max-idle, mlx5 steering mode, firmware, vhost-vdpa doorbell mapping, interrupt moderation) brings it back to or below OVS-kernel?
__
Thank you!
Quoc, Tran Van