(Updated) Linux TC HW Offload not working on ConnectX-5/6

[shane.miller]

Goal : Mirror Incoming/Outgoing Packets Between Two Ethernet Ports in Hardware

Env : Fresh, unmodified install of Unbuntu 22.04 on bare metal Intel x86_64
Dual port ConnectX-5 in PCI slots 0000:01.00/0000:01.01
Two ethernet links ‘enp1s0f0’, ‘enp1s0f1’

Basic Solution: Run these Linux Commands:

0 # enable HW offload
1 ethtool -K enp1s0f0 hw-tc-offload on
2 ethtool -K enp1s0f1 hw-tc-offload on
3 # ingress mirror in HW
4 tc qdisc add dev enp1s0f1 ingress
5 tc filter add dev enp1s0f1 ingress flower verbose skip_sw action mirred egress mirror dev enp1s0f0
7 # egress mirror in HW
8 tc qdisc add dev enp1s0f1 root prio
9 tc filter add dev enp1s0f1 root flower verbose skip_sw action mirred egress mirror dev enp1s0f0

The filter commands give error:
Error: mlx5_core: Not implemented offload action. There is no additional detail in dmesg on Ubuntu.

These filter variations DO WORK:

• In line 5, 9 replace keyword “mirror” with “redirect” i.e. HW redirect works
• In line 5, 9 remove keyword “skip_sw” i.e. mirror works in kernel/software

This is a very basic problem:

• A successful install of MLNX_OFED_LINUX-23.10-1.1.9.0-ubuntu22.04-x86_64.tgz
  with or without updating the ConnectX-5 firmware give the same error
• The same errors occurs on stock RHEL8. In the RHEL dmesg cryptically mentions
  class(ifier) error.
• I’ve seen this issue on Connect6-X NICs too
• I’ve seen the same errors on Ubuntu 24.xx

This question is a repeat of,

ASAP2 port mirroring with tc

Does ASAP2 support port mirroring on a MCX512A-ACAT? i.e can we replicate what’s going out of one port on a second. We are trying this with 4.7 driver on Linux and latest firmware as follows: tc qdisc add dev ens1f0 handle 1: root prio tc filter add dev ens1f0 parent 1: flower skip_sw dst_mac e4:11:22:11:4a:51 action mirred egress mirror dev ens1f1 But we get a cryptic error message from tc, is this supported and if so what filters are allowed?

which was never answered right. Variations can be found across the WWW too.

This error comes from omitting act->id=5 (MIRRER) in the MLX5_CORE module but supporting REDIRECT (v5.15.136 is the kernel version for Ubuntu 22.04LTS):

flow_offload.h - include/net/flow_offload.h - Linux source code (v5.15.136) -...

https://elixir.bootlin.com/linux/v5.15.136/source/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c#L3614

This means the MLX5_CORE driver does not support the TC fragment “action mirred (ingress|egress) mirror”.

How can packets be mirrored then between ethernet links?

[sribhargavid]

Hello @shane.miller,

Thank you for posting your query on our community. in order to debug this issue further, we would request you to submit a support ticket along with a sysinfo snapshot captured from the node for further troubleshooting. The support ticket can be opened by emailing " Networking-support@nvidia.com ".

Thanks,
Bhargavi

[shane.miller]

NVIDIA support has been no help.

However, I believe I was able to figure out how to do this. When testing is done I will post a link here to the solution, which is how NVIDIA should provide technical documentation.