How to enable MAC address spoofing check for SFs

Infrastructure & Networking Software And Drivers Virtualization For Infiniband And Ethernet
1

I want to prevent an SF from using an arbitrary MAC address. In legacy esw mode with VF, this can be done via ip-link spoofchk on, but in switchdev mode, I’m not sure how to achieve this.

After reading the driver source code, I have a hypothesis that, unlike ixgbe, mlx5 doesn’t configure a specific MAC address filter but instead allows only one specific MAC address on the port. Is this correct?

If so, it can be configured, for example, using tc filter or rte_flow, right?

Additionally, if I want to have different MAC addresses on the vDPA (vnet) “port” and on the eth port on the SF (devlink param enable_vnet & enable_eth), is this possible?
I.e. I want to prevent MAC address changes/bypasses on both vnet and eth ports, but the MAC addresses should be different on both ports. Is this possible? (I want this for combined shaping for both the VM network and its sidecars like SDS client)

Maybe there is a way to determine which port (eth or vnet) a packet came from, in the hws engine, by configuring it via rte_flow/doca-flow? To allow a specific MAC addresses on eth and vnet ports.

Also, what does hw_addr in devlink port function set affect?
And, what does the MAC address affect on the SF representor netdev? Nothing?

2

-devlink-port(8) - Linux manual page

To prevent a SmartNIC Function (SF) or VF from using an arbitrary MAC address in switchdev mode, use devlink port function set to assign MAC.

IE: devlink port function set pci/0000:01:00.0/1 hw_addr 00:00:00:11:22:33 state active trusted false

Assigns a MAC address to the function.

Makes it non-trusted, which disables the ability to change MAC or spoof it.

This prevents the SF from sending with any MAC other than the one configured.

Note: Only trusted=true allows arbitrary MACs; trusted=false restricts it to the assigned MAC.

-You can also install hardware filter rules that only allow egress packets with a specific MAC source. On the representor interface.

-After reading the driver source code, I have a hypothesis that, unlike ixgbe, mlx5 doesn’t configure a specific MAC address filter but instead allows only one specific MAC address on the port. Is this correct? YES

-If so, it can be configured, for example, using tc filter or rte_flow, right? YES

-Yes — it is possible to assign different MAC addresses to the eth and vDPA (vnet) ports of a Single SF — but with key constraints depending on the NIC model (e.g., ConnectX-6 Dx, ConnectX-7), firmware version, and driver/kernel support.

-Yes, you can have different MAC addresses on the eth port and the vDPA port of a single SF, as long as:

  • You assign the eth MAC via devlink.
  • You assign the vDPA MAC via vdpa-tool or QEMU.
  • You explicitly allow the second MAC at the eswitch level (via tc or by enabling trust if supported).

Note: When both eth and vnet are enabled for a single SF, you’re dealing with multi-functionality over a single function, but they share the underlying devlink port.

The default MAC address for the SF (via devlink port function set) is propagated to both eth and vnet.

-Also, what does hw_addr in devlink port function set affect?

Field Purpose Applies To Enforced by HW? Used as src MAC?

mac Active MAC address of netdev PF, VF, SF (ethX) ✅ Yes ✅ Yes

hw_addr Stored hardware MAC (low-level) Management, auxiliary, fw ❌ Often No ❌ Usually No

  • And, what does the MAC address affect on the SF representor netdev? Nothing?

The representor (rep) is a host-side netdev that represents the traffic of a VF or SF in switchdev mode. It acts like a virtual patch cable into the NIC’s embedded switch (eswitch), allowing the host to:

  • Apply tc rules for classification/offload.
  • Monitor traffic.
  • Control flows between SFs/VFs and uplinks.

The MAC address of the SF representor (rep) netdev is mostly informational, and does not affect actual traffic handling or filtering.

3

devlink port function set pci/0000:01:00.0/1 hw_addr 00:00:00:11:22:33 state active trusted false

There is no trust parameter in devlink, even in the manpage you provided. However, a closer analysis of the mlnx-ofed-kernel-dkms source code shows that the trust parameter is supported by mlxdevm netlink. But mlxdevm netlink is supported only in mlnx-ofed-kernel-dkms, and vDPA is not supported when it is installed.
I think this is not a fatal problem, solvable by backporting mlxdevm to the upstream mlx5 driver, but it is quite unpleasant. And I do not find any mentions (on linux mailing lists) of adding such parameters (max_uc_macs, trust) from mlxdevm to devlink, have you [Nvidia] thought about it?

After reading the driver source code, I have a hypothesis that, unlike ixgbe, mlx5 doesn’t configure a specific MAC address filter but instead allows only one specific MAC address on the port. Is this correct? YES

I read the source code of the mlx5_esw_set_hca_trusted function used by mlxdevm netlink, and I think the answer is “no”. Because it, unlike the ip-link spoofchkoption (in mainline), sets a trust level register rather than adding a generic filter.

Yes — it is possible to assign different MAC addresses to the eth and vDPA (vnet) ports of a Single SF — but with key constraints depending on the NIC model (e.g., ConnectX-6 Dx, ConnectX-7), firmware version, and driver/kernel support.

Yes, you can have different MAC addresses on the eth port and the vDPA port of a single SF, as long as:

  • You assign the eth MAC via devlink.

Okay, if I do this and enable trust mode, then the vnet and eth ports will be restricted to this mac, right? This is unacceptable because the VM shouldn’t be able to access SDS and other private services, they should only be accessible through the eth and rdma ports (under the same representor (devlink port)) so that the VM network and the VM disk share the same network bandwidth.

  • You assign the vDPA MAC via vdpa-tool or QEMU.

Okay, but note that the VM can freely change (and spoof) MAC addresses, and there’s no way to prevent that except filtering ingress traffic from vnet ports.

  • You explicitly allow the second MAC at the eswitch level (via tc or by enabling trust if supported).

How? If I enable trust mode, then the VM (through the vnet port) will be able to spoof MAC addresses.

The representor (rep) is a host-side netdev that represents the traffic of a VF or SF in switchdev mode. It acts like a virtual patch cable into the NIC’s embedded switch (eswitch), allowing the host to:
* Monitor traffic.

Hmm, I don’t think tcpdump works on SF representors. But it works on VF representors 🤔

4

Also, it seems that setting trust off is only supported if the esw manager is ECPF of the DPU, but when I removed the mlx5_core_is_ecpf(esw->dev) check from the driver, that field is actually set to off, but I can’t test right now if it works correctly, and in general I don’t like such “magic”.

Especially since it’s a downstream driver with its own magic things, many of which should be upstreamed - just like the trust field in devlink-port(8).

@spruitt any comments on that?

5

Okay I tested it and it doesn’t works.

Anyway, the main problem is to have different MAC addresses on eth&rdma and vnet ports, with spoofing protection on the vnet port, to use SF-based traffic shaping applied to both the VM NIC and the VM’s host side, such as SDS clients.

image
image2022×546 198 KB