Assigning VLAN IDs to Virtual NICs under SR-IOV: "Operation not supported"

Goal: Assign a VLAN id to a virtual NIC under SR-IOV

Problem: the putative command “ip link set dev ieth2 vf 0 vlan 153” reports “RTNETLINK answers: Operation not supported.”

Context:
I am able to configure (under Redhat) a ConnectX-6 NIC into SR-IOV mode and make virtual NICs. The NIC is programmed into switch dev mode. I can assign MAC addresses to the virtual functions, including IPV4 addresses. All that works fine.

For the purposes of this question the error arises this way:

  1. Assume NIC configure in HW for SR-IOV OK
  2. Assume host BIOS has SR-IOV, Vt-d ON
  3. Assume Linux box has IOMMU in boot line

Now the next steps are,

  1. devlink dev eswitch set pci/ mode switchdev
  2. echo 4 > /sys/class/net/ieth2/device/sriov_numvfs
  3. echo > /sys/bus/pci/drivers/mlx5_core/unbind
  4. ip link set ieth2 vf 0 mac xx:xx:…xx

Lines 4 through 7 work fine. The next line DOES NOT WORK reporting “Operation not supported”:

  1. ip link set ieth2 vf 0 vlan 153

What should I do?

The issue here is order of commands thanks to
https://bugzilla.redhat.com/show_bug.cgi?id=1856468

The correct order is make SR-IOV VFs, then configure VF mac/qlan etc., then put the link into switchdev mode:

DEV="ieth3"
MLX="mlx5_0"
PCI="0000:12:00.0"

# turn off SR-IOV
echo 0 > /sys/class/net/$DEV/device/sriov_numvfs
# put link into normal, legacy mode
devlink dev eswitch set pci/$PCI mode legacy
devlink dev eswitch show pci/$PCI

# OK, starting in a define state turn on SR-IOV
set -x
# make four virtual NICs
echo 4 > /sys/class/net/$DEV/device/sriov_numvfs
# unbind the VF0 on 0000:12:00.2 
echo 0000:12:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
# turn trust on, spookchk on, and assign qlan plus mac
ip link set $DEV vf 0 trust on
ip link set $DEV vf 0 spoofchk on
ip link set $DEV vf 0 vlan 153
ip link set $DEV vf 0 mac  80:37:57:17:44:00
# rebind it to MLX5 driver
echo 0000:12:00.2 > /sys/bus/pci/drivers/mlx5_core/bind
set +x

# and end with putting ieth3 (the physical function) into switchdev mode
devlink dev eswitch set pci/$PCI mode switchdev encap-mode none
devlink dev eswitch show pci/$PCI

ip -d link show $DEV