Assigning VLAN IDs to Virtual NICs under SR-IOV: "Operation not supported"

Infrastructure & Networking Software And Drivers Mellanox OFED
, ,
1

Goal: Assign a VLAN id to a virtual NIC under SR-IOV

Problem: the putative command “ip link set dev ieth2 vf 0 vlan 153” reports “RTNETLINK answers: Operation not supported.”

Context:
I am able to configure (under Redhat) a ConnectX-6 NIC into SR-IOV mode and make virtual NICs. The NIC is programmed into switch dev mode. I can assign MAC addresses to the virtual functions, including IPV4 addresses. All that works fine.

For the purposes of this question the error arises this way:

  1. Assume NIC configure in HW for SR-IOV OK
  2. Assume host BIOS has SR-IOV, Vt-d ON
  3. Assume Linux box has IOMMU in boot line

Now the next steps are,

  1. devlink dev eswitch set pci/ mode switchdev
  2. echo 4 > /sys/class/net/ieth2/device/sriov_numvfs
  3. echo > /sys/bus/pci/drivers/mlx5_core/unbind
  4. ip link set ieth2 vf 0 mac xx:xx:…xx

Lines 4 through 7 work fine. The next line DOES NOT WORK reporting “Operation not supported”:

  1. ip link set ieth2 vf 0 vlan 153

What should I do?

2

The issue here is order of commands thanks to
https://bugzilla.redhat.com/show_bug.cgi?id=1856468

The correct order is make SR-IOV VFs, then configure VF mac/qlan etc., then put the link into switchdev mode:

DEV="ieth3"
MLX="mlx5_0"
PCI="0000:12:00.0"

# turn off SR-IOV
echo 0 > /sys/class/net/$DEV/device/sriov_numvfs
# put link into normal, legacy mode
devlink dev eswitch set pci/$PCI mode legacy
devlink dev eswitch show pci/$PCI

# OK, starting in a define state turn on SR-IOV
set -x
# make four virtual NICs
echo 4 > /sys/class/net/$DEV/device/sriov_numvfs
# unbind the VF0 on 0000:12:00.2 
echo 0000:12:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
# turn trust on, spookchk on, and assign qlan plus mac
ip link set $DEV vf 0 trust on
ip link set $DEV vf 0 spoofchk on
ip link set $DEV vf 0 vlan 153
ip link set $DEV vf 0 mac  80:37:57:17:44:00
# rebind it to MLX5 driver
echo 0000:12:00.2 > /sys/bus/pci/drivers/mlx5_core/bind
set +x

# and end with putting ieth3 (the physical function) into switchdev mode
devlink dev eswitch set pci/$PCI mode switchdev encap-mode none
devlink dev eswitch show pci/$PCI

ip -d link show $DEV
3

After changing the mode to switchdev, I can no longer modify VLANs. It seems that KVM managed by libvirt cannot use hardware-offloaded VFs.

4

Hi,
I use connectx-6 in switchdev mode
I m testing passing VF to a virtual machine on proxmox (with VFIO)
it s work fine
At boot , before switch to eswitch switchdev mode , I set mac address and “spoofchk on”
settings are here :

11: ens1f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 10000
    link/ether b8:3f:d2:e9:b1:07 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether a4:da:22:6a:90:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state disable, trust off, query_rss off

but the eswitch filter nothing, if I change the (vm) mac-address
so VF connectivity in eswitch is working good but we cant filter any mac without tc rules

It seems eswitch does not honor VF featureset so I m not surprise you can t set vlan on VF iface

do you know what is the difference beytween legacy and switchdev modes ?