Setting CX4 lx adapter in mode switchdev

[shambulance]

Hi,

Trying to get offloading (vlan) to work, but can’t seem to get the adapter in to mode switchdev,
Running Ubuntu 20.04, latest firmware on the adapter. Tried to remove the adapter from openvswitch, but it reveals the same result. I know the openvswitch version is outdated for offloading but does it matter when i am only trying to set the card in the correct mode?
Followed an openstack based guide, have a working SR-IOV setup failing on the step 2 (setting mode switchdev).
https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html#configure-open-vswitch-hardware-offloading

root@aio1:~# ofed_info -s
mlnx-en-5.9-0.5.6.0:

root@aio1:~# uname -r
5.4.0-139-generic

root@aio1:~# devlink dev eswitch set pci/0000:04:00.1 mode switchdev
Error: mlx5_core: Failed setting eswitch to offloads.
devlink answers: Invalid argument
root@aio1:~#

• dmesg
  Feb 13 21:27:52 aio1 START: -bash
  Feb 13 21:27:57 aio1 kernel: mlx5_core 0000:04:00.1: E-Switch: Disable: mode(LEGACY), nvfs(4), active vports(0)
  Feb 13 21:27:57 aio1 kernel: mlx5_core 0000:04:00.1: mlx5_cmd_out_err:808:(pid 76675): CREATE_FLOW_TABLE(0x930) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x98afbb), err(-22)
  Feb 13 21:27:57 aio1 kernel: mlx5_core 0000:04:00.1: E-Switch: Failed to create slow path FDB Table err -22
  Feb 13 21:28:04 aio1 END: -bash

root@aio1:~# ethtool -i enp4s0f1np1
driver: mlx5_core
version: 5.9-0.5.5
firmware-version: 14.32.1010 (MT_2420110034)
expansion-rom-version:
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
root@aio1:~# dpkg -l | grep iproute
ii iproute2 5.5.0-1ubuntu1 amd64 networking and traffic control tools
root@aio1:~# dpkg -l openvswitch-switch
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
++±==================-======================-============-===================================
ii openvswitch-switch 2.16.4-0ubuntu1~cloud0 amd64 Open vSwitch switch implementations

Greatful for any tips or pointers,
Wbr
///Mathias

[chenh1]

Hi Mathias,

Please ensure all VF devices are unbound before setting the PF device in switchdev mode. For example:

echo 0000:04:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind

Thanks,
Chen

[shambulance]

Hi,

Thank you for your reply, this was very true on another card i did the same thing to.
Not until every last one of the VF’s was unbound could i change to switch mode on a ConnectX 6 DX. I’ll give the Cx4 lx a try once its no longer in use.
Thanks again,
WBR
///Mathias