Problem trying to test a ConnectX-4 LC with switchdev

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hi there,

I’m hoping someone can point out what I’m doing wrong when trying to configure switchdev mode with a Mellanox ConnectX-4 LC adapter.

I also see that the dmesg output states the normal SRIOV mode is listed as “LEGACY”, does this mean it could be removed in future?

For testing switchdev I’m getting the below error:
Error: mlx5_core: Error checking for existing bridge with same ifindex.

I’m testing out on Archlinux on a clean install.

# uname -a
Linux localhost.localdomain 6.14.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 10 Apr 2025 18:43:59 +0000 x86_64 GNU/Linux

# ip link
...
2: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
     link/ether ec:0d:9a:3a:ce:ca brd ff:ff:ff:ff:ff:ff
     altname enp1s0f0np0
     altname enxec0d9a3aceca
3: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
     link/ether ec:0d:9a:3a:ce:cb brd ff:ff:ff:ff:ff:ff
     altname enp1s0f1np1
     altname enxec0d9a3acecb
...

# lspci -vnn -s 01:00.0; lspci -vnn -s 01:00.1
01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
         DeviceName: NIC1
         Subsystem: Mellanox Technologies ConnectX-4 Lx 25 GbE Dual Port SFP28 rNDC [15b3:0025]
         Flags: bus master, fast devsel, latency 0, IRQ 126, NUMA node 0, IOMMU group 40
         Memory at a6000000 (64-bit, prefetchable) [size=32M]
         Expansion ROM at b7e00000 [disabled] [size=1M]
         Capabilities: [60] Express Endpoint, IntMsgNum 0
         Capabilities: [48] Vital Product Data
         Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
         Capabilities: [c0] Vendor Specific Information: Len=18 <?>
         Capabilities: [40] Power Management version 3
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
         Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
         Capabilities: [1c0] Secondary PCI Express
         Capabilities: [230] Access Control Services
         Kernel driver in use: mlx5_core
         Kernel modules: mlx5_core
01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
         DeviceName: NIC2
         Subsystem: Mellanox Technologies ConnectX-4 Lx 25 GbE Dual Port SFP28 rNDC [15b3:0025]
         Flags: bus master, fast devsel, latency 0, IRQ 129, NUMA node 0, IOMMU group 41
         Memory at a4000000 (64-bit, prefetchable) [size=32M]
         Expansion ROM at b7f00000 [disabled] [size=1M]
         Capabilities: [60] Express Endpoint, IntMsgNum 0
         Capabilities: [48] Vital Product Data
         Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
         Capabilities: [c0] Vendor Specific Information: Len=18 <?>
         Capabilities: [40] Power Management version 3
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
         Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
         Capabilities: [230] Access Control Services
         Kernel driver in use: mlx5_core
         Kernel modules: mlx5_core

# lspci
...
01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family 
[ConnectX-4 Lx] [15b3:1015]
01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family 
[ConnectX-4 Lx] [15b3:1015]
01:00.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family 
[ConnectX-4 Lx Virtual Function] [15b3:1016]
01:00.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family 
[ConnectX-4 Lx Virtual Function] [15b3:1016]
01:00.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family 
[ConnectX-4 Lx Virtual Function] [15b3:1016]
01:00.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family 
[ConnectX-4 Lx Virtual Function] [15b3:1016]
...

I then unbind from mlx5_core…

# echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind

And set the device into switchdev mode…

# devlink dev eswitch set pci/0000:01:00.0 mode switchdev
# devlink dev eswitch show pci/0000:01:00.0
pci/0000:01:00.0: mode switchdev inline-mode link encap-mode basic
# devlink dev show
pci/0000:01:00.0:
   nested_devlink:
     auxiliary/mlx5_core.eth.0
auxiliary/mlx5_core.eth.0
pci/0000:01:00.1:
   nested_devlink:
     auxiliary/mlx5_core.eth.1
auxiliary/mlx5_core.eth.1
pci/0000:04:00.0:
   nested_devlink:
     auxiliary/mlx5_core.eth.2
auxiliary/mlx5_core.eth.2
pci/0000:04:00.1:
   nested_devlink:
     auxiliary/mlx5_core.eth.3
auxiliary/mlx5_core.eth.3

# dmesg
...
[ 1184.307616] mlx5_core 0000:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(4), necvfs(0), active vports(5)
[ 1187.238127] mlx5_core 0000:01:00.0: E-Switch: Tc chains and priorities offload aren't supported
[ 1187.656245] mlx5_core 0000:01:00.0: sampler_termtbl_create:72:(pid 2002): termination table is not supported
[ 1187.764322] mlx5_core 0000:01:00.0 eno1np0: Dropping C-tag vlan stripping offload due to S-tag vlan
[ 1187.764330] mlx5_core 0000:01:00.0 eno1np0: Disabling HW_VLAN CTAG FILTERING, not supported in switchdev mode
[ 1187.905188] mlx5_core 0000:01:00.0 eno1r0: renamed from eth0
[ 1187.997947] mlx5_core 0000:01:00.0 eno1r1: renamed from eth0
[ 1188.094991] mlx5_core 0000:01:00.0 eno1r2: renamed from eth0
[ 1188.186064] mlx5_core 0000:01:00.0: E-Switch: Enable: mode(OFFLOADS), 
nvfs(4), necvfs(0), active vports(4)
[ 1188.188408] mlx5_core 0000:01:00.0 eno1r3: renamed from eth0

All looks good so far…

# ip link
...
2: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether ec:0d:9a:3a:ce:ca brd ff:ff:ff:ff:ff:ff
     vf 0     link/ether 42:10:90:00:01:00 brd ff:ff:ff:ff:ff:ff, spoof 
checking off, link-state disable, trust off, query_rss off
     vf 1     link/ether 42:10:90:00:01:01 brd ff:ff:ff:ff:ff:ff, spoof 
checking off, link-state disable, trust off, query_rss off
     vf 2     link/ether 42:10:90:00:01:02 brd ff:ff:ff:ff:ff:ff, spoof 
checking off, link-state disable, trust off, query_rss off
     vf 3     link/ether 42:10:90:00:01:03 brd ff:ff:ff:ff:ff:ff, spoof 
checking off, link-state disable, trust off, query_rss off
     altname enp1s0f0np0
     altname enxec0d9a3aceca
...
11: eno1r0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether 2e:4c:3a:17:70:2e brd ff:ff:ff:ff:ff:ff
     altname enp1s0f0r0
12: eno1r1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether 46:16:78:95:a5:1f brd ff:ff:ff:ff:ff:ff
     altname enp1s0f0r1
13: eno1r2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether 5e:7d:1d:8e:09:d2 brd ff:ff:ff:ff:ff:ff
     altname enp1s0f0r2
14: eno1r3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether 66:28:62:9c:2c:59 brd ff:ff:ff:ff:ff:ff
     altname enp1s0f0r3

The problem comes in when I try add the device to a bridge… from
networking/device_drivers/ethernet/mellanox/mlx5/switchdev.html

# ip link add testbr0 type bridge
# ip link set eno1np0 master testbr0
Error: mlx5_core: Error checking for existing bridge with same ifindex.

I get the same error trying to add the representor devices…

# ip link set eno1r0 master testbr0
Error: mlx5_core: Error checking for existing bridge with same ifindex.

# dmesg
...
[ 1515.581631] testbr0: port 1(eno1np0) entered blocking state
[ 1515.581637] testbr0: port 1(eno1np0) entered disabled state
[ 1515.581661] mlx5_core 0000:01:00.0 eno1np0: entered allmulticast mode
[ 1515.581724] mlx5_core 0000:01:00.0 eno1np0: left allmulticast mode
[ 1515.581730] mlx5_core 0000:01:00.0 eno1np0: failed (err=-22) to set 
attribute (id=6)
[ 1703.825900] testbr0: port 1(eno1r0) entered blocking state
[ 1703.825906] testbr0: port 1(eno1r0) entered disabled state
[ 1703.825937] mlx5_core 0000:01:00.0 eno1r0: entered allmulticast mode
[ 1703.826043] mlx5_core 0000:01:00.0 eno1r0: left allmulticast mode
[ 1703.826051] mlx5_core 0000:01:00.0 eno1r0: failed (err=-22) to set 
attribute (id=6)
...

Additional details…

# devlink -V
devlink utility, iproute2-6.14.0

# ip -V
ip utility, iproute2-6.14.0, libbpf 1.5.0

# bridge -V
bridge utility, 6.14.0

# mstflint -d 01:00.0 q
Image type:            FS3
FW Version:            14.32.2004
FW Release Date:       13.1.2022
Product Version:       14.32.2004
Rom Info:              type=UEFI version=14.25.18 cpu=AMD64
                        type=PXE version=3.6.502 cpu=IA32
Description:           UID                GuidsNumber
Base GUID:             ec0d9a03003aceca        8
Base MAC:              ec0d9a3aceca            8
Image VSD:             N/A
Device VSD:            N/A
PSID:                  DEL2810000034
Security Attributes:   N/A

Any help or guidance would be greatly appreciated!

Kind Regards
Nigel

2

Make sure to use our Nvidia MLNX driver, the OS mlx5 Inbox driver is not tested with Nvidia but rather the respective OSs vendor.
Ref to our drivers:

We have multiple solutions with “switchdev” configuration + OVS
Ref https://docs.nvidia.com/networking/display/nvidia-mlnx-ofed-documentation-v24-10-2-1-8-0-lts.0%20LTS.pdf

OVS offload using ASAP2 direct
OVS-kernel hardware offloads

Note: The solutions were implemented for ConnectX-5 onwards, the ConnectX4 LX was not part of these solutions.

You can further consult with the OS vendor for a qualified solution or refer to DELL as I see the PSID is DEL2810000034