Unable to change to switch Dev mode

Infrastructure & Networking BlueField
1

Hi all , I’m trying to do OVS Hardware Offload

I don’t know why I cannot set the eSwitch to switchdev mode even though I had unbound VFs before attempting to set it.

devlink dev eswitch set pci/0000:43:00.0 mode switchdev

The error message is as follows:

kernel answers: Operation not supported

I thought BlueField DPU mode could set switchdev, but it’s doesn’t work for me.

Here is the hardware information

lspci -nnD |grep Me
0000:21:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
0000:43:00.0 Ethernet controller [0200]: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller [15b3:a2d6] (rev 01)
0000:43:00.1 Ethernet controller [0200]: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller [15b3:a2d6] (rev 01)
0000:43:00.2 DMA controller [0801]: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface [15b3:c2d3] (rev 01)
0000:43:00.3 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e] (rev 01)
0000:43:00.4 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e] (rev 01)
0000:43:00.5 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e] (rev 01)
0000:43:00.6 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e] (rev 01)

sudo mlxconfig -d /dev/mst/mt41686_pciconf0 q

Device #1:
----------

Device type:        BlueField2
Name:               MBF2H332A-AECO_Ax_Bx
Description:        BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL
Device:             /dev/mst/mt41686_pciconf0

Configurations:                                          Next Boot
        MEMIC_BAR_SIZE                              0
        MEMIC_SIZE_LIMIT                            _256KB(1)
        HOST_CHAINING_MODE                          DISABLED(0)
        HOST_CHAINING_CACHE_DISABLE                 False(0)
        HOST_CHAINING_DESCRIPTORS                   Array[0..7]
        HOST_CHAINING_TOTAL_BUFFER_SIZE             Array[0..7]
        INTERNAL_CPU_MODEL                          EMBEDDED_CPU(1)
        INTERNAL_CPU_PAGE_SUPPLIER                  ECPF(0)
        INTERNAL_CPU_ESWITCH_MANAGER                ECPF(0)
        INTERNAL_CPU_IB_VPORT0                      ECPF(0)
        INTERNAL_CPU_OFFLOAD_ENGINE                 ENABLED(0)
        FLEX_PARSER_PROFILE_ENABLE                  0
        PROG_PARSE_GRAPH                            False(0)
        FLEX_IPV4_OVER_VXLAN_PORT                   0
        ROCE_NEXT_PROTOCOL                          254
        ESWITCH_HAIRPIN_DESCRIPTORS                 Array[0..7]
        ESWITCH_HAIRPIN_TOT_BUFFER_SIZE             Array[0..7]
        PF_BAR2_SIZE                                3
        DPU_RESET_NOTIFICATION_ENABLED              ENABLED(1)
        INTERNAL_CPU_RSHIM                          ENABLED(0)
        PF_NUM_OF_VF_VALID                          False(0)
        NON_PREFETCHABLE_PF_BAR                     False(0)
        VF_VPD_ENABLE                               False(0)
        PF_NUM_PF_MSIX_VALID                        False(0)
        PER_PF_NUM_SF                               False(0)
        STRICT_VF_MSIX_NUM                          False(0)
        VF_NODNIC_ENABLE                            False(0)
        NUM_PF_MSIX_VALID                           True(1)
        NUM_OF_VFS                                  16
        NUM_OF_PF                                   2
        PF_BAR2_ENABLE                              True(1)
        HIDE_PORT2_PF                               False(0)
        SRIOV_EN                                    True(1)
        PF_LOG_BAR_SIZE                             5
        VF_LOG_BAR_SIZE                             1
        NUM_PF_MSIX                                 63
        NUM_VF_MSIX                                 11
        INT_LOG_MAX_PAYLOAD_SIZE                    AUTOMATIC(0)
        PCIE_CREDIT_TOKEN_TIMEOUT                   0
        RT_PPS_ENABLED_ON_POWERUP                   False(0)
        LAG_RESOURCE_ALLOCATION                     DEVICE_DEFAULT(0)
        PHY_COUNT_LINK_UP_DELAY                     DELAY_NONE(0)
        ACCURATE_TX_SCHEDULER                       False(0)
        PARTIAL_RESET_EN                            False(0)
        RESET_WITH_HOST_ON_ERRORS                   False(0)
        NVME_EMULATION_ENABLE                       False(0)
        NVME_EMULATION_NUM_VF                       0
        NVME_EMULATION_NUM_PF                       1
        NVME_EMULATION_VENDOR_ID                    5555
        NVME_EMULATION_DEVICE_ID                    24577
        NVME_EMULATION_CLASS_CODE                   67586
        NVME_EMULATION_REVISION_ID                  0
        NVME_EMULATION_SUBSYSTEM_VENDOR_ID          0
        NVME_EMULATION_SUBSYSTEM_ID                 0
        NVME_EMULATION_NUM_MSIX                     0
        NVME_EMULATION_MAX_QUEUE_DEPTH              0
        PCI_SWITCH_EMULATION_NUM_PORT               0
        VIRTIO_EMULATION_HOTPLUG_TRANS              False(0)
        PCI_SWITCH_EMULATION_ENABLE                 False(0)
        VIRTIO_NET_EMULATION_VF_PCI_LAYOUT          VIRTIO_1_X(0)
        VIRTIO_NET_EMULATION_PF_PCI_LAYOUT          VIRTIO_1_X(0)
        VIRTIO_NET_EMULATION_ENABLE                 False(0)
        VIRTIO_NET_EMULATION_NUM_VF                 0
        VIRTIO_NET_EMULATION_NUM_PF                 0
        VIRTIO_NET_EMU_SUBSYSTEM_VENDOR_ID          6900
        VIRTIO_NET_EMULATION_SUBSYSTEM_ID           4161
        VIRTIO_NET_EMULATION_NUM_MSIX               2
        VIRTIO_BLK_EMULATION_VF_PCI_LAYOUT          VIRTIO_1_X(0)
        VIRTIO_BLK_EMULATION_PF_PCI_LAYOUT          VIRTIO_1_X(0)
        VIRTIO_BLK_EMULATION_ENABLE                 False(0)
        VIRTIO_BLK_EMULATION_NUM_VF                 0
        VIRTIO_BLK_EMULATION_NUM_PF                 0
        VIRTIO_BLK_EMU_SUBSYSTEM_VENDOR_ID          6900
        VIRTIO_BLK_EMULATION_SUBSYSTEM_ID           4162
        VIRTIO_BLK_EMULATION_NUM_MSIX               2
        PCI_DOWNSTREAM_PORT_OWNER                   Array[0..15]
        CQE_COMPRESSION                             BALANCED(0)
        IP_OVER_VXLAN_EN                            False(0)
        MKEY_BY_NAME                                False(0)
        PRIO_TAG_REQUIRED_EN                        False(0)
        UCTX_EN                                     True(1)
        REAL_TIME_CLOCK_ENABLE                      False(0)
        RDMA_SELECTIVE_REPEAT_EN                    False(0)
        PCI_ATOMIC_MODE                             PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
        TUNNEL_ECN_COPY_DISABLE                     False(0)
        LRO_LOG_TIMEOUT0                            6
        LRO_LOG_TIMEOUT1                            7
        LRO_LOG_TIMEOUT2                            8
        LRO_LOG_TIMEOUT3                            13
        LOG_TX_PSN_WINDOW                           7
        VF_MIGRATION_MODE                           DEVICE_DEFAULT(0)
        LOG_MAX_OUTSTANDING_WQE                     7
        ROCE_ADAPTIVE_ROUTING_EN                    False(0)
        TUNNEL_IP_PROTO_ENTROPY_DISABLE             False(0)
        USER_PROGRAMMABLE_CC                        False(0)
        PCC_INT_NP_RTT_DSCP                         26
        PCC_INT_NP_RTT_DSCP_EN                      False(0)
        PCC_INT_NP_RTT_DATA_MODE                    RTT_V0(64)
        PCC_INT_EN                                  False(0)
        PCC_INT_SYSTEM_RTT                          0
        STEERING_CACHE_REFRESH                      0
        MULTI_PCI_RESOURCE_SHARING                  DEVICE_DEFAULT(0)
        ICM_CACHE_MODE                              DEVICE_DEFAULT(0)
        HAIRPIN_DATA_BUFFER_LOCK                    False(0)
        TLS_OPTIMIZE                                False(0)
        TX_SCHEDULER_BURST                          0
        ZERO_TOUCH_TUNING_ENABLE                    False(0)
        ROCE_CC_LEGACY_DCQCN                        False(0)
        LOG_MAX_QUEUE                               17
        UPT_EMULATION_ENABLE                        False(0)
        LARGE_MTU_TWEAK_64                          False(0)
        AES_XTS_TWEAK_INC_64                        False(0)
        CRYPTO_POLICY                               UNRESTRICTED(1)
        RDE_DISABLE                                 False(0)
        PLDM_FW_UPDATE_DISABLE                      False(0)
        RBT_DISABLE                                 False(0)
        PCIE_SMBUS_DISABLE                          False(0)
        PCIE_IN_BAND_VDM_DISABLE                    False(0)
        LOG_DCR_HASH_TABLE_SIZE                     11
        MAX_PACKET_LIFETIME                         0
        DCR_LIFO_SIZE                               16384
        ROCE_CC_PRIO_MASK_P1                        255
        ROCE_CC_PRIO_MASK_P2                        255
        CLAMP_TGT_RATE_AFTER_TIME_INC_P1            True(1)
        CLAMP_TGT_RATE_P1                           False(0)
        RPG_TIME_RESET_P1                           300
        RPG_BYTE_RESET_P1                           32767
        RPG_THRESHOLD_P1                            1
        RPG_MAX_RATE_P1                             0
        RPG_AI_RATE_P1                              5
        RPG_HAI_RATE_P1                             50
        RPG_GD_P1                                   11
        RPG_MIN_DEC_FAC_P1                          50
        RPG_MIN_RATE_P1                             1
        RATE_TO_SET_ON_FIRST_CNP_P1                 0
        DCE_TCP_G_P1                                1019
        DCE_TCP_RTT_P1                              1
        RATE_REDUCE_MONITOR_PERIOD_P1               4
        INITIAL_ALPHA_VALUE_P1                      1023
        MIN_TIME_BETWEEN_CNPS_P1                    4
        CNP_802P_PRIO_P1                            6
        CNP_DSCP_P1                                 48
        CLAMP_TGT_RATE_AFTER_TIME_INC_P2            True(1)
        CLAMP_TGT_RATE_P2                           False(0)
        RPG_TIME_RESET_P2                           300
        RPG_BYTE_RESET_P2                           32767
        RPG_THRESHOLD_P2                            1
        RPG_MAX_RATE_P2                             0
        RPG_AI_RATE_P2                              5
        RPG_HAI_RATE_P2                             50
        RPG_GD_P2                                   11
        RPG_MIN_DEC_FAC_P2                          50
        RPG_MIN_RATE_P2                             1
        RATE_TO_SET_ON_FIRST_CNP_P2                 0
        DCE_TCP_G_P2                                1019
        DCE_TCP_RTT_P2                              1
        RATE_REDUCE_MONITOR_PERIOD_P2               4
        INITIAL_ALPHA_VALUE_P2                      1023
        MIN_TIME_BETWEEN_CNPS_P2                    4
        CNP_802P_PRIO_P2                            6
        CNP_DSCP_P2                                 48
        LLDP_NB_DCBX_P1                             False(0)
        LLDP_NB_RX_MODE_P1                          OFF(0)
        LLDP_NB_TX_MODE_P1                          OFF(0)
        LLDP_NB_DCBX_P2                             False(0)
        LLDP_NB_RX_MODE_P2                          OFF(0)
        LLDP_NB_TX_MODE_P2                          OFF(0)
        ROCE_RTT_RESP_DSCP_P1                       0
        ROCE_RTT_RESP_DSCP_MODE_P1                  DEVICE_DEFAULT(0)
        ROCE_RTT_RESP_DSCP_P2                       0
        ROCE_RTT_RESP_DSCP_MODE_P2                  DEVICE_DEFAULT(0)
        DCBX_IEEE_P1                                True(1)
        DCBX_CEE_P1                                 True(1)
        DCBX_WILLING_P1                             True(1)
        DCBX_IEEE_P2                                True(1)
        DCBX_CEE_P2                                 True(1)
        DCBX_WILLING_P2                             True(1)
        KEEP_ETH_LINK_UP_P1                         True(1)
        KEEP_IB_LINK_UP_P1                          False(0)
        KEEP_LINK_UP_ON_BOOT_P1                     False(0)
        KEEP_LINK_UP_ON_STANDBY_P1                  False(0)
        DO_NOT_CLEAR_PORT_STATS_P1                  False(0)
        AUTO_POWER_SAVE_LINK_DOWN_P1                False(0)
        KEEP_ETH_LINK_UP_P2                         True(1)
        KEEP_IB_LINK_UP_P2                          False(0)
        KEEP_LINK_UP_ON_BOOT_P2                     False(0)
        KEEP_LINK_UP_ON_STANDBY_P2                  False(0)
        DO_NOT_CLEAR_PORT_STATS_P2                  False(0)
        AUTO_POWER_SAVE_LINK_DOWN_P2                False(0)
        NUM_OF_VL_P1                                _4_VLs(3)
        NUM_OF_TC_P1                                _8_TCs(0)
        NUM_OF_PFC_P1                               8
        VL15_BUFFER_SIZE_P1                         0
        QOS_TRUST_STATE_P1                          TRUST_PCP(1)
        NUM_OF_VL_P2                                _4_VLs(3)
        NUM_OF_TC_P2                                _8_TCs(0)
        NUM_OF_PFC_P2                               8
        VL15_BUFFER_SIZE_P2                         0
        QOS_TRUST_STATE_P2                          TRUST_PCP(1)
        DUP_MAC_ACTION_P1                           LAST_CFG(0)
        MPFS_MC_LOOPBACK_DISABLE_P1                 False(0)
        MPFS_UC_LOOPBACK_DISABLE_P1                 False(0)
        UNKNOWN_UPLINK_MAC_FLOOD_P1                 False(0)
        SRIOV_IB_ROUTING_MODE_P1                    LID(1)
        IB_ROUTING_MODE_P1                          LID(1)
        DUP_MAC_ACTION_P2                           LAST_CFG(0)
        MPFS_MC_LOOPBACK_DISABLE_P2                 False(0)
        MPFS_UC_LOOPBACK_DISABLE_P2                 False(0)
        UNKNOWN_UPLINK_MAC_FLOOD_P2                 False(0)
        SRIOV_IB_ROUTING_MODE_P2                    LID(1)
        IB_ROUTING_MODE_P2                          LID(1)
        PHY_AUTO_NEG_P1                             DEVICE_DEFAULT(0)
        PHY_RATE_MASK_OVERRIDE_P1                   False(0)
        PHY_FEC_OVERRIDE_P1                         DEVICE_DEFAULT(0)
        PHY_AUTO_NEG_P2                             DEVICE_DEFAULT(0)
        PHY_RATE_MASK_OVERRIDE_P2                   False(0)
        PHY_FEC_OVERRIDE_P2                         DEVICE_DEFAULT(0)
        PF_TOTAL_SF                                 0
        PF_SD_GROUP                                 0
        PF_SF_BAR_SIZE                              0
        PF_NUM_PF_MSIX                              63
        SILENT_MODE                                 False(0)
        ROCE_CONTROL                                ROCE_ENABLE(2)
        PCI_WR_ORDERING                             per_mkey(0)
        MULTI_PORT_VHCA_EN                          False(0)
        PORT_OWNER                                  True(1)
        ALLOW_RD_COUNTERS                           True(1)
        RENEG_ON_CHANGE                             True(1)
        TRACER_ENABLE                               True(1)
        IP_VER                                      IPv4(0)
        BOOT_UNDI_NETWORK_WAIT                      0
        UEFI_HII_EN                                 True(1)
        BOOT_DBG_LOG                                False(0)
        UEFI_LOGS                                   DISABLED(0)
        BOOT_VLAN                                   1
        LEGACY_BOOT_PROTOCOL                        NONE(0)
        BOOT_INTERRUPT_DIS                          False(0)
        BOOT_LACP_DIS                               True(1)
        BOOT_VLAN_EN                                False(0)
        BOOT_PKEY                                   0
        P2P_ORDERING_MODE                           DEVICE_DEFAULT(0)
        EXP_ROM_VIRTIO_NET_PXE_ENABLE               True(1)
        EXP_ROM_VIRTIO_NET_UEFI_ARM_ENABLE          True(1)
        EXP_ROM_VIRTIO_NET_UEFI_x86_ENABLE          True(1)
        EXP_ROM_VIRTIO_BLK_UEFI_ARM_ENABLE          True(1)
        EXP_ROM_VIRTIO_BLK_UEFI_x86_ENABLE          True(1)
        EXP_ROM_NVME_UEFI_x86_ENABLE                True(1)
        ATS_ENABLED                                 False(0)
        DYNAMIC_VF_MSIX_TABLE                       False(0)
        EXP_ROM_UEFI_ARM_ENABLE                     True(1)
        EXP_ROM_UEFI_x86_ENABLE                     True(1)
        EXP_ROM_PXE_ENABLE                          True(1)
        ADVANCED_PCI_SETTINGS                       False(0)
        SAFE_MODE_THRESHOLD                         10
        SAFE_MODE_ENABLE                            True(1)

Can anyone give me advice?

2

Hi,

Thanks for your question.
switchdev mode is already the default mode on DPU, and it’s controlled by DPU, so can be changed from DPU shell (you need to connect DPU via ssh or console to change that).
The message you see on the host side is expected.

Best Regards,
Anatoly

Best Regards,
Anatoly

3

Hi abirman,

Thanks for your reply,

I followed the documentation in the Switching Support - NVIDIA Docs

In this section, I thought that switchdev mode should be changed in the host OS.

This is because the document states:

"Change the eSwitch mode from legacy to switchdev on the PF device:

# devlink dev eswitch set pci/0000:3b:00.0 mode switchdev

This also creates the VF representor netdevices in the host OS."

I don’t have VF representors on the host OS side, but they appear on the ARM OS side after I created VF on the host side.

It is switchdev mode in the ARM OS side but it still can’t configure in the host side.

I just want to figure out the behavior is correct or not ? Thanks!

4

Hi,

On the DPU representors are on the ARM side only.
The switchdev mode is the default mode on DPU. This is controlled on the ARM side.
You are creating VFs from the host side, but you don’t need to change to switchdev on the host side.
Representors are seen on the ARM side only and connected to OVS bridge.
On the host you will see VFs only.

Best Regards,
Anatoly

1 Like
5

Thanks for your help!

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.