Hello there!
I’m install DOCA on my host (x86-64) ubuntu and bf-bundle image on my BlueField 2. I run doca_secure_channel and it’s executed well. Now i need to test DOCA Flow library and i create 2 VF according to DOCA Virtual Functions User Guide - NVIDIA Docs . I’m create hugepages according to guide and trying to run doca_switch reference application but i get following errors:
$ sudo ./doca_switch -- -r pci/03:00.1,pf1vf[0-1] -l 70
[05:29:42:373443][2225324064][DOCA][INF][doca_log.cpp:628] DOCA version 3.1.0105
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
[05:29:42:415395][2225324064][DOCA][INF][switch.c:85][main] doca_argp_start SUCCESS
EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: 0000:03:00.1 (socket -1)
mlx5_net: Unified FDB is not supported with this FW version.
EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: 0000:03:00.1 (socket -1)
mlx5_net: Unified FDB is not supported with this FW version.
[05:29:42:515872][2225324064][DOCA][INF][switch.c:93][main] init_doca_flow_devs SUCCESS
[05:29:42:515910][2225324064][DOCA][INF][flow_switch_common.c:137][get_dpdk_nb_ports] Port ID 0 is valid DPDK port
[05:29:42:515921][2225324064][DOCA][INF][flow_switch_common.c:137][get_dpdk_nb_ports] Port ID 1 is valid DPDK port
[05:29:42:515931][2225324064][DOCA][INF][flow_switch_common.c:137][get_dpdk_nb_ports] Port ID 2 is valid DPDK port
mlx5_net: port 0 cannot enable promiscuous mode in flow isolation mode
[05:29:42:788864][2225324064][DOCA][DBG][dpdk_utils.c:181][port_init] Port 0 MAC: b8 ce f6 4d c9 41
[05:29:42:788913][2225324064][DOCA][INF][dpdk_utils.c:126][port_init] Skip represent port 1 init in switch mode
[05:29:42:788929][2225324064][DOCA][INF][dpdk_utils.c:126][port_init] Skip represent port 2 init in switch mode
[05:29:42:788940][2225324064][DOCA][INF][switch.c:105][main] dpdk_queues_and_ports_init SUCCESS
[05:29:42:789023][2225324064][DOCA][WRN][engine_model.c:88] adapting queue depth to 128.
Hello from switch_init[05:29:42:794118][2225324064][DOCA][INF][switch_core.c:392][switch_init] Array init started
[05:29:42:794162][2225324064][DOCA][INF][switch_core.c:394][switch_init] Array INITIALIZED
[05:29:42:794183][2225324064][DOCA][INF][flow_common.c:254][create_doca_flow_port] doca_flow_port_cfg CONFIGURED
NV_HWS[cmd_stc_create:514]: Failed to create STC (syndrome: 0x6fa08b)
NV_HWS[pool_create_one_resource:63]: Failed to allocate resource objects
NV_HWS[pool_resource_alloc:87]: Failed allocating resource
NV_HWS[pool_create_resource_on_index:271]: Failed to create resource type: 1: size 15 index: 0
NV_HWS[pool_element_create_new_elem:309]: Failed to create resource type: 1: size 15 index: 0
NV_HWS[pool_onesize_element_get_mem_chunk:362]: Failed to allocate element for order: 0
NV_HWS[pool_onesize_element_db_get_chunk:452]: Failed to get free slot for chunk with order: 0
NV_HWS[action_alloc_single_stc:384]: Failed to allocate single action STC
NV_HWS[action_dest_create_stc:463]: Failed to allocate STC table type [6]
[05:29:42:927248][2225324064][DOCA][ERR][nv_hws_wrappers.c:418] failed to create dest action ROOT, flag 64, err -121
[05:29:42:938836][2225324064][DOCA][ERR][dpdk_port_legacy.c:275] failed to create port - creating dpdk port
Segfault happen on doca_flow_port_start call. As i understood, my application cannot allocate resource objects in STC. But i don’t know why. STC almost not mentioned in DOCA Docs and DPDK docs. Maybe someone has encountered something similar or knows how to fix it? I would be grateful for any help.
Don’t know if it’s important, but i couldn’t set my VF’s interfaces as trusted because Operation not permitted via ip link and similar via sysfs:
$ sudo ip link set ens2f1np1 vf 0 trust on
RTNETLINK answers: Operation not permitted
$ ip link show
31: ens2f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether b8:ce:f6:4d:c9:3d brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
altname enp55s0f1np1
My configuration
Host x86-64 Ubuntu 22.04.5 LTS
$ lspci | grep Blue
37:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
37:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
37:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01)
devlink port show
pci/0000:37:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
function:
hw_addr 00:00:00:00:00:00
pci/0000:37:00.1/131071: type eth netdev ens2f1np1 flavour physical port 1 splittable false
function:
hw_addr 00:00:00:00:00:00
pci/0000:5d:00.0/65535: type eth netdev eno5np0 flavour physical port 0 splittable false
function:
hw_addr 00:00:00:00:00:00
pci/0000:5d:00.1/131071: type eth netdev eno6np1 flavour physical port 1 splittable false
function:
hw_addr 00:00:00:00:00:00
pci/0000:37:02.3/1376256: type eth netdev ens2f1v0 flavour virtual splittable false
pci/0000:37:02.4/1441792: type eth netdev ens2f1v1 flavour virtual splittable false
BlueField 2
devlink port show
pci/0000:03:00.0/262143: type eth netdev p0 flavour physical port 0 splittable false
function:
hw_addr 00:00:00:00:00:00
pci/0000:03:00.0/196608: type eth netdev pf0hpf flavour pcipf controller 1 pfnum 0 external true splittable false
function:
hw_addr b8:ce:f6:4d:c9:3c
pci/0000:03:00.1/327679: type eth netdev p1 flavour physical port 1 splittable false
function:
hw_addr 00:00:00:00:00:00
pci/0000:03:00.1/262144: type eth netdev pf1hpf flavour pcipf controller 1 pfnum 1 external true splittable false
function:
hw_addr b8:ce:f6:4d:c9:3d
pci/0000:03:00.1/262145: type eth netdev pf1vf0 flavour pcivf controller 1 pfnum 1 vfnum 0 external true splittable false
function:
hw_addr 00:00:00:00:00:00
pci/0000:03:00.1/262146: type eth netdev pf1vf1 flavour pcivf controller 1 pfnum 1 vfnum 1 external true splittable false
function:
hw_addr 00:00:00:00:00:00
sudo mlxconfig -d /dev/mst/mt41686_pciconf0.1 query
Device #1:
----------
Device type: BlueField2
Name: MBF2M516A-EEEO_Ax_Bx
Description: BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL
Device: /dev/mst/mt41686_pciconf0.1
Configurations: Next Boot
ACCURATE_TX_SCHEDULER False(0)
ADVANCED_PCI_SETTINGS False(0)
AES_XTS_TWEAK_INC_64 False(0)
ALLOW_RD_COUNTERS True(1)
ATS_ENABLED False(0)
AUTO_POWER_SAVE_LINK_DOWN_P1 False(0)
AUTO_POWER_SAVE_LINK_DOWN_P2 False(0)
BOOT_DBG_LOG False(0)
BOOT_INTERRUPT_DIS False(0)
BOOT_LACP_DIS True(1)
BOOT_PKEY 0
BOOT_UNDI_NETWORK_WAIT 0
BOOT_VLAN_EN False(0)
BOOT_VLAN 1
CLAMP_TGT_RATE_AFTER_TIME_INC_P1 True(1)
CLAMP_TGT_RATE_AFTER_TIME_INC_P2 True(1)
CLAMP_TGT_RATE_P1 False(0)
CLAMP_TGT_RATE_P2 False(0)
CNP_802P_PRIO_P1 6
CNP_802P_PRIO_P2 6
CNP_DSCP_P1 48
CNP_DSCP_P2 48
CQE_COMPRESSION BALANCED(0)
CRYPTO_POLICY UNRESTRICTED(1)
DCBX_CEE_P1 True(1)
DCBX_CEE_P2 True(1)
DCBX_IEEE_P1 True(1)
DCBX_IEEE_P2 True(1)
DCBX_WILLING_P1 True(1)
DCBX_WILLING_P2 True(1)
DCE_TCP_G_P1 1019
DCE_TCP_G_P2 1019
DCE_TCP_RTT_P1 1
DCE_TCP_RTT_P2 1
DCR_LIFO_SIZE 16384
DO_NOT_CLEAR_PORT_STATS_P1 False(0)
DO_NOT_CLEAR_PORT_STATS_P2 False(0)
DPU_RESET_NOTIFICATION_ENABLED ENABLED(1)
DUP_MAC_ACTION_P1 LAST_CFG(0)
DUP_MAC_ACTION_P2 LAST_CFG(0)
DYNAMIC_VF_MSIX_TABLE False(0)
ECPF_ESWITCH_MANAGER ECPF(1)
ECPF_PAGE_SUPPLIER ECPF(1)
ESWITCH_HAIRPIN_DESCRIPTORS Array[0..7]
ESWITCH_HAIRPIN_TOT_BUFFER_SIZE Array[0..7]
EXP_ROM_NVME_UEFI_x86_ENABLE True(1)
EXP_ROM_PXE_ENABLE True(1)
EXP_ROM_UEFI_ARM_ENABLE True(1)
EXP_ROM_UEFI_x86_ENABLE True(1)
EXP_ROM_VIRTIO_BLK_UEFI_ARM_ENABLE False(0)
EXP_ROM_VIRTIO_BLK_UEFI_x86_ENABLE False(0)
EXP_ROM_VIRTIO_NET_PXE_ENABLE False(0)
EXP_ROM_VIRTIO_NET_UEFI_ARM_ENABLE False(0)
EXP_ROM_VIRTIO_NET_UEFI_x86_ENABLE False(0)
FLEX_IPV4_OVER_VXLAN_PORT 0
FLEX_PARSER_PROFILE_ENABLE 0
HAIRPIN_DATA_BUFFER_LOCK False(0)
HIDE_PORT2_PF False(0)
HOST_CHAINING_CACHE_DISABLE False(0)
HOST_CHAINING_DESCRIPTORS Array[0..7]
HOST_CHAINING_MODE DISABLED(0)
HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7]
IB_PROTO_WIDTH_EN_MASK_P1 0
IB_PROTO_WIDTH_EN_MASK_P2 0
IB_ROUTING_MODE_P1 LID(1)
IB_ROUTING_MODE_P2 LID(1)
ICM_CACHE_MODE DEVICE_DEFAULT(0)
INITIAL_ALPHA_VALUE_P1 1023
INITIAL_ALPHA_VALUE_P2 1023
INTERNAL_CPU_ESWITCH_MANAGER ECPF(0)
INTERNAL_CPU_IB_VPORT0 ECPF(0)
INTERNAL_CPU_MODEL EMBEDDED_CPU(1)
INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0)
INTERNAL_CPU_PAGE_SUPPLIER ECPF(0)
INTERNAL_CPU_RSHIM ENABLED(0)
INT_LOG_MAX_PAYLOAD_SIZE AUTOMATIC(0)
IP_OVER_VXLAN_EN False(0)
IP_VER IPv4(0)
KEEP_ETH_LINK_UP_P1 True(1)
KEEP_ETH_LINK_UP_P2 True(1)
KEEP_IB_LINK_UP_P1 False(0)
KEEP_IB_LINK_UP_P2 False(0)
KEEP_LINK_UP_ON_BOOT_P1 False(0)
KEEP_LINK_UP_ON_BOOT_P2 False(0)
KEEP_LINK_UP_ON_STANDBY_P1 False(0)
KEEP_LINK_UP_ON_STANDBY_P2 False(0)
LAG_RESOURCE_ALLOCATION DEVICE_DEFAULT(0)
LARGE_MTU_TWEAK_64 False(0)
LEGACY_BOOT_PROTOCOL PXE(1)
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LLDP_NB_DCBX_P1 False(0)
LLDP_NB_DCBX_P2 False(0)
LLDP_NB_RX_MODE_P1 OFF(0)
LLDP_NB_RX_MODE_P2 OFF(0)
LLDP_NB_TX_MODE_P1 OFF(0)
LLDP_NB_TX_MODE_P2 OFF(0)
LOAD_BALANCE_MODE_P1 DEVICE_DEFAULT(0)
LOAD_BALANCE_MODE_P2 DEVICE_DEFAULT(0)
LOG_DCR_HASH_TABLE_SIZE 11
LOG_MAX_OUTSTANDING_WQE 7
LOG_MAX_QUEUE 17
LOG_TX_PSN_WINDOW 7
LRO_LOG_TIMEOUT0 6
LRO_LOG_TIMEOUT1 7
LRO_LOG_TIMEOUT2 8
LRO_LOG_TIMEOUT3 13
MANAGEMENT_PF_MODE DEVICE_DEFAULT(0)
MAX_PACKET_LIFETIME 0
MEMIC_BAR_SIZE 0
MEMIC_SIZE_LIMIT _256KB(1)
MIN_TIME_BETWEEN_CNPS_P1 4
MIN_TIME_BETWEEN_CNPS_P2 4
MKEY_BY_NAME False(0)
MPFS_MC_LOOPBACK_DISABLE_P1 False(0)
MPFS_MC_LOOPBACK_DISABLE_P2 False(0)
MPFS_UC_LOOPBACK_DISABLE_P1 False(0)
MPFS_UC_LOOPBACK_DISABLE_P2 False(0)
MULTI_PCI_RESOURCE_SHARING DEVICE_DEFAULT(0)
MULTI_PORT_VHCA_EN False(0)
NON_PREFETCHABLE_PF_BAR False(0)
NUM_OF_PFC_P1 8
NUM_OF_PFC_P2 8
NUM_OF_PF 2
NUM_OF_PLANES_P1 0
NUM_OF_PLANES_P2 0
NUM_OF_TC_P1 _8_TCs(0)
NUM_OF_TC_P2 _8_TCs(0)
NUM_OF_VFS 16
NUM_OF_VL_P1 _4_VLs(3)
NUM_OF_VL_P2 _4_VLs(3)
NUM_PF_MSIX 63
NUM_PF_MSIX_VALID False(0)
NUM_VF_MSIX 11
NVME_EMULATION_CLASS_CODE 67586
NVME_EMULATION_DEVICE_ID 24577
NVME_EMULATION_ENABLE False(0)
NVME_EMULATION_MAX_QUEUE_DEPTH 0
NVME_EMULATION_NUM_MSIX 0
NVME_EMULATION_NUM_PF 1
NVME_EMULATION_NUM_VF 0
NVME_EMULATION_REVISION_ID 0
NVME_EMULATION_SUBSYSTEM_ID 0
NVME_EMULATION_SUBSYSTEM_VENDOR_ID 0
NVME_EMULATION_VENDOR_ID 5555
P2P_ORDERING_MODE DEVICE_DEFAULT(0)
PARTIAL_RESET_EN False(0)
PCC_INT_EN False(0)
PCC_INT_NP_RTT_DATA_MODE RTT_V0(64)
PCC_INT_NP_RTT_DSCP_EN False(0)
PCC_INT_NP_RTT_DSCP 26
PCC_INT_SYSTEM_RTT 0
PCIE_CREDIT_TOKEN_TIMEOUT 0
PCI_ATOMIC_MODE PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
PCI_DOWNSTREAM_PORT_OWNER Array[0..15]
PCI_SWITCH_EMULATION_ENABLE False(0)
PCI_SWITCH_EMULATION_NUM_PORT 0
PCI_WR_ORDERING per_mkey(0)
PER_PF_NUM_SF True(1)
PF_BAR2_ENABLE False(0)
PF_BAR2_SIZE 3
PF_LOG_BAR_SIZE 5
PF_NUM_OF_VF_VALID False(0)
PF_NUM_PF_MSIX 1000
PF_NUM_PF_MSIX_VALID True(1)
PF_SD_GROUP 0
PF_SF_BAR_SIZE 10
PF_TOTAL_SF 250
PHY_AUTO_NEG_P1 DEVICE_DEFAULT(0)
PHY_AUTO_NEG_P2 DEVICE_DEFAULT(0)
PHY_COUNT_LINK_UP_DELAY DELAY_NONE(0)
PHY_FEC_OVERRIDE_P1 DEVICE_DEFAULT(0)
PHY_FEC_OVERRIDE_P2 DEVICE_DEFAULT(0)
PHY_RATE_MASK_OVERRIDE_P1 False(0)
PHY_RATE_MASK_OVERRIDE_P2 False(0)
PORT_OWNER True(1)
PRIO_TAG_REQUIRED_EN False(0)
PROG_PARSE_GRAPH False(0)
QOS_TRUST_STATE_P1 TRUST_PCP(1)
QOS_TRUST_STATE_P2 TRUST_PCP(1)
RATE_REDUCE_MONITOR_PERIOD_P1 4
RATE_REDUCE_MONITOR_PERIOD_P2 4
RATE_TO_SET_ON_FIRST_CNP_P1 0
RATE_TO_SET_ON_FIRST_CNP_P2 0
RDMA_SELECTIVE_REPEAT_EN False(0)
REAL_TIME_CLOCK_ENABLE False(0)
RENEG_ON_CHANGE True(1)
RESET_WITH_HOST_ON_ERRORS False(0)
ROCE_ADAPTIVE_ROUTING_EN False(0)
ROCE_CC_CNP_MODERATION_P1 DEVICE_DEFAULT(0)
ROCE_CC_CNP_MODERATION_P2 DEVICE_DEFAULT(0)
ROCE_CC_LEGACY_DCQCN True(1)
ROCE_CC_PRIO_MASK_P1 255
ROCE_CC_PRIO_MASK_P2 255
ROCE_CONTROL ROCE_ENABLE(2)
ROCE_NEXT_PROTOCOL 254
ROCE_RTT_RESP_DSCP_MODE_P1 DEVICE_DEFAULT(0)
ROCE_RTT_RESP_DSCP_MODE_P2 DEVICE_DEFAULT(0)
ROCE_RTT_RESP_DSCP_P1 0
ROCE_RTT_RESP_DSCP_P2 0
RPG_AI_RATE_P1 5
RPG_AI_RATE_P2 5
RPG_BYTE_RESET_P1 32767
RPG_BYTE_RESET_P2 32767
RPG_GD_P1 11
RPG_GD_P2 11
RPG_HAI_RATE_P1 50
RPG_HAI_RATE_P2 50
RPG_MAX_RATE_P1 0
RPG_MAX_RATE_P2 0
RPG_MIN_DEC_FAC_P1 50
RPG_MIN_DEC_FAC_P2 50
RPG_MIN_RATE_P1 1
RPG_MIN_RATE_P2 1
RPG_THRESHOLD_P1 1
RPG_THRESHOLD_P2 1
RPG_TIME_RESET_P1 300
RPG_TIME_RESET_P2 300
RT_PPS_ENABLED_ON_POWERUP False(0)
SAFE_MODE_ENABLE True(1)
SAFE_MODE_THRESHOLD 10
SRIOV_EN True(1)
SRIOV_IB_ROUTING_MODE_P1 LID(1)
SRIOV_IB_ROUTING_MODE_P2 LID(1)
STRICT_VF_MSIX_NUM False(0)
SWITCH_COMPT_FEATURE_MASK 0x0(0)
TLS_OPTIMIZE False(0)
TRACER_ENABLE True(1)
TUNNEL_ECN_COPY_DISABLE False(0)
TUNNEL_IP_PROTO_ENTROPY_DISABLE False(0)
TX_SCHEDULER_BURST 0
UCTX_EN True(1)
UEFI_HII_EN True(1)
UEFI_LOGS DISABLED(0)
UNKNOWN_UPLINK_MAC_FLOOD_P1 False(0)
UNKNOWN_UPLINK_MAC_FLOOD_P2 False(0)
VF_LOG_BAR_SIZE 1
VF_MIGRATION_MODE DEVICE_DEFAULT(0)
VF_NODNIC_ENABLE False(0)
VF_VPD_ENABLE False(0)
VIRTIO_BLK_EMULATION_ENABLE False(0)
VIRTIO_BLK_EMULATION_NUM_MSIX 2
VIRTIO_BLK_EMULATION_NUM_PF 0
VIRTIO_BLK_EMULATION_NUM_VF 0
VIRTIO_BLK_EMULATION_PF_PCI_LAYOUT VIRTIO_1_X(0)
VIRTIO_BLK_EMULATION_SUBSYSTEM_ID 2
VIRTIO_BLK_EMULATION_VF_PCI_LAYOUT VIRTIO_1_X(0)
VIRTIO_BLK_EMU_SUBS_VENDOR_ID 6900
VIRTIO_EMULATION_HOTPLUG_TRANS False(0)
VIRTIO_NET_EMULATION_ENABLE False(0)
VIRTIO_NET_EMULATION_NUM_MSIX 2
VIRTIO_NET_EMULATION_NUM_PF 0
VIRTIO_NET_EMULATION_NUM_VF 0
VIRTIO_NET_EMULATION_PF_PCI_LAYOUT VIRTIO_1_X(0)
VIRTIO_NET_EMULATION_SUBSYSTEM_ID 1
VIRTIO_NET_EMULATION_VF_PCI_LAYOUT VIRTIO_1_X(0)
VIRTIO_NET_EMU_SUBSYSTEM_VENDOR_ID 6900
VL15_BUFFER_SIZE_P1 0
VL15_BUFFER_SIZE_P2 0
ZERO_TOUCH_TUNING_ENABLE False(0)
$ethtool -i p1
driver: mlx5_core
version: 25.07-0.9.7
firmware-version: 24.35.2000 (MT_0000000559)
expansion-rom-version:
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
$ ethtool -i pf1vf0
driver: mlx5e_rep
version: 5.15.0-1074-bluefield
firmware-version: 24.35.2000 (MT_0000000559)
expansion-rom-version:
bus-info:
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no