According to the image provided, the scenario we implemented involves two Ubuntu 24.04 servers, both connected via BlueField-3 network cards and a Mellanox MQM 8700 Infiniband switch over an Infiniband network.
In addition, the opensmd service is active and running on both the Ubuntu hosts and Ubuntu DPU operating systems.
Furthermore, the QSFP ports of the network card in the DPU operating system are recognized with the following names:
lo: UNKNOWN, 127.0.0.1/8, ::1/128
oob_net0: UP, 192.168.62.174/24, fe80::a288:c2ff:fe81:a210/64
tmfifo_net0: UP, 192.168.100.2/30, fe80::21a:caff:feff:ff01/64
ib0: UP
ib1: UP
And they are identified in the Ubuntu host operating system with the following names:
tmfifo_net0: UNKNOWN, fe80::21a:caff:feff:ff02/64
ibp68s0f1: UP
ibp68s0f0: UP
Additionally, some of the important variables from the mlxconfig query are as follows:
INTERNAL_CPU_MODEL EMBEDDED_CPU(1)
EXP_ROM_UEFI_ARM_ENABLE True(1)
INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0)
LINK_TYPE_P1 IB(1)
LINK_TYPE_P2 IB(1)
NVME_EMULATION_NUM_VF 0
NVME_EMULATION_NUM_PF 1
NVME_EMULATION_NUM_MSIX 0
NVME_EMULATION_NUM_VF_MSIX 0
PCI_SWITCH_EMULATION_NUM_PORT 16
VIRTIO_BLK_EMULATION_NUM_VF 0
VIRTIO_BLK_EMULATION_NUM_PF 0
VIRTIO_BLK_EMULATION_NUM_MSIX 2
VIRTIO_BLK_EMULATION_NUM_VF_MSIX 0
VIRTIO_FS_EMULATION_NUM_VF 0
VIRTIO_FS_EMULATION_NUM_PF 0
VIRTIO_FS_EMULATION_NUM_MSIX 2
VIRTIO_FS_EMULATION_NUM_VF_MSIX 0
VIRTIO_NET_EMULATION_NUM_VF 0
VIRTIO_NET_EMULATION_NUM_PF 0
VIRTIO_NET_EMULATION_NUM_MSIX 2
VIRTIO_NET_EMULATION_NUM_VF_MSIX 0
PF_NUM_OF_VF_VALID True(1)
PF_NUM_PF_MSIX_VALID False(0)
PER_PF_NUM_SF True(1)
STRICT_VF_MSIX_NUM False(0)
NUM_PF_MSIX_VALID True(1)
NUM_OF_VFS 4
NUM_OF_PF 3
NUM_PF_MSIX 127
NUM_VF_MSIX 11
NUM_OF_VL_P1 _4_VLs(3)
NUM_OF_TC_P1 _8_TCs(0)
NUM_OF_PFC_P1 8
NUM_OF_VL_P2 _4_VLs(3)
NUM_OF_TC_P2 _8_TCs(0)
NUM_OF_PFC_P2 8
PF_NUM_PF_MSIX 63
PF_NUM_OF_VF 32
SRIOV_EN True(1)
SRIOV_IB_ROUTING_MODE_P1 LID(1)
SRIOV_IB_ROUTING_MODE_P2 LID(1)
SAFE_MODE_THRESHOLD 10
SAFE_MODE_ENABLE True(1)
Our challenges: Despite following the documentation step by step, we have still not been able to create VFs, SFs, and representors to perform network offloading over Infiniband. When executing the iperf3 command between the two Ubuntu systems on the host server, the load is being applied to the host server’s CPU instead of the DPU processor.