Unable to change BF2 to switch Dev mode

Infrastructure & Networking BlueField
1

Hi Team,

I am trying to offload OVS with BF2. Trying to change the BF2 to switchdev mode but getting below error. Please advice.

2 VFs are created but are not attached to any VM.
[root@node01-1 ~]# lspci | grep -i Mel
af:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
af:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
af:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01)
af:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
af:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
unbind the VFs before changing to switchdev
[root@node01-1 ~]# ls /sys/bus/pci/drivers/mlx5_core/
0000:af:00.0 0000:af:00.1 0000:af:00.3 0000:af:00.4 bind module new_id remove_id uevent unbind
[root@node01-1 ~]# echo 0000:af:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind
[root@node01-1 ~]# echo 0000:af:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind
[root@node01-1 ~]# ls /sys/bus/pci/drivers/mlx5_core/
0000:af:00.0 0000:af:00.1 bind module new_id remove_id uevent unbind

changing to switchdev mode failed.
[root@node01-1 ~]# devlink dev eswitch set pci/0000:af:00.0 mode switchdev
kernel answers: Operation not supported

2

Hi ramachaitanya.k,

Welcome, and thanks for posting your inquiry to the NVIDIA developer forums!

In ECPF mode (which is the default mode for BF-2), the ARM device already already owns the ECPF, and the NIC is already operating in switchdev mode:

https://docs.nvidia.com/networking/display/BlueFieldSWv35011563/Modes+of+Operation#ModesofOperation-SmartNICmode

As such, this is not a valid/permitted configuration.

Best regards,
NVIDIA Enterprise Experience

3

Hi [ssimcoejr],

Thanks you for the reply.
I want to use host machine to configure the BF-2 in NIC mode and then try moving from legacy to switchdev mode. I want to avoid ARM CPU for now. Is its possible ?

I am trying “OVS-Kernel Hardware Offloads” using below document. As a initial step I am trying to use Host CPU and later I will move OVS to ARM CPU.
https://docs.nvidia.com/networking/pages/viewpage.action?pageId=129567766#OVSOffloadUsingASAP²Direct-OVS-KernelHardwareOffloads

4

Basically we are not using SRIOV in this case. VMs are communicated to the host with virtio.

5

@ramachaitanya.k : So, you want to avoid using ARM CPU and OvS running on it completely? Or you just want the OvS running on the ARM cores to be HW offloaded?

In the first case, as @ssimcoejr pointed out, your Bluefield is not in the right mode. You can switch it to SEPARATED_HOST mode, when it does not act as a SmartNIC but a normal NIC. Then, you can use it with an OvS on the Host.
I had a blog series on my experiences with Bluefield. In Part II, I explained (from sources gathered from NVidia docs, etc.) how you can change the mode of operation.

In the second case, the latest firmware for Bluefield already runs OvS-kernel with HW offload (via tc flowers) enabled. So, this means that while the ARM cores are used to run OvS process, most of the packet processing will not go through the ARM cores.

I hope this helps.

6

Hi,

Thankyou for the reply.

I didn’t see SEPERATED_HOST mode in the latest doc. I want to avoid ARM core completely. As per below document I want to move to NIC mode and then try switchdev for offload.

https://docs.nvidia.com/networking/display/BlueFieldDPUOSv395/Modes+of+Operation

7

Okay, so is SEPARATED_HOST mode the solution you were looking for? Does it work for you?
Just to make it a solution of this thread.

8

As per my understanding SEPARATED_HOST mode which runs in Old firmware won’t support switchdev mode for Offload. But the NIC mode which is in new firmware supports Offloads with host itself with out using ARM. The question is this NIC mode is not working as per the document.

9

I see. I cannot comment on whether SEPARATED_HOST is supported in the new firmware or not.
I used to run Bluefield in that mode, but recently, after installing the latest firmware, I was only dealing with EMBEDDED_CPU mode.
At the moment, I don’t have access to a Bluefield to verify (but will do).

Meanwhile, you can keep us updated here if you could advance in your setup :)

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.