Network access in the embedded mode

Infrastructure & Networking DOCA Enterprise Networking
1

Hi all,
I’m having difficulty for network access on the embedded mode.
My Bluefield2 smart nic can ping the host and host can ping bluefield2.
But I don’t have network connection.
Can some one explain how to use the ovs bridges to set the route for network connection?

ovs-vsctl show

31d2a29e-d043-4f8b-94d0-5dd4d47198d7
Bridge ovsbr2
Port p1
Interface p1
Port pf1hpf
Interface pf1hpf
Port ovsbr2
Interface ovsbr2
type: internal
Port en3f1pf1sf0
Interface en3f1pf1sf0
Bridge ovsbr1
Port p0
Interface p0
Port ovsbr1
Interface ovsbr1
type: internal
Port en3f0pf0sf0
Interface en3f0pf0sf0
Port pf0hpf
Interface pf0hpf
ovs_version: “2.14.1”

2

Hey there. This guide might help demystify all of these interfaces in the ovs bridge: vSwitch and Representors Model :: NVIDIA DOCA SDK Documentation

When in ECPF ownership mode, we see 2 representors for each one of the DPU’s network ports: one for the uplink, and another one for the host side PF (the PF representor created even if the PF is not probed on the host side).

if you check the output of mlnx-sf -a show on the DPU you’ll see the kernel netdev and the representor names for the DPU itself.

root@localhost:/home/ubuntu# mlnx-sf -a show
 
SF Index: pci/0000:03:00.0/294928
  Parent PCI dev: 0000:03:00.0
  Representor netdev: en3f0pf0sf0
  Function HWADDR: 02:36:6a:1e:4d:9d
  Auxiliary device: mlx5_core.sf.2
    netdev: enp3s0f0s0
    RDMA dev: mlx5_2
 
SF Index: pci/0000:03:00.1/360464
  Parent PCI dev: 0000:03:00.1
  Representor netdev: en3f1pf1sf0
  Function HWADDR: 02:33:e3:2f:c8:42
  Auxiliary device: mlx5_core.sf.3
    netdev: enp3s0f1s0
    RDMA dev: mlx5_3

The Representor netdev devices are already connected (added to) the OVS bridge by default, so we only need to use netplan to assign an IP address to enp3s0f1s0 or enp3s0f0s0 (or both) and that should work. Here’s an example netplan config:

root@localhost:/home/ubuntu# cat /etc/netplan/60-mlnx.yaml 
network:
  ethernets:
    enp3s0f0s0:
      dhcp4: no
      addresses:
        - X.X.X.X/24
      gateway4: X.X.X.X
      nameservers:
          addresses: [X.X.X.X, Y.Y.Y.Y]
    enp3s0f1s0:
      dhcp4: 'false'
  renderer: networkd
  version: 2
1 Like
3

Thanks Justine for the clarification,
one follow up question: does the oob_net0 give us network connectivity only on the separated mode? How should we set the oob_net0 interface to give us network access? Our oob_net0 interface is connected to VLAN, but we still can’t perform apt-get update to finish doca installation.
Also in the link you shared with us, what are p3p1 and p4p2 interfaces? Are these internal interfaces that connect bluefield to the host? or the interfaces that connect bluefield to the switch?

4

The oob_net0 interface is meant to be used to connect into an IMPI network that’s typically 1gbps copper. Most data centers have a low speed 1gbps out of band management network.

For the apt update to work, DNS resolution must work and you must have a route to the internet. Do you have DNS servers configured and do you have a route to reach the configured DNS servers?

5

Hi Justin! My name is Anu, and I work with Diman. So we have everything setup correctly now I think. This is the setup right now:

Smartnic on embedded mode:

Host ens4f0 x.x.231.115 <------> pf0hpf x.x.231.116 SmartNIC
ens4f1 x.x.231.117 <------> pf1hpf x.x.231.118

Bus info Device Class Description

pci@0000:a3:00.0 ens4f0 network MT42822 BlueField-2 integrated ConnectX-6 Dx network controller
pci@0000:a3:00.1 ens4f1 network MT42822 BlueField-2 integrated ConnectX-6 Dx network controller

We used netplan to set the IP addresses and gateway. The DNS is the same as the host’s DNS. The host is able to ping the DNS server but from the smartnic , we are unable to ping the DNS server. We are behind proxy.

And the ovs-vsctl show :
root@localhost:~# ovs-vsctl show
c8599771-cb29-432e-9087-070acfe56d20
Bridge ovsbr1
Port p0
Interface p0
Port ovsbr1
Interface ovsbr1
type: internal
Port en3f0pf0sf0
Interface en3f0pf0sf0
Port pf0hpf
Interface pf0hpf
Bridge ovsbr2
Port pf1hpf
Interface pf1hpf
Port p1
Interface p1
Port ovsbr2
Interface ovsbr2
type: internal
Port en3f1pf1sf0
Interface en3f1pf1sf0
ovs_version: “2.14.1”

Can you tell us where we could be going wrong? The host can ping the DNS but the smartnic cannot ping the IP assigned to the host side as well it cannot ping the DNS.

6

Also, as per this doc: Virtual Switch on BlueField SmartNIC - BlueField SW Manual v2.0.1.10841 - NVIDIA Networking Docs

We need to have openvswitch service working but for some reason, this service is not found on the smartnic ubuntu OS:

root@localhost:~# service openvswitch start
Failed to start openvswitch.service: Unit openvswitch.service not found.

root@localhost:~# systemctl restart openvswitch
Failed to restart openvswitch.service: Unit openvswitch.service not found.

How do we enable openvswtich on the smartnic? Could this be the issue?

1 Like
7

Hi Anu,
Try $ systemctl status openvswitch-switch.service

Regarding the ping issue.
The topology is not very clear to me.
The host is connected to a BF2 with a PCI slot. Did you connect the BF2 ports to a switch which has access to the DNS server? is it possible the server you’re using has an additional management port (not related to the BF2) which is allows access to the DNS server? Can you dump the routing table for both the host and the ARM device using ‘route -n’?

Thanks!

2 Likes
8

Dear Shaulia,

Justin had a call with us and we were able to fix the issue. Thank you very much for your response.

Thank you,
Anu

2 Likes
9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.