BlueField-2 host connectivity problem

Infrastructure & Networking DOCA Enterprise Networking
1

Hello,

I have some issues to connect host to BlueField-2. I will appreciate if you can help on this to me.

My HOST configuration is as follows:

mashemat@n017:~$ sudo mst status -v
MST modules:

MST PCI module is not loaded
MST PCI configuration module loaded

PCI devices:

DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField2(rev:1) /dev/mst/mt41686_pciconf0.1 c1:00.1 mlx5_3 net-enp193s0f1 1

BlueField2(rev:1) /dev/mst/mt41686_pciconf0 c1:00.0 mlx5_2 net-ib0 1

ConnectX6(rev:0) /dev/mst/mt4123_pciconf0.1 a1:00.1 mlx5_5 net-ib2 1

ConnectX6(rev:0) /dev/mst/mt4123_pciconf0 a1:00.0 mlx5_4 net-ib1 1

ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 21:00.1 mlx5_1 net-enp33s0f1 0

ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 21:00.0 mlx5_0 net-enp33s0f0 0

Cable devices:

mt41686_pciconf0.1_cable_1
mt41686_pciconf0_cable_0
mt4123_pciconf0_cable_0
mt4117_pciconf0_cable_0

mashemat@n017:~$ sudo mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -i internal_cpu_model
INTERNAL_CPU_MODEL EMBEDDED_CPU(1)

mashemat@n017:~$ ifconfig
enp193s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.10 netmask 255.255.255.0 broadcast 192.168.200.255
ether b8:ce:f6:e6:ec:4d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7470 bytes 448200 (448.2 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp33s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.128.1.17 netmask 255.255.254.0 broadcast 10.128.1.255
ether b8:ce:f6:ce:e2:20 txqueuelen 1000 (Ethernet)
RX packets 3358949 bytes 3126947094 (3.1 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3589130 bytes 3621327041 (3.6 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp33s0f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.200.20 netmask 255.255.255.0 broadcast 192.168.200.255
ether b8:ce:f6:ce:e2:21 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp68s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.30 netmask 255.255.255.0 broadcast 192.168.200.255
ether 3c:ec:ef:5e:8b:d6 txqueuelen 1000 (Ethernet)
RX packets 23517 bytes 1716930 (1.7 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp68s0f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.200.40 netmask 255.255.255.0 broadcast 192.168.200.255
ether 3c:ec:ef:5e:8b:d7 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ib1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 10.128.3.17 netmask 255.255.254.0 broadcast 10.128.3.255
unspec 20-00-10-49-FE-80-00-00-00-00-00-00-00-00-00-00 txqueuelen 256 (UNSPEC)
RX packets 39467 bytes 6391446 (6.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 63690 bytes 5477612 (5.4 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 345006 bytes 28018594 (28.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 345006 bytes 28018594 (28.0 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

tmfifo_net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255
ether 00:1a:ca:ff:ff:02 txqueuelen 1000 (Ethernet)
RX packets 5641 bytes 900292 (900.2 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9332 bytes 784838 (784.8 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

my DPU configuration is as follows:

ubuntu@n17-dpu:~$ ifconfig
en3f1pf1sf0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.1 netmask 255.255.255.0 broadcast 192.168.200.255
ether 22:1b:62:d8:04:8c txqueuelen 1000 (Ethernet)
RX packets 4980 bytes 1587026 (1.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp3s0f1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.50 netmask 255.255.255.0 broadcast 192.168.200.255
inet6 fe80::87:abff:fe54:8917 prefixlen 64 scopeid 0x20
ether 02:87:ab:54:89:17 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4980 bytes 1587026 (1.5 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 75 bytes 8106 (8.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 75 bytes 8106 (8.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

oob_net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.128.1.21 netmask 255.255.254.0 broadcast 10.128.1.255
inet6 fe80::bace:f6ff:fee6:ec56 prefixlen 64 scopeid 0x20
ether b8:ce:f6:e6:ec:56 txqueuelen 1000 (Ethernet)
RX packets 619415 bytes 60136794 (60.1 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 87347 bytes 7813013 (7.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

p1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether b8:ce:f6:e6:ec:51 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

pf1hpf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::a8b5:7dff:fe6f:a6b4 prefixlen 64 scopeid 0x20
ether aa:b5:7d:6f:a6:b4 txqueuelen 1000 (Ethernet)
RX packets 7470 bytes 448200 (448.2 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

tmfifo_net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.2 netmask 255.255.255.252 broadcast 192.168.100.3
inet6 fe80::21a:caff:feff:ff01 prefixlen 64 scopeid 0x20
ether 00:1a:ca:ff:ff:01 txqueuelen 1000 (Ethernet)
RX packets 9435 bytes 793448 (793.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5699 bytes 910404 (910.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ubuntu@n17-dpu:~$ sudo systemctl status openvswitch-switch
● openvswitch-switch.service - LSB: Open vSwitch switch
Loaded: loaded (/etc/init.d/openvswitch-switch; generated)
Active: active (running) since Mon 2022-03-07 13:46:22 UTC; 50min ago
Docs: man:systemd-sysv-generator(8)
Process: 636766 ExecStart=/etc/init.d/openvswitch-switch start (code=exited, status=0/SUCCESS)
Tasks: 14 (limit: 19077)
Memory: 196.9M
CGroup: /system.slice/openvswitch-switch.service
├─636822 ovsdb-server: monitoring pid 636823 (healthy)
├─636823 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:O>
├─636839 ovs-vswitchd: monitoring pid 636840 (healthy)
└─636840 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswit>

Mar 07 13:46:21 n17-dpu systemd[1]: Starting LSB: Open vSwitch switch…
Mar 07 13:46:21 n17-dpu openvswitch-switch[636802]: * Starting ovsdb-server
Mar 07 13:46:21 n17-dpu ovs-vsctl[636824]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait – init – set Open_vSwitch . db-version=8.3.0
Mar 07 13:46:21 n17-dpu ovs-vsctl[636829]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.15.1 “external-ids:system-id="0f4d895c-2f66-4994-bb5f-37a7adc04a38"” "external->
Mar 07 13:46:21 n17-dpu openvswitch-switch[636802]: * Configuring Open vSwitch system IDs
Mar 07 13:46:22 n17-dpu openvswitch-switch[636802]: * Starting ovs-vswitchd
Mar 07 13:46:22 n17-dpu ovs-vsctl[636857]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait add Open_vSwitch . external-ids hostname=localhost.localdomain
Mar 07 13:46:22 n17-dpu openvswitch-switch[636802]: * Enabling remote OVSDB managers
Mar 07 13:46:22 n17-dpu systemd[1]: Started LSB: Open vSwitch switch.
ubuntu@n17-dpu:~$ sudo ovs-vsctl show
57d70fe0-b54e-4a77-8720-eb96d26d5744
Bridge ovsbr1
Port pf1hpf
Interface pf1hpf
Port ovsbr1
Interface ovsbr1
type: internal
Port en3f1pf1sf0
Interface en3f1pf1sf0
Port p1
Interface p1
Port enp3s0f1s0
Interface enp3s0f1s0
ovs_version: “2.15.1”

ubuntu@n17-dpu $ sudo ovs-ofctl dump-flows ovsbr1
cookie=0x0, duration=3083.972s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL

When I try to ping from host to DPU:
mashemat@n017:~$ ping 192.168.200.1

PING 192.168.200.1 (192.168.200.1) 56(84) bytes of data.

From 192.168.200.10 icmp_seq=1 Destination Host Unreachable
From 192.168.200.10 icmp_seq=2 Destination Host Unreachable
From 192.168.200.10 icmp_seq=3 Destination Host Unreachable

It does not connect. But the packet arrives up to pf1hpf

pf1hpf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::a8b5:7dff:fe6f:a6b4 prefixlen 64 scopeid 0x20
ether aa:b5:7d:6f:a6:b4 txqueuelen 1000 (Ethernet)
RX packets 7479 bytes 448740 (448.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

But it does not reach to any of interfaces or even outside.

Could you please let em know where would be possible issue?

Thanks,

2

Same issue here…

3

My issue was solved. The personality of the ports were not match with the switch. Try to check the link type as follows:

test@n018:~$ sudo mlxconfig query |grep -e LINK_TYPE -e “Device.*mst”
Device: /dev/mst/mt41686_pciconf0
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 IB(1)

Device: /dev/mst/mt4123_pciconf0
LINK_TYPE_P1 IB(1)
LINK_TYPE_P2 IB(1)
Device: /dev/mst/mt4117_pciconf0

They should be connected correctly to the proper switch (IB/ETH).

Thanks,

4

I see enp3s0f1s0 is in Bridge ovsbr1.
Is it a must?
It’s absent in Bridge ovsbr1 on my testbed.
And there’s no output “$ sudo mlxconfig query |grep -e LINK_TYPE” on my test bed.
Thanks.

5

Hi wangyw,

No actually, when I reboot the DPU it back to the default config:

ubuntu@n19-dpu :~$ sudo ovs-dpctl show
system@ovs-system:
lookups: hit:108569 missed:17973 lost:3
flows: 1
masks: hit:126532 total:1 hit/pkt:1.00
port 0: ovs-system (internal)
port 1: ovsbr1 (internal)
port 2: en3f0pf0sf0
port 3: p0
port 4: pf0hpf

Here is my query:

mashemat@n019:~$ sudo mlxconfig query |grep -e LINK_TYPE
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 IB(1)

     LINK_TYPE_P1                        IB(1)           
     LINK_TYPE_P2                        IB(1)

Seems you dont see your links.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.