BlueField-2 host connectivity problem

Hello,

I have some issues to connect host to BlueField-2. I will appreciate if you can help on this to me.

My HOST configuration is as follows:

mashemat@n017:~$ sudo mst status -v
MST modules:

MST PCI module is not loaded
MST PCI configuration module loaded

PCI devices:

DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField2(rev:1) /dev/mst/mt41686_pciconf0.1 c1:00.1 mlx5_3 net-enp193s0f1 1

BlueField2(rev:1) /dev/mst/mt41686_pciconf0 c1:00.0 mlx5_2 net-ib0 1

ConnectX6(rev:0) /dev/mst/mt4123_pciconf0.1 a1:00.1 mlx5_5 net-ib2 1

ConnectX6(rev:0) /dev/mst/mt4123_pciconf0 a1:00.0 mlx5_4 net-ib1 1

ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 21:00.1 mlx5_1 net-enp33s0f1 0

ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 21:00.0 mlx5_0 net-enp33s0f0 0

Cable devices:

mt41686_pciconf0.1_cable_1
mt41686_pciconf0_cable_0
mt4123_pciconf0_cable_0
mt4117_pciconf0_cable_0

mashemat@n017:~$ sudo mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -i internal_cpu_model
INTERNAL_CPU_MODEL EMBEDDED_CPU(1)

mashemat@n017:~$ ifconfig
enp193s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.10 netmask 255.255.255.0 broadcast 192.168.200.255
ether b8:ce:f6:e6:ec:4d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7470 bytes 448200 (448.2 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp33s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.128.1.17 netmask 255.255.254.0 broadcast 10.128.1.255
ether b8:ce:f6:ce:e2:20 txqueuelen 1000 (Ethernet)
RX packets 3358949 bytes 3126947094 (3.1 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3589130 bytes 3621327041 (3.6 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp33s0f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.200.20 netmask 255.255.255.0 broadcast 192.168.200.255
ether b8:ce:f6:ce:e2:21 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp68s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.30 netmask 255.255.255.0 broadcast 192.168.200.255
ether 3c:ec:ef:5e:8b:d6 txqueuelen 1000 (Ethernet)
RX packets 23517 bytes 1716930 (1.7 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp68s0f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.200.40 netmask 255.255.255.0 broadcast 192.168.200.255
ether 3c:ec:ef:5e:8b:d7 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ib1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 10.128.3.17 netmask 255.255.254.0 broadcast 10.128.3.255
unspec 20-00-10-49-FE-80-00-00-00-00-00-00-00-00-00-00 txqueuelen 256 (UNSPEC)
RX packets 39467 bytes 6391446 (6.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 63690 bytes 5477612 (5.4 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 345006 bytes 28018594 (28.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 345006 bytes 28018594 (28.0 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

tmfifo_net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255
ether 00:1a:ca:ff:ff:02 txqueuelen 1000 (Ethernet)
RX packets 5641 bytes 900292 (900.2 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9332 bytes 784838 (784.8 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

my DPU configuration is as follows:

ubuntu@n17-dpu:~$ ifconfig
en3f1pf1sf0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.1 netmask 255.255.255.0 broadcast 192.168.200.255
ether 22:1b:62:d8:04:8c txqueuelen 1000 (Ethernet)
RX packets 4980 bytes 1587026 (1.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp3s0f1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.50 netmask 255.255.255.0 broadcast 192.168.200.255
inet6 fe80::87:abff:fe54:8917 prefixlen 64 scopeid 0x20
ether 02:87:ab:54:89:17 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4980 bytes 1587026 (1.5 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 75 bytes 8106 (8.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 75 bytes 8106 (8.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

oob_net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.128.1.21 netmask 255.255.254.0 broadcast 10.128.1.255
inet6 fe80::bace:f6ff:fee6:ec56 prefixlen 64 scopeid 0x20
ether b8:ce:f6:e6:ec:56 txqueuelen 1000 (Ethernet)
RX packets 619415 bytes 60136794 (60.1 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 87347 bytes 7813013 (7.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

p1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether b8:ce:f6:e6:ec:51 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

pf1hpf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::a8b5:7dff:fe6f:a6b4 prefixlen 64 scopeid 0x20
ether aa:b5:7d:6f:a6:b4 txqueuelen 1000 (Ethernet)
RX packets 7470 bytes 448200 (448.2 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

tmfifo_net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.2 netmask 255.255.255.252 broadcast 192.168.100.3
inet6 fe80::21a:caff:feff:ff01 prefixlen 64 scopeid 0x20
ether 00:1a:ca:ff:ff:01 txqueuelen 1000 (Ethernet)
RX packets 9435 bytes 793448 (793.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5699 bytes 910404 (910.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ubuntu@n17-dpu:~$ sudo systemctl status openvswitch-switch
● openvswitch-switch.service - LSB: Open vSwitch switch
Loaded: loaded (/etc/init.d/openvswitch-switch; generated)
Active: active (running) since Mon 2022-03-07 13:46:22 UTC; 50min ago
Docs: man:systemd-sysv-generator(8)
Process: 636766 ExecStart=/etc/init.d/openvswitch-switch start (code=exited, status=0/SUCCESS)
Tasks: 14 (limit: 19077)
Memory: 196.9M
CGroup: /system.slice/openvswitch-switch.service
├─636822 ovsdb-server: monitoring pid 636823 (healthy)
├─636823 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:O>
├─636839 ovs-vswitchd: monitoring pid 636840 (healthy)
└─636840 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswit>

Mar 07 13:46:21 n17-dpu systemd[1]: Starting LSB: Open vSwitch switch…
Mar 07 13:46:21 n17-dpu openvswitch-switch[636802]: * Starting ovsdb-server
Mar 07 13:46:21 n17-dpu ovs-vsctl[636824]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait – init – set Open_vSwitch . db-version=8.3.0
Mar 07 13:46:21 n17-dpu ovs-vsctl[636829]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.15.1 "external-ids:system-id=“0f4d895c-2f66-4994-bb5f-37a7adc04a38"” "external->
Mar 07 13:46:21 n17-dpu openvswitch-switch[636802]: * Configuring Open vSwitch system IDs
Mar 07 13:46:22 n17-dpu openvswitch-switch[636802]: * Starting ovs-vswitchd
Mar 07 13:46:22 n17-dpu ovs-vsctl[636857]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait add Open_vSwitch . external-ids hostname=localhost.localdomain
Mar 07 13:46:22 n17-dpu openvswitch-switch[636802]: * Enabling remote OVSDB managers
Mar 07 13:46:22 n17-dpu systemd[1]: Started LSB: Open vSwitch switch.
ubuntu@n17-dpu:~$ sudo ovs-vsctl show
57d70fe0-b54e-4a77-8720-eb96d26d5744
Bridge ovsbr1
Port pf1hpf
Interface pf1hpf
Port ovsbr1
Interface ovsbr1
type: internal
Port en3f1pf1sf0
Interface en3f1pf1sf0
Port p1
Interface p1
Port enp3s0f1s0
Interface enp3s0f1s0
ovs_version: “2.15.1”

ubuntu@n17-dpu $ sudo ovs-ofctl dump-flows ovsbr1
cookie=0x0, duration=3083.972s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL

When I try to ping from host to DPU:
mashemat@n017:~$ ping 192.168.200.1

PING 192.168.200.1 (192.168.200.1) 56(84) bytes of data.

From 192.168.200.10 icmp_seq=1 Destination Host Unreachable
From 192.168.200.10 icmp_seq=2 Destination Host Unreachable
From 192.168.200.10 icmp_seq=3 Destination Host Unreachable

It does not connect. But the packet arrives up to pf1hpf

pf1hpf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::a8b5:7dff:fe6f:a6b4 prefixlen 64 scopeid 0x20
ether aa:b5:7d:6f:a6:b4 txqueuelen 1000 (Ethernet)
RX packets 7479 bytes 448740 (448.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

But it does not reach to any of interfaces or even outside.

Could you please let em know where would be possible issue?

Thanks,

Same issue here…

My issue was solved. The personality of the ports were not match with the switch. Try to check the link type as follows:

test@n018:~$ sudo mlxconfig query |grep -e LINK_TYPE -e “Device.*mst”
Device: /dev/mst/mt41686_pciconf0
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 IB(1)

Device: /dev/mst/mt4123_pciconf0
LINK_TYPE_P1 IB(1)
LINK_TYPE_P2 IB(1)
Device: /dev/mst/mt4117_pciconf0

They should be connected correctly to the proper switch (IB/ETH).

Thanks,

I see enp3s0f1s0 is in Bridge ovsbr1.
Is it a must?
It’s absent in Bridge ovsbr1 on my testbed.
And there’s no output “$ sudo mlxconfig query |grep -e LINK_TYPE” on my test bed.
Thanks.

Hi wangyw,

No actually, when I reboot the DPU it back to the default config:

ubuntu@n19-dpu :~$ sudo ovs-dpctl show
system@ovs-system:
lookups: hit:108569 missed:17973 lost:3
flows: 1
masks: hit:126532 total:1 hit/pkt:1.00
port 0: ovs-system (internal)
port 1: ovsbr1 (internal)
port 2: en3f0pf0sf0
port 3: p0
port 4: pf0hpf

Here is my query:

mashemat@n019:~$ sudo mlxconfig query |grep -e LINK_TYPE
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 IB(1)

     LINK_TYPE_P1                        IB(1)           
     LINK_TYPE_P2                        IB(1) 

Seems you dont see your links.

(post deleted by author)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.