connect/RDMA between host and DPU

[cxinyic]

I am using the cloudlab r7525 node and the bluefield 2 on it.
I follow the suggestion from my previous post to install the DOCA on the DPU.
https://docs.nvidia.com/doca/sdk/installation-guide-for-linux/index.html#installing-software-on-host

Now I can ssh to the DPU. However, there is no network between the host and the DPU. I want to use RDMA between them.
The ip link on the dpu is shown as below:
ubuntu@localhost:~$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tmfifo_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 00:1a:ca:ff:ff:01 brd ff:ff:ff:ff:ff:ff
3: oob_net0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:42:a1:da:aa:50 brd ff:ff:ff:ff:ff:ff
4: p0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:42:a1:da:aa:4a brd ff:ff:ff:ff:ff:ff
5: p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:42:a1:da:aa:4b brd ff:ff:ff:ff:ff:ff
6: pf0hpf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether 0a:b7:7b:37:08:6e brd ff:ff:ff:ff:ff:ff
7: pf1hpf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether c6:68:bf:e8:78:66 brd ff:ff:ff:ff:ff:ff
8: en3f0pf0sf0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether 62:68:db:37:8d:38 brd ff:ff:ff:ff:ff:ff
9: enp3s0f0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 02:69:57:54:5b:91 brd ff:ff:ff:ff:ff:ff
10: en3f1pf1sf0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether 92:28:11:e6:78:b3 brd ff:ff:ff:ff:ff:ff
11: enp3s0f1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 02:39:79:94:6d:95 brd ff:ff:ff:ff:ff:ff
12: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 2a:3b:d7:f4:c3:47 brd ff:ff:ff:ff:ff:ff
13: ovsbr1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:42:a1:da:aa:4a brd ff:ff:ff:ff:ff:ff
14: ovsbr2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:42:a1:da:aa:4b brd ff:ff:ff:ff:ff:ff

The ip link on the host is shown below:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 4c:d9:8f:22:01:d8 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether 4c:d9:8f:22:01:d9 brd ff:ff:ff:ff:ff:ff
13: tmfifo_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 00:1a:ca:ff:ff:02 brd ff:ff:ff:ff:ff:ff
18: eno33np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 1c:34:da:72:f9:3e brd ff:ff:ff:ff:ff:ff
19: eno34np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 1c:34:da:72:f9:3f brd ff:ff:ff:ff:ff:ff
20: ens5f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 0c:42:a1:da:aa:46 brd ff:ff:ff:ff:ff:ff
21: ens5f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 0c:42:a1:da:aa:47 brd ff:ff:ff:ff:ff:ff

I have assigned ip to both ens5f0np0 and ens5f1np1 on the host. And I have tried to assign ip to the DPU’s interface(pf0hpf/enp3s0f0s0). None of them can work. I wonder what might be a possible solution or cause for this?

[ssimcoejr]

Hello cxinyic,

Thank you for posting your inquiry to the NVIDIA Developer Forums.

Please review the ‘No connectivity between network interfaces of source host to destination device’ section of the ‘Connectivity Troubleshooting’ section of the Bluefield OS guide:
https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Connectivity+Troubleshooting

Please also review the ‘Verifying Host Connection on Linux’ and ‘Verifying Connection from Host to BlueField’ section in the ‘Virtual Switch on DPU’ section for host-to-ARM connection validation:
https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Virtual+Switch+on+DPU

If after following those guidelines, you are still unable to configure host-to-DPU networking, and you have valid support entitlement, we recommend opening a support ticket with Enterprise Support via the following link: https://enterprise-support.nvidia.com/s/create-case

Thanks, and best regards,
NVIDIA Enterprise Experience