I want to run the RDMA program in the Docker container, but after following the process of configuration inhttps://enterprise-support.nvidia.com/s/article/Docker-RDMA-SRIOV-Networking-with-ConnectX4-ConnectX5-ConnectX6, I found that the VF bound in the Docker container can not communicate with the outside world, it can not even communicate with the PF on the host, why？I wonder the reason, please.
The Docker container network configuration is as follows:
root@a9862f372e2c:/# show_gids
DEV PORT INDEX GID IPv4 VER DEV
mlx5_17 1 0 fe80:0000:0000:0000:e4ec:18ff:fef6:08a8 v1 eth0
mlx5_17 1 1 fe80:0000:0000:0000:e4ec:18ff:fef6:08a8 v2 eth0
mlx5_17 1 2 0000:0000:0000:0000:0000:ffff:c00a:0b65 192.10.11.101 v1 eth0
mlx5_17 1 3 0000:0000:0000:0000:0000:ffff:c00a:0b65 192.10.11.101 v2 eth0
n_gids_found=4
root@a9862f372e2c:/# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.10.11.101 netmask 255.255.255.0 broadcast 192.10.11.255
ether e6:ec:18:f6:08:a8 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 84 bytes 11659 (11.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
The Dcoker network and host network configuration are as follows:
zxy@guolab10:~$ sudo docker network inspect srnet
[
{
"Name": "srnet",
"Id": "7c3b4ba3b5699507028d15d13448860113baddb31a994e48803c669009777e28",
"Created": "2024-01-10T11:04:01.194009998+08:00",
"Scope": "local",
"Driver": "sriov",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "192.10.11.0/24"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"a9862f372e2c6f2f159336e57400c065390ce455f6e7643a3c6dd4984a86df71": {
"Name": "pensive_knuth",
"EndpointID": "0c05bd6a7b6d20720f34e5f56f404010a3fc244f3c987df27a62ef8401e06ef8",
"MacAddress": "e6:ec:18:f6:08:a8",
"IPv4Address": "192.10.11.101/24",
"IPv6Address": ""
}
},
"Options": {
"netdevice": "enp5s0f1np1",
"privileged": "1"
},
"Labels": {}
}
]
zxy@guolab10:~$ ip link show enp5s0f1np1
57: enp5s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b8:ce:f6:cf:97:99 brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 12:e6:06:ea:bb:60 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 1 link/ether 86:76:02:1f:17:89 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 2 link/ether 5a:58:27:f7:8b:ac brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 3 link/ether b2:b9:4e:55:7e:53 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 4 link/ether ba:10:bf:37:8f:70 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 5 link/ether b6:d6:6f:9a:cc:ff brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 6 link/ether f2:ac:53:de:e6:35 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 7 link/ether 16:63:af:c5:4d:31 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 8 link/ether 8e:17:0a:22:ea:2f brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 9 link/ether 0e:6d:62:f8:e7:8f brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 10 link/ether 02:92:09:e8:d1:b1 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 11 link/ether 0a:f9:2a:48:46:fb brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 12 link/ether da:c8:80:e4:98:9a brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 13 link/ether 6e:30:6e:c2:98:89 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 14 link/ether 3a:64:ef:25:7a:01 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 15 link/ether e6:ec:18:f6:08:a8 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
zxy@guolab10:~$ ifconfig enp5s0f1np1
enp5s0f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.10.11.10 netmask 255.255.255.0 broadcast 192.10.11.255
ether b8:ce:f6:cf:97:99 txqueuelen 1000 (Ethernet)
RX packets 2833 bytes 880274 (880.2 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 291 bytes 38456 (38.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
zxy@guolab10:~$ show_gids
DEV PORT INDEX GID IPv4 VER DEV
--- ---- ----- --- ------------ --- ---
mlx5_1 1 0 fe80:0000:0000:0000:bace:f6ff:fecf:9799 v1 enp5s0f1np1
mlx5_1 1 1 fe80:0000:0000:0000:bace:f6ff:fecf:9799 v2 enp5s0f1np1
mlx5_1 1 2 0000:0000:0000:0000:0000:ffff:c00a:0b0a 192.10.11.10 v1 enp5s0f1np1
mlx5_1 1 3 0000:0000:0000:0000:0000:ffff:c00a:0b0a 192.10.11.10 v2 enp5s0f1np1
mlx5_10 1 0 fe80:0000:0000:0000:8c17:0aff:fe22:ea2f v1 enp5s0f1v8
mlx5_10 1 1 fe80:0000:0000:0000:8c17:0aff:fe22:ea2f v2 enp5s0f1v8
mlx5_11 1 0 fe80:0000:0000:0000:0c6d:62ff:fef8:e78f v1 enp5s0f1v9
mlx5_11 1 1 fe80:0000:0000:0000:0c6d:62ff:fef8:e78f v2 enp5s0f1v9
mlx5_11 1 2 fe80:0000:0000:0000:54a9:fbf1:57fd:a46e v1 enp5s0f1v9
mlx5_11 1 3 fe80:0000:0000:0000:54a9:fbf1:57fd:a46e v2 enp5s0f1v9
mlx5_12 1 0 fe80:0000:0000:0000:0092:09ff:fee8:d1b1 v1 enp5s0f1v10
mlx5_12 1 1 fe80:0000:0000:0000:0092:09ff:fee8:d1b1 v2 enp5s0f1v10
mlx5_13 1 0 fe80:0000:0000:0000:08f9:2aff:fe48:46fb v1 enp5s0f1v11
mlx5_13 1 1 fe80:0000:0000:0000:08f9:2aff:fe48:46fb v2 enp5s0f1v11
mlx5_14 1 0 fe80:0000:0000:0000:d8c8:80ff:fee4:989a v1 enp5s0f1v12
mlx5_14 1 1 fe80:0000:0000:0000:d8c8:80ff:fee4:989a v2 enp5s0f1v12
mlx5_15 1 0 fe80:0000:0000:0000:6c30:6eff:fec2:9889 v1 enp5s0f1v13
mlx5_15 1 1 fe80:0000:0000:0000:6c30:6eff:fec2:9889 v2 enp5s0f1v13
mlx5_16 1 0 fe80:0000:0000:0000:3864:efff:fe25:7a01 v1 enp5s0f1v14
mlx5_16 1 1 fe80:0000:0000:0000:3864:efff:fe25:7a01 v2 enp5s0f1v14
mlx5_2 1 0 fe80:0000:0000:0000:10e6:06ff:feea:bb60 v1 enp5s0f1v0
mlx5_2 1 1 fe80:0000:0000:0000:10e6:06ff:feea:bb60 v2 enp5s0f1v0
mlx5_2 1 2 fe80:0000:0000:0000:faec:6aa1:15dd:7a1e v1 enp5s0f1v0
mlx5_2 1 3 fe80:0000:0000:0000:faec:6aa1:15dd:7a1e v2 enp5s0f1v0
mlx5_3 1 0 fe80:0000:0000:0000:8476:02ff:fe1f:1789 v1 enp5s0f1v1
mlx5_3 1 1 fe80:0000:0000:0000:8476:02ff:fe1f:1789 v2 enp5s0f1v1
mlx5_4 1 0 fe80:0000:0000:0000:5858:27ff:fef7:8bac v1 enp5s0f1v2
mlx5_4 1 1 fe80:0000:0000:0000:5858:27ff:fef7:8bac v2 enp5s0f1v2
mlx5_4 1 2 fe80:0000:0000:0000:0fc8:5452:12c3:ee5b v1 enp5s0f1v2
mlx5_4 1 3 fe80:0000:0000:0000:0fc8:5452:12c3:ee5b v2 enp5s0f1v2
mlx5_5 1 0 fe80:0000:0000:0000:b0b9:4eff:fe55:7e53 v1 enp5s0f1v3
mlx5_5 1 1 fe80:0000:0000:0000:b0b9:4eff:fe55:7e53 v2 enp5s0f1v3
mlx5_6 1 0 fe80:0000:0000:0000:b810:bfff:fe37:8f70 v1 enp5s0f1v4
mlx5_6 1 1 fe80:0000:0000:0000:b810:bfff:fe37:8f70 v2 enp5s0f1v4
mlx5_7 1 0 fe80:0000:0000:0000:b4d6:6fff:fe9a:ccff v1 enp5s0f1v5
mlx5_7 1 1 fe80:0000:0000:0000:b4d6:6fff:fe9a:ccff v2 enp5s0f1v5
mlx5_8 1 0 fe80:0000:0000:0000:f0ac:53ff:fede:e635 v1 enp5s0f1v6
mlx5_8 1 1 fe80:0000:0000:0000:f0ac:53ff:fede:e635 v2 enp5s0f1v6
mlx5_9 1 0 fe80:0000:0000:0000:1463:afff:fec5:4d31 v1 enp5s0f1v7
mlx5_9 1 1 fe80:0000:0000:0000:1463:afff:fec5:4d31 v2 enp5s0f1v7
n_gids_found=44
The Docker container is bound to VF 15 (mlx5_17).
When I communicate with another host with IP=192.10.11.9 using PF (enp5s0f1np1, IP=192.10.11.10) on the host and VF (IP=192.10.11.101) for Docker container bindings respectively, here’s what happens:
zxy@guolab10:~$ ping 192.10.11.9 -I enp5s0f1np1
PING 192.10.11.9 (192.10.11.9) from 192.10.11.10 enp5s0f1np1: 56(84) bytes of data.
64 bytes from 192.10.11.9: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.10.11.9: icmp_seq=2 ttl=64 time=0.091 ms
64 bytes from 192.10.11.9: icmp_seq=3 ttl=64 time=0.088 ms
64 bytes from 192.10.11.9: icmp_seq=4 ttl=64 time=0.077 ms
root@a9862f372e2c:/# ping 192.10.11.9 -I eth0
PING 192.10.11.9 (192.10.11.9) from 192.10.11.101 eth0: 56(84) bytes of data.
From 192.10.11.101 icmp_seq=1 Destination Host Unreachable
From 192.10.11.101 icmp_seq=2 Destination Host Unreachable
From 192.10.11.101 icmp_seq=3 Destination Host Unreachable
From 192.10.11.101 icmp_seq=4 Destination Host Unreachable
You can see that the PF on one host can talk to another host, but the VF attached to the Docker container can’t. The RNIC of the two hosts are connected directly using a Network Cable without passing through a switch.
When I used PF (enp5s0f1np1,IP=192.10.11.10) on the host machine and VF (IP=192.10.11.101) on the Docker container to talk to each other, they also failed to ping
root@a9862f372e2c:/# ping 192.10.11.10
PING 192.10.11.10 (192.10.11.10) 56(84) bytes of data.
From 192.10.11.101 icmp_seq=1 Destination Host Unreachable
From 192.10.11.101 icmp_seq=5 Destination Host Unreachable
From 192.10.11.101 icmp_seq=6 Destination Host Unreachable
zxy@guolab10:~$ ping 192.10.11.101 -I enp5s0f1np1
PING 192.10.11.101 (192.10.11.101) from 192.10.11.10 enp5s0f1np1: 56(84) bytes of data.
From 192.10.11.10 icmp_seq=1 Destination Host Unreachable
From 192.10.11.10 icmp_seq=2 Destination Host Unreachable
From 192.10.11.10 icmp_seq=3 Destination Host Unreachable
I want Docker to be able to communicate with the PF of the local host and the other host through the bound VF. Please tell me what configuration should be changed.