Two sparks running latest 7.0.0-1019-nvidia kernal. I followed both eugr and NVIDIA’s play book to setup my 2 sparks
node 1
ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
node 2
ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
On node 1
Create the netplan configuration file
sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
version: 2
ethernets:
enp1s0f0np0:
dhcp4: no
dhcp6: no
link-local:
mtu: 9000
addresses:
- 192.168.100.10/24
enP2p1s0f0np0:
dhcp4: no
dhcp6: no
link-local:
mtu: 9000
addresses:
- 192.168.101.10/24
EOF
Set appropriate permissions
sudo chmod 600 /etc/netplan/40-cx7.yaml
Apply the configuration
sudo netplan apply
On node 2
Create the netplan configuration file
sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
version: 2
ethernets:
enp1s0f0np0:
dhcp4: no
dhcp6: no
link-local:
mtu: 9000
addresses:
- 192.168.100.11/24
dhcp4: no
enP2p1s0f0np0:
dhcp4: no
dhcp6: no
link-local:
mtu: 9000
addresses:
- 192.168.101.11/24
EOF
Set appropriate permissions
sudo chmod 600 /etc/netplan/40-cx7.yaml
Apply the configuration
sudo netplan apply
Then I did performance benchmark with eugr’s script
on node 2 prepare to receuve
ib_write_bw -d rocep1s0f0 --report_gbits -q 4 -R --force-link IB
on node 1 sending
ib_write_bw 192.168.100.11 -d rocep1s0f0 --report_gbits -q 4 -R --force-link IB
The speed is soooo slow!
Need help figuring this out
Dual-port : OFF Device : rocep1s0f0
Number of qps : 4 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
CQ Moderation : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP
local address: LID 0000 QPN 0x02e6 PSN 0x74c0b3
local address: LID 0000 QPN 0x02e7 PSN 0xa24821
local address: LID 0000 QPN 0x02e9 PSN 0xc3dd57
local address: LID 0000 QPN 0x02ea PSN 0x9c075a
remote address: LID 0000 QPN 0x02dd PSN 0x249737
remote address: LID 0000 QPN 0x02de PSN 0xeb3655
remote address: LID 0000 QPN 0x02df PSN 0xc4017b
remote address: LID 0000 QPN 0x02e0 PSN 0x53c2ae
BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 20000 13.19 13.19 0.025159