Super slow connection between 2 sparks

Two sparks running latest 7.0.0-1019-nvidia kernal. I followed both eugr and NVIDIA’s play book to setup my 2 sparks

node 1

ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

node 2

ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)



On node 1

Create the netplan configuration file

sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
version: 2
ethernets:
enp1s0f0np0:
dhcp4: no
dhcp6: no
link-local: 

mtu: 9000
addresses:
- 192.168.100.10/24
enP2p1s0f0np0:
dhcp4: no
dhcp6: no
link-local: 

mtu: 9000
addresses:
- 192.168.101.10/24
EOF

Set appropriate permissions

sudo chmod 600 /etc/netplan/40-cx7.yaml

Apply the configuration

sudo netplan apply

On node 2

Create the netplan configuration file

sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
version: 2
ethernets:
enp1s0f0np0:
dhcp4: no
dhcp6: no
link-local: 

mtu: 9000
addresses:
- 192.168.100.11/24
dhcp4: no
enP2p1s0f0np0:
dhcp4: no
dhcp6: no
link-local: 

mtu: 9000
addresses:
- 192.168.101.11/24
EOF

Set appropriate permissions

sudo chmod 600 /etc/netplan/40-cx7.yaml

Apply the configuration

sudo netplan apply

Then I did performance benchmark with eugr’s script

on node 2 prepare to receuve

ib_write_bw -d rocep1s0f0 --report_gbits -q 4 -R --force-link IB

on node 1 sending

ib_write_bw 192.168.100.11 -d rocep1s0f0 --report_gbits -q 4 -R --force-link IB

The speed is soooo slow!

Need help figuring this out


Dual-port       : OFF		Device         : rocep1s0f0
Number of qps   : 4		Transport type : IB
Connection type : RC		Using SRQ      : OFF
PCIe relax order: ON
ibv_wr* API     : ON
CQ Moderation   : 1
Mtu             : 4096[B]
Link type       : IB
Max inline data : 0[B]
rdma_cm QPs	 : ON
Data ex. method : rdma_cm

Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP

local address: LID 0000 QPN 0x02e6 PSN 0x74c0b3
local address: LID 0000 QPN 0x02e7 PSN 0xa24821
local address: LID 0000 QPN 0x02e9 PSN 0xc3dd57
local address: LID 0000 QPN 0x02ea PSN 0x9c075a
remote address: LID 0000 QPN 0x02dd PSN 0x249737
remote address: LID 0000 QPN 0x02de PSN 0xeb3655
remote address: LID 0000 QPN 0x02df PSN 0xc4017b
remote address: LID 0000 QPN 0x02e0 PSN 0x53c2ae


     
    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
65536      20000            13.19              13.19  		   0.025159

Power off the Sparks, unplug from the wall for a minute. Power back on and full speed should return.

Man you are a life saver. I did just that, is this considered normal now?

---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      20000            110.92             110.88 		   0.211486
---------------------------------------------------------------------------------------

It is a common problem that can happen the very first time a DAC cable is plugged in. The total power drain is needed for the ConnectX-7 to apply some unknown reconfiguration.

This behaviour will never happen a second time.