Bonding Spark ConnectX-7 ports between two Sparks with Jumbo frames works fine

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Just FYI, goofing off with some of the lab rats today. Connected two Sparks with a pair of QSFP56 cables, bond mode balance-rr, MTUBytes 9000, assigned static addresses with host file entries. Respectable rsync speeds, mpirun working fine between both units.

Quick hack (using systemctl for network) :

In /etc/systemd/network (the other unit is 10.254.254.2/30 )

cat bond0.netdev 
[NetDev]
Name=bond0
Kind=bond
MTUBytes=9000

[Bond]
Mode=balance-rr


cat bond0.network 
[Match]
Name=bond0

[Network]
Address=10.254.254.1/30

[Link]
MTUBytes=9000


cat enp1s0f0np0.network 
[Match]
Name=enp1s0f0np0

[Network]
Bond=bond0

[Link]
MTUBytes=9000

cat enp1s0f1np1.network 
[Match]
Name=enp1s0f1np1

[Network]
Bond=bond0

[Link]
MTUBytes=9000
1 Like
2

Hi! Can you share outputs of the numbers? Also can you share the network tests output?

I am asking because I just stacked my two sparks and I used the `all_gather_perf` scripts from `nccl_test` and I expected to see bigger bus bandwidths.

My results

3

“If you have no expectations, it’s hard to be disappointed!"“ I’ve been spoiled by much faster speeds which is why I said WTF when I was moving data between the sparks and decided to see what could be done.

rsync -ruv testfile.bin ss2:
sending incremental file list
testfile.bin

sent 17,184,063,578 bytes received 35 bytes 404,330,908.54 bytes/sec
total size is 17,179,869,184 speedup is 1.00

I’ll run nccl tests when I’m done with work later today

4

Please keep us posted on your NCCL test with bonded interfaces.

5

I wouldn’t count these for anything, my space heater lab rats are so far off the stock build image now that I’d have to wipe them and keep a notebook to record step by step changes. I’m a happy camper, great machines! :-) LOL, they are going to keep me very warm this winter!

nccl-tests version 2.17.6 nccl-headers=22803 nccl-library=22803

Collective test starting: all_gather_perf
nThread 1 nGpus 1 minBytes 1073741824 maxBytes 34359738368 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0

Using devices
Rank  0 Group  0 Pid   3089 on        ss1 device  0 [000f:01:00] NVIDIA GB10
Rank  1 Group  0 Pid   2952 on        ss2 device  0 [000f:01:00] NVIDIA GB10


                                                                                        out-of-place                       in-place
size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong 
(B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)

1073741824     134217728     float    none      -1  30360.1   35.37   17.68       0  28562.2   37.59   18.80       0
2147483648     268435456     float    none      -1  56057.6   38.31   19.15       0  52172.4   41.16   20.58       0
4294967296     536870912     float    none      -1   105204   40.83   20.41       0  99036.2   43.37   21.68       0
8589934592    1073741824     float    none      -1   200909   42.76   21.38       0   192992   44.51   22.25       0
17179869184    2147483648     float    none      -1   392016   43.82   21.91       0   382806   44.88   22.44       0
34359738368    4294967296     float    none      -1   773561   44.42   22.21       0   763964   44.98   22.49       0

Out of bounds values : 0 OK

Avg bus bandwidth    : 20.916

Collective test concluded: all_gather_perf