Multiple DGX, QSFP, RDMA/NCCL

Lamerone.net · June 17, 2026, 1:35pm

Hi, everyone.
I was given two Nvidia Spark DGX servers (and I might get a third one in the next few days).
I installed Antirez’s new Dwarfstar 4 on the first one, and I have to say it works pretty well, but now I want to try connecting the two servers so they can share resources.
I haven’t found much information online about the procedure.
From what I understand, I need to:

connect the servers using a QSFP cable (which I already have)
set two IP addresses on the same network (e.g., 169.254.1.1 and 169.254.1.2)
create a user on both machines and copy the SSH keys to both
configure RDMA/NCCL (which I don’t know what it is or how to do it)

I also read that if you have more than one machine, you can’t use a QSFP cable but need to use a 10Gb switch.
I’m getting a bit lost among all this information.

carterlngstn · June 17, 2026, 1:49pm

How one just gets given multiple DGX servers?

Lamerone.net · June 17, 2026, 1:52pm

I developed a RAG that was very well received, and an American user came to my office and gave me these servers to help me develop faster.
I used to use an Nvidia 4060 Ti with 16 GB.
I was speechless.

mashie · June 17, 2026, 2:17pm

Enjoy the read:

github.com/eugr/spark-vllm-docker

docs/NETWORKING.md

main

# DGX Spark Networking

The following guide starts with a two-node cluster, but it is also applicable to larger clusters.

See [this post](https://forums.developer.nvidia.com/t/6x-spark-setup/354399/56) for an example of 6-8 node Spark cluster.
Please keep in mind that tensor-parallel vLLM deployments usually work best with a number of nodes that corresponds to a power of 2, such as 2, 4, or 8 nodes. A 3-node mesh is mainly useful for pipeline parallelism or data parallelism.

The guide assumes that the nodes are named `spark` and `spark2`, but you can use any names.
Same with IP addresses: we use `192.168.177.0/24` subnet with `.11` and `.12` assigned to both nodes, but you can use any IP addresses, as long as they are in the same subnet.

## DGX Spark ConnectX quirks

DGX Spark has a pretty unique ConnectX setup.

To achieve 200G transfer speed, ConnectX NIC needs ~x8 PCIe 5.0 lanes.

However, DGX Spark SOC can't provide more than x4 PCIe lanes per device due to hardware limitations.
So to achieve 200G on a single cable connection, each physical port shares the same pair of PCIe5 x4 connections.
Each PCIe 5 x4 link is represented by two Ethernet and two RoCE interfaces:

This file has been truncated. show original

Lamerone.net · June 23, 2026, 8:44am

This is my procedure to connect 2 dgx spark.
I hope this will be helpful to someone.

PROCEDURE TO LINK TWO NVIDIA SPARK DGX

connect QSFP on same port (no cross connection)

check if the connection is detected with ibdev2netdev

root@sparky-h:# ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

root@sparky-b:# ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

configure the network

root@sparky-h:# ip addr add 10.10.0.1/24 dev enp1s0f0np0
root@sparky-h:# ip link set dev enp1s0f0np0 mtu 9000

and

root@sparky-b:# ip addr add 10.10.0.2/24 dev enp1s0f0np0
root@sparky-b:# ip link set dev enp1s0f0np0 mtu 9000

check connection

root@sparky-h:# ifconfig enp1s0f0np0
enp1s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
inet 10.10.0.1  netmask 255.255.255.0  broadcast 0.0.0.0
ether 4c:bb:47:80:3e:50  txqueuelen 1000  (Ethernet)
RX packets 18357  bytes 4251415 (4.2 MB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 10055  bytes 1786256 (1.7 MB)
TX errors 0  dropped 2 overruns 0  carrier 0  collisions 0

root@sparky-b:# ifconfig enp1s0f0np0
enp1s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
inet 10.10.0.2  netmask 255.255.255.0  broadcast 0.0.0.0
ether 4c:bb:47:2a:ef:64  txqueuelen 1000  (Ethernet)
RX packets 385  bytes 85970 (85.9 KB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 238  bytes 40892 (40.8 KB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@sparky-h:# ping 10.10.0.2
PING 10.10.0.2 (10.10.0.2) 56(84) bytes of data.
64 bytes from 10.10.0.2: icmp_seq=1 ttl=64 time=1.30 ms
64 bytes from 10.10.0.2: icmp_seq=2 ttl=64 time=0.730 ms
^C
--- 10.10.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.730/1.016/1.303/0.286 ms

root@sparky-b:# ping 10.10.0.1
PING 10.10.0.1 (10.10.0.1) 56(84) bytes of data.
64 bytes from 10.10.0.1: icmp_seq=1 ttl=64 time=0.602 ms
64 bytes from 10.10.0.1: icmp_seq=2 ttl=64 time=0.727 ms
^C
--- 10.10.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1039ms
rtt min/avg/max/mdev = 0.602/0.664/0.727/0.062 ms

disable ufw (temp)

root@sparky-h:# ufw disable
root@sparky-b:# ufw disable

check speed

root@sparky-b:# ib_write_bw -d rocep1s0f0 -x 3




Waiting for client to connect... *




then

root@sparky-h:# ib_write_bw -d rocep1s0f0 -x 3 10.10.0.2




                RDMA_Write BW Test

Dual-port       : OFF          Device         : rocep1s0f0
Number of qps   : 1            Transport type : IB
Connection type : RC           Using SRQ      : OFF
PCIe relax order: ON
ibv_wr* API     : ON
TX depth        : 128
CQ Moderation   : 1
Mtu             : 4096[B]
Link type       : Ethernet
GID index       : 3
Max inline data : 0[B]
rdma_cm QPs     : OFF
Data ex. method : Ethernet

local address: LID 0000 QPN 0x0134 PSN 0x1100d8 RKey 0x03a300 VAddr 0x00f550d5fbd000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:10:00:01
remote address: LID 0000 QPN 0x0134 PSN 0xc833d0 RKey 0x020700 VAddr 0x00f1c2b03cd000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:10:00:02


     
    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
65536      5000             1591.42            1345.57            0.021529

ignore the speed and creare a dedicated user

(on sparky-h and sparky-b)
useradd -m -s /bin/bash aiuser
passwd aiuser
usermod -aG docker aiuser
usermod -aG sudo aiuser
usermod -aG _ssh aiuser

(on sparky-h)
aiuser@sparky-h:~$ su - aiuser
aiuser@sparky-h:~$ ssh-keygen -t rsa -b 4096

aiuser@sparky-h:~$ ssh-copy-id -p 10110 aiuser@10.10.0.2
aiuser@sparky-h:~$ ssh-copy-id -p 10110 aiuser@10.10.0.1
aiuser@sparky-h:~$ ssh-copy-id -p 10110 aiuser@127.0.0.1

if you use a different port:
aiuser@sparky-h:$ vi ~/.ssh/config

Host 10.10.0.*
Port 10110
StrictHostKeyChecking no

(try connection from sparky-h to sparky-b)
aiuser@sparky-h:~$ ssh aiuser@10.10.0.2

make deepuser folder and create some files (as aiuser)

aiuser@sparky-h:~$ mkdir -p /home/aiuser/deepspeed
aiuser@sparky-h:~$ cd /home/aiuser/deepspeed

aiuser@sparky-h:~/deepspeed$ cat > ds_config.json

{
"train_micro_batch_size_per_gpu": 1,
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "none"
},
"offload_param": {
"device": "none"
},
"overlap_comm": true,
"contiguous_gradients": true
},
"bf16": {
"enabled": true
},
"zero_allow_untested_optimizer": true
}

aiuser@sparky-h:~/deepspeed$ cat > test_deepseek.py

import os
import deepspeed
import torch
import torch.distributed as dist

def main():
deepspeed.init_distributed()

local_rank = int(os.environ.get("LOCAL_RANK", 0))
torch.cuda.set_device(local_rank)

if local_rank == 0:
print("cluster initialized successfully with gpus.")
print("testing nccl all-reduce bandwidth across dgx stations...")

tensor = torch.ones(1024, device="cuda") * local_rank
dist.all_reduce(tensor)

if local_rank == 0:
print("communication test completed successfully.")

if dist.is_initialized():
dist.destroy_process_group()

if name == "main":
main()

aiuser@sparky-h:~/deepspeed$ cat > hostfile

10.10.0.1 slots=1
10.10.0.2 slots=1

copy deepspeed on sparvy-b

aiuser@sparky-h:~/deepspeed$ cd ..
aiuser@sparky-h:~$ scp -r -P 10110 deepspeed aiuser@10.10.0.2:.

create the venv for deepspeed

aiuser@sparky-h:~/deepspeed$ python3 -m venv venv_deepspeed
aiuser@sparky-h:~/deepspeed$ source venv_deepspeed/bin/activate

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ pip install --upgrade pip setuptools wheel numpy

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ python -c "import torch; print(torch.version, torch.cuda.is_available())"

check "2.12.1+cu130 True"

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ DS_ARCH_TYPE=aarch64 pip install deepspeed --no-build-isolation --force-reinstall

error? no problem, relaunch
(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ DS_ARCH_TYPE=aarch64 pip install deepspeed --no-build-isolation --force-reinstall

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ deepspeed --version

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ python -c "import torch; print('Nome GPU:', torch.cuda.get_device_name(0)); print('CUDA:', torch.cuda.is_available())"

check for
Nome GPU: NVIDIA GB10
Compatibile sm_121: True

repeat step 10 on sparky-b

root@sparky-b:# su - aiuser

create and activate venv

pip install all

etc

install pdsh

root@sparky-h:# apt-get update && sudo apt-get install -y pdsh

do the magic

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ deepspeed --hostfile=hostfile --master_addr="10.10.0.1" --master_port=12345 --venv_script=/home/aiuser/deepspeed/venv_deepspeed/bin/activate test_deepseek.py --deepspeed ds_config.json

check

...
10.10.0.1: testing nccl all-reduce bandwidth across dgx stations...
10.10.0.2: communication test completed successfully.
10.10.0.1: communication test completed successfully.
10.10.0.1: [2026-06-20 04:04:18,130] [INFO] [launch.py:367:main] Process 790368 exits successfully.
10.10.0.2: [2026-06-20 08:04:18,383] [INFO] [launch.py:367:main] Process 1054192 exits successfully.

STEP 2 - IMPROOVEMENT

on both spark

root@sparky-h:# vi /etc/ssh/sshd_config

and add

PermitUserEnvironment yes

save and restart sshd

root@sparky-h:# service ssh restart
root@sparky-h:# systemctl restart ssh

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ vi ~/.ssh/environment

needed by deepspeed stuff

export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_NET_GDR_LEVEL=2
export NCCL_IB_HCA=rocep1s0f0
export NCCL_SOCKET_IFNAME=enp1s0f0np0
export NCCL_IB_GID_INDEX=3
export NCCL_TOPO_DUMP_FILE=/tmp/nccl_topo.xml

check if the env is ok

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ ssh aiuser@10.10.0.2 env | grep NCCL
export NCCL_IB_GID_INDEX=3
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_NET_GDR_LEVEL=2
export NCCL_IB_HCA=rocep1s0f0
export NCCL_SOCKET_IFNAME=enp1s0f0np0
export NCCL_TOPO_DUMP_FILE=/tmp/nccl_topo.xml

final test

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET deepspeed deepspeed --hostfile=hostfile --master_addr="10.10.0.1" --master_port=12345 --venv_script=/home/aiuser/deepspeed/venv_deepspeed/bin/activate test_deepseek.py --deepspeed ds_config.json

...
10.10.0.2: sparky-b:1081380:1081380 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
10.10.0.1: sparky-h:809186:809186 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
10.10.0.2: communication test completed successfully.
10.10.0.1: communication test completed successfully.
10.10.0.1: sparky-h:809186:809186 [0] NCCL INFO comm 0x3f07ceb0 rank 0 nranks 2 cudaDev 0 busId f01000 - Destroy COMPLETE
10.10.0.2: sparky-b:1081380:1081380 [0] NCCL INFO comm 0x39545890 rank 1 nranks 2 cudaDev 0 busId f01000 - Destroy COMPLETE
10.10.0.2: sparky-b:1081380:1081380 [0] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
10.10.0.1: sparky-h:809186:809186 [0] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
10.10.0.1: [2026-06-20 04:31:18,532] [INFO] [launch.py:367:main] Process 809186 exits successfully.
10.10.0.2: [2026-06-20 08:31:18,826] [INFO] [launch.py:367:main] Process 1081380 exits successfully.

OK, and now???
Now we must check if the if is used

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,ENV deepspeed test_deepseek.py

search the magic words on output

...
sparky-h:811135:811135 [0] NCCL INFO NET/IB : Made virtual device [0] name=rocep1s0f0 speed=200000 ndevs=1
sparky-h:811135:811135 [0] NCCL INFO NET/IB : Made virtual device [1] name=roceP2p1s0f0 speed=200000 ndevs=1
sparky-h:811135:811135 [0] NCCL INFO NET/IB : Using [0]rocep1s0f0:1/RoCE [1]roceP2p1s0f0:1/RoCE [RO]; OOB enP7s7:192.168.10.40<0>
sparky-h:811135:811135 [0] NCCL INFO Initialized NET plugin IB
sparky-h:811135:811135 [0] NCCL INFO Assigned NET plugin IB to comm
sparky-h:811135:811135 [0] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
sparky-h:811135:811135 [0] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm
sparky-h:811135:811135 [0] NCCL INFO Assigned RMA plugin GIN_IB_PROXY to comm
sparky-h:811135:811135 [0] NCCL INFO Using network IB
sparky-h:811135:811135 [0] NCCL INFO [Rank 0] ncclCommInitRankConfig comm 0x1ef1b350 rank 0 nranks 1 cudaDev 0 nvmlDev 0 busId f01000 commId 0xf1a257e180d55d20 - Init START
...

perfect! we are not in TCP
but "nranks 1" = we are not in distributed :o((

try this

(venv_deepspeed) aiuser@sparky-h:~/deepspeed$ deepspeed --hostfile=hostfile --master_addr=10.10.0.1 --master_port=12345 test_deepseek.py

...
10.10.0.1: [2026-06-20 04:40:22,397] [INFO] [launch.py:168:main] nnodes=2, num_local_procs=1, node_rank=0
...

ok! 2 nnodes. SOOOO HAPPY!

Topic		Replies	Views
Successful 2 DGX Spark cluster setup? DGX Spark / GB10	11	3881	October 21, 2025
How to connect two DGX Sparks DGX Spark / GB10	5	1292	May 4, 2026
ConnectX-7 NIC in DGX Spark DGX Spark / GB10	66	7117	December 2, 2025
How to connect two Nvidia DGX Spark processors in series? DGX Spark / GB10	2	823	November 25, 2025
Questions on 2-node and later 4-node dgx spark cluster DGX Spark / GB10	3	338	April 20, 2026
NCCL Test Bandwidth is only 3GB/s between 2 DGX Spark using QSFP cable DGX Spark / GB10 spark , nics , dgx	9	600	April 19, 2026
Test the sample about "Connect Three DGX Spark in a Ring Topology" DGX Spark / GB10 cuda	15	913	April 13, 2026
Dual DGX Spark RoCE Bandwidth Expectations DGX Spark / GB10	20	1071	May 14, 2026
Can't stack DGX Sparks - HELP DGX Spark / GB10	10	464	April 21, 2026
DGX Spark 200GbE direct QSFP link negotiates 200G but payload is ~12 Gbps DGX Spark / GB10 dgx-spark-issue	3	332	June 17, 2026

Multiple DGX, QSFP, RDMA/NCCL

Related topics