Hi Nvidia team:
In contrast, in the NCCL for Two Sparks setup, these variables are set to enp1s0f1np1.
These variables need to be consistent across different nodes; otherwise, the test will fail. For the “Connect Three DGX Spark in a Ring Topology” example, it doesn’t seem feasible to set them to enp1s0f1np1 or enp1s0f0np0.
So, does this mean that in the “Connect Three DGX Spark in a Ring Topology” setup, the three DGX Spark systems must additionally communicate via enP7s7 in order to perform NCCL + QSFP validation?
Could you also help confirm whether there are any misunderstandings in my interpretation and provide suggestions? Thank you.
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import logging
show original
NCCL for Two Sparks | DGX Spark
These env variables NCCL_SOCKET_IFNAME, UCX_NET_DEVICES, OMPI_MCA_btl_tcp_if_include are used for bootstrapping communication, not for high speed data transfer.
eugr
March 31, 2026, 7:15pm
3
This is correct. In mesh setups, 10G ethernet will be used for OOB (out of bound) communications between the nodes, but IB links will be utilized for a high-speed connection.
The upcoming version of our community docker (GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub ) supports 3-node Mesh for vLLM.
Does that mean it will get over the power of 2 limitation / feature and will be able to handle -tp 3?
eugr
March 31, 2026, 7:45pm
5
No, the limitation is number of attention heads in the model needs to be divisible by number of nodes in TP.
Hi @eugr and @aniculescu :
Thank you.
Test result:
test_20260401_1.txt (3.5 KB)
Avg bus bandwidth from NCCL test: 2.85932 GB/s
test_20260401_2.txt (3.3 KB)Avg bus bandwidth from NCCL test: 2.86038 GB/s
Network config:nmcli_network_setup_node0-2.zip (1.0 KB)
eugr
April 1, 2026, 7:19am
8
Why is the third host on a different subnet? You should specify IPs that are assigned to enP7s7 (10G ethernet) here.
Can you post the results of ifconfig -a from all three nodes?
You should be getting ~24 GB/s avg. bus bandwidth.
You can also check your configuration against this networking guide .
Hi @eugr :
# DGX Spark Networking
The following guide is for two node cluster, but it is also applicable to larger clusters.
See [this post](https://forums.developer.nvidia.com/t/6x-spark-setup/354399/56) for an example of 6-8 node Spark cluster.
Please keep in mind that to get the most from vLLM you need to have number of nodes that corresponds to power of 2 - e.g. 2, 4 or 8 nodes.
The guide assumes that the nodes are named `spark` and `spark2`, but you can use any names.
Same with IP addresses: we use `192.168.177.0/24` subnet with `.11` and `.12` assigned to both nodes, but you can use any IP addresses, as long as they are in the same subnet.
## DGX Spark ConnectX quirks
DGX Spark has a pretty unique ConnectX setup.
To achieve 200G transfer speed, ConnectX NIC needs ~x8 PCIe 5.0 lanes.
However, DGX Spark SOC can't provide more than x4 PCIe lanes per device due to hardware limitations.
So to achieve 200G on a single cable connection, each physical port shares the same pair of PCIe5 x4 connections.
Each PCIe 5 x4 link is represented by two Ethernet and two RoCE interfaces:
show original
Reference config file:dgx-spark-playbooks/nvidia/multi-sparks-through-switch/assets/spark_cluster_setup/config/spark_config_ring.json at main · NVIDIA/dgx-spark-playbooks · GitHub
Hi @eugr :
Is it necessary for enP7s7 to be connected to a 10G-capable port and network, or is that not required?
Thanks.
Test reference:dgx-spark-playbooks/nvidia/multi-sparks-through-switch/assets/spark_cluster_setup at main · NVIDIA/dgx-spark-playbooks · GitHub
Test command by (bash spark_cluster_setup.sh -c spark_config_ring.json --run-setup)
Test resulttest_20260401_referenceip_1.txt (5.1 KB)
ifconfig -a for three devices:ifconfig_node0.txt (3.6 KB)ifconfig_node1.txt (3.9 KB)ifconfig_node2.txt (3.5 KB)
eugr
April 1, 2026, 3:45pm
11
Instead of using CX7 IPs, try to launch it like this:
export CUDA_HOME=“/usr/local/cuda” && \
export MPI_HOME=“/usr/lib/aarch64-linux-gnu/openmpi” && \
export NCCL_HOME=“$HOME/nccl_spark_cluster/build/” && \
export LD_LIBRARY_PATH=“$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH” && \
mpirun -np 3 -H 192.168.1.46:1,192.168.1.246:1,192.168.1.164:1 --mca plm_rsh_agent “ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no” -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x UCX_NET_DEVICES=enP7s7 -x NCCL_SOCKET_IFNAME=enP7s7 -x OMPI_MCA_btl_tcp_if_include=enP7s7 -x NCCL_IB_HCA=rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1 -x NCCL_IB_SUBNET_AWARE_ROUTING=1 -x NCCL_IB_MERGE_NICS=0 -x NCCL_NET_PLUGIN=none $HOME/nccl-tests_spark_cluster/build/all_gather_perf -b 16G -e 16G -f 2
The IP addresses you are supposed to provide to mpirun should be OOB interface addresses, in this case assigned to enP7s7.
If that doesn’t work, check the physical wiring against my networking guide to make sure it wired according to IP allocation, and try to use manual steps from the guide instead of the script from the playbook.
The playbook was incorrect before, and while they fixed one issue, there could be other bugs there too.
eugr
April 1, 2026, 4:17pm
12
1G is sufficient, it is used as a control channel and it’s speed/latency is not really important here.
eugr
April 2, 2026, 7:17pm
14
Yes, because that’s how the nodes will be discovered.
Hi @eugr :
Thank you.
Test Command1:
# nccl-tests version 2.18.2 nccl-headers=22907 nccl-library=22907
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
# Rank 0 Group 0 Pid 5353 on gx10-f0df device 0 [000f:01:00] NVIDIA GB10
# Rank 1 Group 0 Pid 4685 on gx10-f21a device 0 [000f:01:00] NVIDIA GB10
# Rank 2 Group 0 Pid 5482 on gx10-9680 device 0 [000f:01:00] NVIDIA GB10
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
33554400 2796200 float none -1 1196.48 28.04 18.70 0 1203.57 27.88 18.59 0
# Out of bounds values : 0 OK
# Avg bus bandwidth : 18.6411
#
# Collective test concluded: all_gather_perf
Test Command 2:
# nccl-tests version 2.18.2 nccl-headers=22907 nccl-library=22907
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 3(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
# Rank 0 Group 0 Pid 4998 on gx10-f21a device 0 [000f:01:00] NVIDIA GB10
# Rank 1 Group 0 Pid 6343 on gx10-f0df device 0 [000f:01:00] NVIDIA GB10
# Rank 2 Group 0 Pid 6344 on gx10-9680 device 0 [000f:01:00] NVIDIA GB10
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
17179869168 1431655764 float none -1 551876 31.13 20.75 0 547491 31.38 20.92 0
# Out of bounds values : 0 OK
# Avg bus bandwidth : 20.8364
#
# Collective test concluded: all_gather_perf
eugr
April 13, 2026, 6:18am
16
It is still lower than it should be, but closer to expected.
Hi @eugr :
