Dual DGX Spark RoCE Bandwidth Expectations

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi, new here and new to the Spark. I have two Sparks and I have successfully connected them using the Amphenol cable purchased from the Marketplace.

I have been working to get Nemotron3 120 Super running in the cluster via vLLM, and started with a working configuration that gave me 12-14 TPS. However, the connection was falling back to TCP and not leveraging RoCE. Once I had RoCE visible in the container, I was able to get 17+ TPS. A decent bump.

In tests I’m seeing about 115 Gbps. I have no idea if this is good or not. Seems I read that it’s expected, then, no - you should be able to get close to 200. On one cable. Nope, need two. No you don’t… etc. I just can’t find a clear answer!

Can someone clarify this for me? And, if I should be able to get 200 on one cable, where is the configuration documented?

Thanks!

3

You will get 200G with a single cable but you need to get two of the virtual interfaces configured.

Easiest is to use sparkrun to get it all setup:

4

I tried Sparkrun a while back and couldn’t get it working. I’ll give it another look.

5

This article explains it in depth: The NVIDIA GB10 ConnectX-7 200GbE Networking is Really Different - ServeTheHome and there have been some previous posts on this here too ( ConnectX-7 NIC in DGX Spark - #65 by isdias )

The short version is that they did not have a PCIe x8 lane but had two PCIe x4 lanes. The x4 lane isn’t enough for a full 200GB. So the physical interfaces are split across two PCIe x4 lanes. As such, there is limited (bandwidth) value in hooking up more than one 200GB physical interface on the spark.

6

So does that mean you need 2 cables between the 2 sparks to get the full throughput?

7

One physical cable / port will get you 200G. However, the operating system sees that one physical port as two ports because of the PCIe lane issue.

When you run

$ ibdev2netdev
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Down)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)

The two up interfaces (enP2p1s0f1np1 and enp1s0f1np1) are actually the same physical network port/cable.

This playbook ( NCCL for Two Sparks | DGX Spark ) also shows how to test the connectivity.

If you hook up both cables, you could probably get around 250G since that’s around the theoretical max of two x4 lanes. The extra 50G is likely not worth the extra complexity.

8

You should be getting ~26+ t/s on this model even without MTP on two nodes.

TCP tests or IB tests on a single sub-interface will max out at ~100 GBps, but NCCL is able to use both and achieve full 200G.

You can read more details about unique Spark networking situation and how to set it up properly in our community networking guide.

9

OK, thanks all for the replies.

I verified that I have the full bandwidth available now, and ran spark run recipie for Nemotron3 120B Super clustered.

Everything appears to be working properly - but my TPS wen DOWN to ~15 again when I had it up to 17-19 before using spark run.

Any diagnostic suggestions?

10

The OOB recipe for for Nemotron-3-Super-120B had terrible performance for me, ~15 TPS.

I made some changes and now have it back to 18-19 TPS. The logs seem to show it using the full RoCE capability (I think) and I can see both GPUs being engaged during inference.

Here is my WIP attached recipie:

# Recipe: Nemotron-3-Super-NVFP4

# Uses VLLM_CUTLASS for NVFP4

recipe_version: "1"

name: Nemotron-3-Super-NVFP4-Marlin-FlashInfer

description: vLLM serving Nemotron-3-Super-120B using Marlin + FlashInfer kernels




model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

container: vllm-node

cluster_only: false

solo_only: false




# mods: 

#   - mods/nemotron-super




env:

  VLLM_ALLOW_LONG_MAX_MODEL_LEN: 1

  VLLM_ATTENTION_BACKEND: FLASHINFER

  VLLM_FLASHINFER_FORCE_TENSOR_CORES: 1

  VLLM_NVFP4_GEMM_BACKEND: marlin

  NCCL_IB_DISABLE: 0

  NCCL_IB_HCA: rocep1s0f0,roceP2p1s0f0

  NCCL_SOCKET_IFNAME: enp1s0f0np0,enP2p1s0f0np0

  GLOO_SOCKET_IFNAME: enp1s0f0np0

  NCCL_DEBUG: INFO




container: vllm-node

defaults:

  port: 8000

  host: 0.0.0.0

  tensor_parallel: 2

  gpu_memory_utilization: 0.9

  max_model_len: 262144

  max_num_seqs: 16




command: |

  vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \

  --kv-cache-dtype fp8 \

  --quantization fp4 \

  --moe-backend marlin \

  --trust-remote-code \

  --gpu-memory-utilization {gpu_memory_utilization} \

  --max-model-len {max_model_len} \

  --max-num-seqs {max_num_seqs} \

  --enable-prefix-caching \

  --enable-chunked-prefill \

  --host {host} \

  --port {port} \

  --enable-auto-tool-choice \

  --load-format fastsafetensors \

  --tool-call-parser qwen3_coder \

  --reasoning-parser nemotron_v3 \

  --mamba_ssm_cache_dtype float16 \

  --tensor-parallel-size {tensor_parallel} \

  --attention-backend FLASHINFER \

  --distributed-executor-backend ray \

If anyone has any feedback, thanks in advance! I am not sure what I am doing wrong, but the best I’ve seen with both Sparks is 19 tps.

11

Marlin backend had some issues with NVFP4, so I switched all recipes to VLLM_CUTLASS about a month ago. You can also try running it with FLASHINFER_CUTLASS now that it is stable on Spark:

# Recipe: Nemotron-3-Super-NVFP4
# Uses VLLM_CUTLASS for NVFP4
recipe_version: "1"
name: Nemotron-3-Super-NVFP4-CUTLASS-Optimized
description: vLLM serving Nemotron-3-Super-120B using CUTLASS kernels

model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
container: vllm-node
cluster_only: false
solo_only: false

# mods:
#   - mods/nemotron-super

env:
  VLLM_FLASHINFER_ALLREDUCE_BACKEND: trtllm
  VLLM_ALLOW_LONG_MAX_MODEL_LEN: 1
  VLLM_FLOAT32_MATMUL_PRECISION: "high"

container: vllm-node
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.7
  max_model_len: 262144
  max_num_seqs: 10

command: |
  vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --gpu-memory-utilization {gpu_memory_utilization} \
  --max-model-len {max_model_len} \
  --max-num-seqs {max_num_seqs} \
  --enable-prefix-caching \
  --host {host} \
  --port {port} \
  --enable-auto-tool-choice \
  --load-format fastsafetensors \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3 \
  --mamba_ssm_cache_dtype float32 \
  --tensor-parallel-size {tensor_parallel} \
  --distributed-executor-backend ray
12

I’ll try your above setup - but my initial results with the out-of-box recipe for this model in the repo used Cutlass and I saw ~15 tps, which was worse than with Marlin.

13

Make sure you rebuild the container using latest wheels:

git pull
./build-and-copy.sh

I have this model in my build pipeline and it shows ~25 t/s on dual sparks. It was dropping to ~18 t/s last week due to one of the vLLM PRs, but those builds were never published because the pipeline was failing, but if you built from the source (like by forcing with --rebuild-vllm), you may have one of those degraded builds.

14

Which commit do you recommend?

15

Just run the latest “stable” release from the repo (it will be used if no parameters are passed to ./build-and-copy.sh.

16

I am apparently already running the latest Wheels.

Commit hash matches (4f072c4f) — wheels are up to date.
All flashinfer wheels are up to date — skipping download.
FlashInfer wheels ready.
Commit hash matches (17ed5e61f) — wheels are up to date.
All vllm wheels are up to date — skipping download.
vLLM wheels ready.
Generated build-metadata.yaml
Building runner image with command: docker build -t vllm-node --build-arg BUILD_JOBS=16 --build-arg TORCH_CUDA_ARCH_LIST=12.1a --build-arg FLASHINFER_CUDA_ARCH_LIST=12.1a .
[+] Building 4.7s (20/20) FINISHED

17

how do you launch the recipe? Can you also post the startup log? You should be getting ~25 t/s on dual Sparks. Another possibility is that one (or both) Sparks are stuck in low power state. In this case, shut down both Sparks, unplug the USB-C power cable from the back (this is important!), plug it back in, and start the sparks back up.

18

I launch the recipe (above in this thread) using a little script which just wraps run-recipe.sh

#!/usr/bin/env bash

set -euo pipefail


RECIPE="${1:?Usage: spark-launch.sh <recipe-yaml> [--setup]}"

SETUP_ARG="${2:-}"


source ~/.config/sparkrun-local/env.sh


if [[ -z "${HF_TOKEN:-}" ]]; then

  echo "HF_TOKEN is not set."

  exit 1

fi


cd ~/spark-vllm-docker

echo "Launching recipe: $RECIPE"

if [[ "$SETUP_ARG" == "--setup" ]]; then

  ./run-recipe.sh "$RECIPE" --setup -e HF_TOKEN="$HF_TOKEN"

else

  ./run-recipe.sh "$RECIPE" -e HF_TOKEN="$HF_TOKEN"

fi
19

can you post the vLLM startup log (including the output from ./run-recipe.sh)?

20

Attached. My tests are showing ~17-20 tps end to end

nemotron_start.log (131.9 KB)

21

Can you try to run with the current nemotron-3-super-nvfp4.yaml recipe and see if the performance improves? Pull the updates from the repo in case it’s not up to date.