Install and Use vLLM for Inference on two Sparks does not work

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
22

In the ray-head container:

nvidia-smi shows the GPU on both devices.
NCCL_SOCKET_IFNAME=enp1s0f1np1 (on both)

Node 1: enp1s0f1np1 = 169.254.63.129
Node 2: enp1s0f1np1 = 169.254.114.178

NCCL Tests

When running the NCCL test (NCCL for Two Sparks | DGX Spark)

Output:

nccl_test.txt (2.0 KB)

Ray status:

======== Autoscaler status: 2025-11-10 13:22:07.585565 ========
Node status

Active:
1 node_0fc6ddd39c05508165e32bb458c997d72d78e69e4cb1b9bf4c8d6899
1 node_42a2d138136ca000986197db2d32663887f107c601f3b44021a09811
Pending:
(no pending nodes)
Recent failures:
(no failures)

Resources

Total Usage:
0.0/40.0 CPU
0.0/2.0 GPU
0B/207.53GiB memory
0B/30.40GiB object_store_memory

From request_resources:
(none)
Pending Demands:
(no resource demands)

Update

I have found the error for not finding the IB device.

In the start_worker_vllm script i have changed the NCCL_IB_HCA variable from mlx5_0,mlx5_1 to rocep1s0f1,roceP2p1s0f1

Now the vllm server is running. With gpt-oss-20b i get 50 token/s.

23

Ah, so it was trying the wrong addresses.

Glad to hear that you are now up and running!!

24

Yes thank you! However, I had an issue with the Docker command not running to start VLLM and dying w/out any logs. Unfortunately, I have the patience of gnat, so I tried a couple of things at once then it ran. Also, I am using the .env file approach and not the setup.sh.

Here’s my step 8/9:

# Kill any existing vLLM processes
log "  Killing any existing vLLM processes..."
docker exec "${NAME}" pkill -f 'vllm serve' 2>/dev/null || true

# Ensure log directory exists
log "  Creating log directory..."
docker exec "${NAME}" mkdir -p /var/log

# Start vLLM in background using docker exec -d
log "  Starting vLLM in background..."
docker exec -d "${NAME}" bash -c "
  export HF_HOME=/root/.cache/huggingface &&
  export RAY_ADDRESS=127.0.0.1:6379 &&
  export PYTHONUNBUFFERED=1 &&
  export VLLM_LOGGING_LEVEL=INFO &&
  vllm serve '${MODEL}' \
    --distributed-executor-backend ray \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size ${TENSOR_PARALLEL} \
    --max-model-len ${MAX_MODEL_LEN} \
    --gpu-memory-utilization ${GPU_MEMORY_UTIL} \
    --download-dir \$HF_HOME \
    --enforce-eager \
    > /var/log/vllm.log 2>&1
"

Run with …

source .env && bash nv_start_head_vllm.sh
25

FYI. I updated the script to remove the hardcoded IB info and it now dynamically obtains the correct IB info.

26

I found the issue with the script on not properly launching vLLM.

Just posted new versions of both head and worker scripts that resolve this issue and a few other items we found.

27

We have benchmarking in this blog post [How NVIDIA DGX Spark’s Performance Enables Intensive AI Tasks | NVIDIA Technical Blog])

Qwen3 235B was tested on dual sparks connected with the ConnectX-7

28

Two follow-ups:

  1. I want Still can’t run FP4 Quants though…

nvidia/Qwen3-235B-A22B-FP4
RuntimeError: Failed to initialize GEMM

  1. Can you point me toward some benchmark procedures/instructions? I’ll be glad to share. (I saw the recent post)

Thanks!

29

@mark440 and @mpanthofer I see you have found issues with this playbook [Install and Use vLLM for Inference | DGX Spark](Install and Use vLLM for Inference | DGX Spark

Are all your modifications in this forum post or did you submit a pull request: dgx-spark-playbooks/nvidia/vllm at main · NVIDIA/dgx-spark-playbooks · GitHub

30

I started with the playbook, but when I hit a few snags, i created a new playbook and posted it on my git repo: GitHub - mark-ramsey-ri/vllm-dgx-spark: Run vLLM on one or two Nvidia DGX Spark servers

31

I only have 1 spark, got it working by adding the following to the docker command line: --shm-size 32g, I used the original playbook instructions.

32

Perfect, thank you.

Do you perhaps have any idea why I only get < 5 tokens/s with lama3.3-70b-instruct?

vLLM startup:

vllm_startup.txt (54.7 KB)

33

i think it is using standard Ethernet and not the Infiniband connection between the two Sparks.

I have made adjustments to the scripts to assure it uses IB and not standard Ethernet.

Update the repo, stop and rm the head and worker docker containers and run them again. You should see 10-20x faster rates when properly using IB.

34

Thanks for the quick fixes! I will try the new version. At least the version from yesterday starts without errors. When i check the IP of the used interface in the container it shows the correct IB IP.

I will redeploy and share the results.

35

Good luck.

If you still see the low rates, run the vllm_system_checkout.sh and post the log and we will sort it out from that.

36

I don’t think you can expect significantly more speed from a dense model, even on two Sparks. The memory throughput (which defines token generation speed) on Spark is only 273 GB/s.

37

After redeployment, unfortunately still < 5 tokens/s.

vllm_system_checkout:

logs.txt (101.8 KB)

38

What quant are you using? Can you post your VLLM commandline arguments?

In any case, as a rule of thumb, max inference speed for a dense model is roughly memory_bandwidth_in_GB / model_weights_in_GB.

So, for a 70B dense model, like llama 3.3, it would be:

  • BF16: 273/140 = 1.95 t/s
  • FP8/INT8: 3.9 t/s
  • FP4/NVFP4/INT4: 7.8 t/s

Running this model in the cluster will not improve inference performance, and will likely make it slower due to communication overhead. It will only improve overall throughput (parallel requests) and prompt processing (as it is more compute bound than bandwidth bound).

39

Thanks for this info. I will test with different models.

In the current setup i use meta-llama/Llama-3.3-70B-Instruct · Hugging Face which is BF16.

40

Generally, I would avoid running anything more than 32B active parameters on Spark, as it will be just way too slow. Also, 8-bit quants generally perform at the same level as BF16 model, so there is very little reason to serve full model, use FP8 version instead (if available) or other quants (AWQ 8-bit, for instance).

The sweet spot for Spark are MOE models. Your dual Spark setup will allow you to run Qwen3-235B-A22B FP4 quant at about 12 tokens/s.

For a single Spark, gpt-oss-120b is a great choice. You will get between 40-60 tps (depending on inference engine) and decent prompt processing speed.

41

i have added a benchmark script to test the tokens per second using vLLM to the repo.

I am also working to get TRT working across 2 sparks to see if that boosts the performance.