Launching nemotron-3-super-nvfp3 on a 2 node cluster

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

@eugr Hi, I have a two node cluster consisting of spark-02 and spark-03.

I have the spark-vllm-docker repository cloned on spark-02, and the docker images built on spark-02 and copied over to spark-03

Q01: Is there a way to launch the nemotron-3-super-nvfp3.yaml recipe on the spark-02 and spark-03 dual-node cluster?

2

Yes, easiest is to use https://sparkrun.dev/

3

@eugr When I run the following command on the head node spark-01, it launches vllm but nothing happens. I don’t see any download activity over the network for spark-02.

Q01: Could you please let me know if the following command is correct? I adapted it from the nemotron-3 recipe.

./launch-cluster.sh \                             
  -t vllm-node \
  --non-privileged \
  --nodes "192.168.120.112,192.168.120.113" \
  --eth-if enp1s0f1np1 \
  --ib-if rocep1s0f1,roceP2p1s0f1 \
  --localhost-port 8888 \
  --apply-mod mods/nemotron-super \
  -e VLLM_FLASHINFER_ALLREDUCE_BACKEND="trtllm" \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  exec vllm serve nvidia/nvidia-nemotron-3-super-120b-a12b-nvfp4 \
    --host 0.0.0.0 \
    --port 8888 \
    --max-model-len 262144 \
    --max-num-seqs 10 \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.8 \
    --enable-auto-tool-choice \
    --enable-prefix-caching \
    --load-format fastsafetensors \
    --tool-call-parser qwen3_coder \
    --reasoning-parser nemotron_v3 \
    --mamba_ssm_cache_dtype float32 \
    --moe-backend cutlass \
    --attention-backend TRITON_ATTN \
    --tensor-parallel-size 2 \
    --distributed-executor-backend ray
4

Have you downloaded the model first?

./hf-download.sh nvidia/nvidia-nemotron-3-super-120b-a12b-nvfp4 -c
5

@eugr No. I did not download the models apriori.

I ran this command to copy it to the host node and worker node.

HOSTS="192.168.120.112,192.168.120.113"
./hf-download.sh nvidia/nvidia-nemotron-3-super-120b-a12b-nvfp4 -c $HOSTS --copy-parallel

Q01: Did the launch-cluster.sh script download models for single node runs in the past?, e.g. qwen3.5 models.

6

When I launch the mode, I get the following warnings:

(Worker_TP0 pid=298) WARNING 07-31 09:37:47 [kv_cache.py:134] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP0 pid=298) WARNING 07-31 09:37:47 [kv_cache.py:148] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(Worker_TP0 pid=298) WARNING 07-31 09:37:47 [kv_cache.py:187] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.

Q01: Is this something to be concerned about? How can I make this warning go away?

7

@eugr What sort of performance can I expect from a dual-node DGX Spark cluster setup for the nvidia/nvidia-nemotron-3-super-120b-a12b-nvfp4 model?

When I run llama-bency from another remote node, and specify the master node as the model serving end-point, I get between 13.67 to 14.33 tokens/sec.

Q01: Is that the maximum performance one can expect using this model with a 2 node cluster setup?

I ran the same model on a single node a couple of months earlier, and according to my notes, I got 15 tokens/sec on a single node.

Q02: Is this normal, ie a single node running nvidia/nvidia-nemotron-3-super-120b-a12b-nvfp4 performs slightly better than a dual-node cluster setup?