DeepSeek-V4-Flash (official FP8) running across 2x DGX Spark — TP=2, MTP, 200K ctx, recipe + numbers

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
174

DeepSeek-V4-Flash (official FP8) on 4× DGX Spark — TP=4, 500K ctx, b12x, ~70 tok/s single-stream + concurrency results

Just got official DeepSeek-V4-Flash running on 4× DGX Spark (GB10) at TP=4 with 500K context using aidendle94’s b12x-optimized vLLM fork. Sharing full numbers since I hadn’t seen a confirmed 4-node benchmark post yet.

Hardware

  • 4× DGX Spark (GB10), 128GB unified memory each
  • MikroTik CRS812 switch, each node 2×200G RoCE (400G ports broken out)
  • RoCE / NCCL over CX-7 NICs

Software

  • Image: aidendle94/sparkrun-vllm-ds4-gb10:production-ready (b12x branch)
  • vLLM: v0.21.1rc1.dev339+g1967a5627bc3

Key launch flags

--tensor-parallel-size 4
--max-model-len 500000
--max-num-seqs 8
--block-size 256
--gpu-memory-utilization 0.8
--kv-cache-dtype fp8
--distributed-executor-backend mp
--compilation-config {"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}
--speculative-config {"method":"mtp","num_speculative_tokens":2}
--enable-flashinfer-autotune
--enable-prefix-caching
--enable-chunked-prefill

Environment: NCCL_NET=IB, NCCL_IB_DISABLE=0, NCCL_IB_GID_INDEX=0, VLLM_USE_B12X_MOE=1.

Single-stream decode (short prompt, 200 output tokens, server-side)

~70 tok/s sustained with 83% MTP draft acceptance rate (mean acceptance length 2.66).

Prefill throughput (single request, varying context)

Context Prompt tokens Time Prefill tok/s TTFT
4K 705 7.05s 100* 7.05s
32K 5,505 6.81s 809* 6.81s
128K 22,005 11.05s 1,991 11.05s
256K 44,005 13.25s 3,322 13.25s
500K 85,005 21.25s 4,000 21.25s

*Short contexts are network-latency-dominated from client-side measurement; server-side prefill is faster.

Decode concurrency (200 output tokens per request, client-side measured)

C Total output Time Aggregate tok/s Per-request avg
1 200 8.70s 23.0 23.0
2 400 10.11s 39.6 19.8
4 800 12.79s 62.5 15.6
8 1,600 13.55s 118.1 14.8

Prefill concurrency (128K context, 5 output tokens, client-side measured)

C Total prompt Time Aggregate tok/s Avg TTFT
1 22K 5.35s 4,111 5.35s
2 44K 5.58s 7,894 2.79s
4 88K 5.83s 15,106 1.46s

KV cache capacity:
5.3M tokens (~10 concurrent 500K requests safely)

Cold start time (first run):

  • Model load: ~160s
  • DeepGEMM warmup: ~2min
  • TileLang + FlashInfer autotune: ~37s
  • Total: ~3-4 min (subsequent starts ~40s with cache)

Comparison to known 2-node results:

The 2-node recipe showed ~42 tok/s single-stream decode and ~2000 tok/s prefill (short ctx). On 4 nodes we see:

  • Decode: ~70 tok/s (1.66× improvement, expected sub-linear due to MoE all-to-all overhead)
  • Prefill 128K: ~1991 tok/s (matched at equal context)
  • Prefill 500K: 4000 tok/s (long-context prefills benefit from more TP shards)
  • Concurrency scaling is near-linear for prefill up to C=4 (15K aggregate)

Gotchas encountered:

  1. GID index on RoCE: NCCL defaulted to GID index 3 which was empty. Fix: NCCL_IB_GID_INDEX=0.
  2. NCCL_NET=IB required: Without it, pip-distributed NCCL won’t use RoCE, causing ibv_modify_qp failures.
  3. Missing /workspace: aidendle94’s image doesn’t have the WORKDIR that the launch script expects. Added docker exec mkdir -p /workspace.
  4. Persistent GID table changed WARN: RoCE interface generates netlink events during runtime. Does not affect functionality; suppress with sysctl net.ipv6.conf.roce*.accept_ra=0 if desired.

Summary:

aidendle94’s b12x fork on 4× DGX Spark delivers solid performance: ~70 tok/s single-stream decode, 4000 tok/s prefill at 500K context, and near-linear prefill concurrency scaling up to 15K tok/s aggregate with sub-1.5s TTFT. The limiting factor is clearly MoE all-to-all cross-node communication, not compute.