Hi all,
I’m running DeepSeek-V4-Flash-Vision-exp across 2x DGX Spark (TP=2) using the MIAI Labs Recipe.
Our target is ≥500K context with 8 concurrent users, but I’m noticing token generation speed and TTFT dropping sharply as both prompt size and concurrency increase — not just linearly, but into a steep floor. For example, on our cluster at 128K prompt tokens with 6 concurrent streams, aggregate decode throughput falls to ~2 tok/s with a median TTFT around 280s. At 32K+ prompts with 4-6 concurrent streams we sit around an ~8 tok/s aggregate floor, essentially one decode token per serialized prefill step — large prefills appear to run near-sequentially rather than truly in parallel, so wall clock becomes almost all prefill time. Single-stream (c=1) performance stays flat and healthy across all prompt sizes, so this looks like a prefill-scheduling/batching bottleneck under concurrency rather than a raw compute or KV-memory ceiling (our KV pool sizes to ~2.3-2.4x the 1M ceiling, so memory isn’t the constraint at these token counts).
Questions for the community:
- What are the recommended vLLM/scheduler settings (chunked prefill tuning, prefill/decode split, batching policy, etc.) to prevent large concurrent prefills from serializing and starving decode throughput for other users at high context lengths?
- Would stacking to 4x DGX Spark meaningfully change this bottleneck, or does it primarily just add KV/compute headroom without addressing the prefill-serialization behavior we’re seeing? Has anyone benchmarked Vision-Exp specifically at high concurrency + long context on a multi-Spark cluster?