C1 1058pp/s, 52 tg/s on 1x dgx spark on Deepseek V4 flash 0731 full 256 experts

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

screenshot 0812
screenshot 08121740×1362 396 KB

im only introducing this here.

in korean communities i have been visiting, this recipe has been used quite well by people who have only one dgx spark. i thought there should already be something similar in the spark forum, so honestly i did not pay much attention to it before. but recently i checked and could not find it here, so i thought maybe it is worth bringing it over.

some information below is only a small part of what i got from sol.

the interesting point is that this keeps all 256 routed experts in all 43 target layers. nothing is pruned. the checkpoint is around 78.2GB and uses exl3 k2 around 2 bpw for the routed expert weights, while keeping the complete expert topology and routing structure.

it also includes the native three block dspark drafter and proposes five tokens at once.

on one dgx spark, the reported long context prefill is 1058.8 pp/s.

the 52.47 tg/s c1 result is from five code generation tests with 512 output tokens. this is important because it should not be understood as normal chat always running at 52 tg/s.

in the more boring matched test with 2048 fresh prompt tokens and 128 output tokens, c1 was around 24.7 tg/s and c4 aggregate was 31.7 tg/s.

dspark acceptance was 63.81 percent in the qualification tests, but only 28.95 percent across the matched hardware sweep, so speculative decoding benefit looks very dependent on workload.

another nice part is that the repository includes the validation scripts and raw results, so other spark users can repeat the same tests instead of only trusting the numbers.

there are still some limits. there is no proper same revision fp8 versus k2 quality comparison yet, so it is difficult to know the exact quality loss from this aggressive quantization. also the roughly one million token setting mainly shows available kv capacity, not that semantic quality at one million tokens has been fully validated.

anyway, i am just sharing because it seems to be working quite well for some one spark users and i was a little surprised that i could not find it mentioned here.

I havent play it around by myself but hopefully it is useful for single spark owners.

2

Thanks for this!

Ran this recipe as check of the numbers in the OP.

Concurrency Aggregate tok/s Per-request median
1 57.3 57.3
8 62.7 10.1
64 114.5 12.8

Opus 4.8:
Config: MAX_NUM_SEQS=12, MAX_MODEL_LEN=262144, GPU_MEMORY_UTILIZATION=0.85, otherwise recipe defaults. For reference, the same harness on the same box measures the Entrpi/ds4 v0.5.5 stack (IQ2XXS + DSpark drafter) at 21.5 single / 62 @c8 — so this is ~2.7× single-stream at comparable aggregate, with headroom at c16 the ds4 stack doesn’t reach.

Tool calling: 3/3 on our agent-style test (multi-step call with correct args, chained follow-up after a tool result, and restraint on a no-tool question) — the DSML output parses cleanly through the recipe’s own deepseek_v4 parser. Quality spot-checks (code correctness, instruction following, arithmetic reasoning at T=0) came out equal-or-better versus the IQ2XXS quant; notably the reasoning probe finishes its answer where the 2-bit imatrix quant tends to exhaust its budget.

Two gotchas worth knowing before you launch:

  1. Empty env strings crash the kernel compiler. The compose file passes several variables as ${VAR:-}, which Docker delivers as empty strings, not unset vars. On our box that surfaced as a crash loop deep in the CUTE DSL during MLA kernel compile — KeyError: ‘’ in an enum lookup — before the server ever became healthy. Fix: set them explicitly in .env (at minimum CUTE_DSL_ARCH=sm_121a on Spark, plus GPU_MEMORY_UTILIZATION and MAX_MODEL_LEN). After that, clean boot every time.
  2. Don’t repeat entrypoint flags in EXTRA_VLLM_ARGS. The entrypoint already sets the correct --tool-call-parser deepseek_v4 (and --served-model-name, --port, …). Anything you add via EXTRA_VLLM_ARGS is appended after those, and vLLM’s arg parsing lets the last occurrence win — it even logs a “Found duplicate keys” warning that’s easy to miss. I initially added --tool-call-parser deepseek_v3 out of habit and got raw DSML markup in content with tool_calls: null, which looks exactly like a broken model. It isn’t — it’s a config foot-gun. Only add flags, never override, and grep the boot log for “duplicate keys” if tools misbehave.
3

this sounds very impressive. is it just me or are these numbers better than the various GitHub - antirez/ds4: DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm · GitHub clones for the spark? how do they actually compare accross benchmarks? I would try myself but I’m on holiday with no spark access.

4

@chargeuk I ran a 3-axis head-to-head on my box (single Spark, 128GB): EXL3 K2 here vs the ds4 stack I’ve been running (v0.5.6.3, IQ2XXS + DSpark). Same box, same probes.

tool-eval-bench (agentic tool use, 69 scenarios): ds4 IQ2XXS 91/100, EXL3 K2 88/100. ds4 slightly ahead but within noise, it was a few extra fails not a different tier.

Two real-code probes (find and fix defects in an actual buggy repo, plus a code-review precision task): EXL3 came out ahead on both. It caught a real bug the IQ2XXS runs missed, and it did the least of the repetition-looping the 2-bit imatrix quant tends to fall into on long tasks. On the review task it cited exact line numbers with real quoted code where IQ2XXS was more approximate. Lines up with what timhbl said about it finishing where the 2-bit runs out of budget.

Speed: EXL3 was ~20-30% faster per generation on my probes, and those are prefill-heavy so the decode win is diluted, consistent with timhbl’s 2.7x single-stream on pure decode.

Net for me: within margin on tool use, better on coding, faster. I moved my agent harness (Hermes) onto it for real-world benching. One heads-up for anyone trying it: the empty-env-var crash timhbl flagged is real, I hit the exact KeyError ‘’ in the MLA kernel compile on first boot. Setting CUTE_DSL_ARCH=sm_121a plus GPU_MEMORY_UTILIZATION and MAX_MODEL_LEN in .env fixed it, clean boot every time after. Thanks timhbl for documenting that.

5

thank you @scope_cr33p will be trying this out when I’m back!

6

Thanks for posting this. So this is using a custom quant as well? I wonder if it has more intelligence.

Edit: I also see that the repo it links to might be using an even more intelligent quant?

This is some good stuff. Always trust some rando that we never heard of (no offense meant obviously) to come up with some superior version that no one is discussing lmao.

7

0xSero seems to work really well, I did a quick test (find and summarize 10 Jira subtasks).
It produced a good result and was 10x as quick as the ds4 solution.
maybe its because thinking is off?

i had to patch xgrammar in the image to make it work with opencode.
Start docker, then run the fix:

#!/bin/sh

docker exec -it deepseek-v4-flash-0731-spark-sparkinfer-deepseek-v4-flash-1 /opt/runtime-venv/bin/python -m pip install --no-deps --upgrade xgrammar==0.2.4

docker exec -it deepseek-v4-flash-0731-spark-sparkinfer-deepseek-v4-flash-1 /opt/runtime-venv/bin/python -m pip install --force-reinstall --no-deps transformers==5.13.1

docker restart deepseek-v4-flash-0731-spark-sparkinfer-deepseek-v4-flash-1
8

reading the huggingface pages, 0xSero is reaped to 216 experts but higher bits, and tpuretell is not reaped, but has less bits. so interesting options there

10

This is, by far the best coding model for single spark!!

11

While using it, I noticed a very unusual model behavior. I don’t know how to describe it in plain language.

When using hermes-agent or open code, I always set it to show the model’s thinking process on screen. On long context agentic job, the model sometimes keep on echoing ‘hello’ or list some contiguous numbers such as ‘1, 2, 3…’.

It does not go into malformed tool calling or looping, and after several minutes, it usually comback to the correct course and finish the job.

It sure is hell lot faster than entrpi’s ds4. But entrpi’s version is much more mature in my opinion.

12

@sdh2749 - in which of the two models mentioned in this thread did you see this? does it happen with both? thanks

13

It’s fast, but I have seen some weird words thrown into prose I wasn’t expecting. I was drafting a linkedin post with it this morning and it randomly decided to put some guy named Gregg in it. Not sure about this one yet.

14

It was ‘wrldsuksgo2mars/DeepSeek-V4-Flash-0731-EXL3-K2-calibrated-v0’.

15

I also tested the latest version of tool-eval-bench with the new engine:

One clarification: I did not explicitly set the thinking/reasoning mode to MAX for this run. I simply used the command below with the default model/server behavior:

$env:PYTHONUTF8="1"; & "$env:USERPROFILE\.local\bin\tool-eval-bench.exe" --base-url "http://192.168.0.25:8000" --model "deepseek-v4-flash" --parallel 1 --timeout 600 --max-turns 12 --hardmode --seed 42 --temperature 0 --no-probe-engine --output-dir "$env:USERPROFILE\Desktop\runs"

And these are the results:

Engine Total execution time Median turn Score
Old engine — Aug 11 26:49 4.7 s 85
New engine — Aug 14 13:51 2.5 s 88

By “old engine” here I mean the ds4-on-spark implementation from Entrpi:

Please pay particular attention to the execution time.

The new engine completed the benchmark about 12:58 faster, which is roughly a 48.3% reduction in total execution time, or about 1.94× faster overall.

At the same time, the score improved from 85 to 88.

So this result was achieved without me explicitly forcing MAX thinking/reasoning mode. I just ran the benchmark with the configuration above and got this result.

image
image3078×1369 145 KB

16

I also tested the model in my normal day-to-day workflow, not just in the benchmark.

So far, I haven’t noticed any regression compared to the previous version - tool calling, reasoning, multi-step workflows, etc. all continue to work as expected.

The main difference is simply that it feels significantly faster in actual use as well.

17

@siertum were you testing the lower bit model or the pruned 0xSero model? thanks

18

I agree that the model does feel fast using this recipe, and overall it does not seem degraded.

19

The model I used is the regular (non-abliterated) 2-bit EXL3 K2 quantized checkpoint of DeepSeek-V4-Flash-0731 — the version we ran before, not the uncensored variant.

HuggingFace repository: wrldsuksgo2mars/DeepSeek-V4-Flash-0731-EXL3-K2-calibrated-v0 (revision dff9afc6f5fe50a890590f7b6d5339ceaf5ba51e). It is a self-contained EXL3 K2 (2 bpw, MCG codebook) quantization of deepseek-ai/DeepSeek-V4-Flash-0731: routed expert weights (gate/up/down, including the integrated dSpark blocks) are quantized; attention, routing, shared-expert, embedding, and output head retain their source representation. Downloaded via Hugging Face Hub (snapshot_download). MIT license; tokenizer/format per the original model card.

It is served with the tpurtell EXL3 K2 Spark recipe — GitHub: GitHub - tpurtell/deepseek-v4-flash-0731-exl3-k2-spark: Pinned one-DGX-Spark Docker recipe for DeepSeek V4 Flash with EXL3, SparkInfer, and 262K NVFP4 MLA KV cache · GitHub (image Package deepseek-v4-flash-0731-exl3-k2-spark · GitHub), a vLLM-based stack on 1× DGX Spark (GB10), TP1.

Launch parameters: --max-model-len 1000000 (1M token context, model supports 1,048,576 positions), --gpu-memory-utilization 0.85, --kv-cache-dtype nvfp4_ds_mla, DSpark speculative decoding with 5 draft tokens from the built-in K2 MTP drafter, max_num_seqs 4, max_num_batched_tokens 4096. Server-side served model name: deepseek-v4-flash-0731-exl3-k2 (benchmark alias deepseek-v4-flash). The benchmark ran with default model/server behavior — no MAX thinking/reasoning forced, temperature 0, seed 42.

20

this looks very impressive - given your usage, do you think it is now at the point where it is better than qwen 3.5 122b?

21

Honestly? No — it’s not even close. Qwen 3.5 122B is simply a very weak model compared to this one. Putting them side by side is like comparing a 10-year-old child to a 20-year-old adult: the kid can do a few things and gets excited about them, but there’s no contest when it comes to real, concrete work. DeepSeek V4 Flash is several heads above Qwen in solving concrete tasks — and especially in programming. Tool calling, multi-turn agentic workflows, and code come out measurably better, and it does it faster. This isn’t a “which is better” question — it’s a different league entirely.