DeepSeekv4 Flash for 1x Spark [REAP25] [PrismaAURA]

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, ,
1

Measured-quantization DeepSeekv4-Flash build for a single GB10 (92/100 tool-use, 16.5 tok/s spec decode)

Sharing what I’ve been building: a stripped down CUDA-only fork of antirez/ds4 tuned for a single NVIDIA GB10, plus a matching GGUF whose quantization was measured, not hand-picked. It runs fully resident with room for a 1M-token context, with the DSpark speculative drafter merged into the model file.

It ships as a single binary, ds4-server — an OpenAI- and Anthropic-compatible HTTP server, so you can point Codex CLI (/v1/responses), Claude Code style clients (/v1/messages), or anything OpenAI-shaped straight at it (./ds4-server -m ds4flash.gguf --ctx 1048576).

There are a couple of other excellent GB10 ds4 forks going around right now (@marco.palaferri and @Entrpi) — this is a third independent take. Our angle is the quantization allocation so I’ll focus there.

Results (measured on one GB10)

Metric Result
Tool-use quality (tool-eval-bench hardmode, composite) 92 / 100 ★★★★★
Decode, speculative (0–8k context) 16.5 tok/s, flat with depth
DSpark draft acceptance (α), structured/tool workloads 77.2%
Prefill, 2k → 8k prompt ~420 → ~390 tok/s
Resident size (weights + merged drafter) 91 GB

Numbers come from tool-eval-bench (agentic tool-use, hardmode, seed 42, temperature 0.95, top-p 0.38) and our own prefill/decode microbench. The composite folds quality, deployability, and responsiveness. Decode is measured with the drafter on (acceptance-adjusted effective tokens/s); it stays flat from 0 to 8k context.

What’s actually different: measured-KL format allocation

The interesting result isn’t the top-line speed — it’s that choosing quant formats by measurement beat a good hand rule by 8 composite points at equal size and equal speed.

Instead of “2-bit experts everywhere,” each routed-expert tensor’s reconstruction error was measured per candidate format against the FP8/FP4 QAT source, weighted by an empirical Fisher sensitivity, and allocated under the byte budget by an exact knapsack (built on PrismaQuant). The result is a per-layer, per-projection mix:

  • An IQ2_XXS floor (2.06 bpw) on most experts.
  • MXFP4 (4.25 bpw) promoted on the quality-sensitive layers — and the measurement found a depth pattern: early layers want it on gate/up (they shape routing), late layers want it on down (they write results).
  • MXFP8 on attention, shared experts, and the head, on the tensor-core
    FP8 path.

The MXFP4/MXFP8 point is the part a portable k-quant GGUF can’t copy: those formats are the checkpoint’s source encoding, so promoting a layer to MXFP4 is a byte-lossless re-encode — zero requantization loss at 4.25 bpw. Experts are also REAP-pruned 25% to buy the residency headroom.

Spec decode: exact sampled acceptance, and deterministic

The drafter is used at every temperature, not just greedy. We build the draft distribution and verify with the standard p/q rejection rule (min(1, p/q) accept, residual resample on rejection), so the output distribution is provably identical to plain sampling — we validated it with a χ² oracle against the target’s own per-position marginals, not just an eyeball check. The decode/prefill numerics are also run-to-run deterministic (same seed, same hardware → same tokens), which we wanted for reproducible evals and debugging.

Honest scope

  • One session, up to 1M tokens. A single-user local-inference design point today; multi-session batching is on the roadmap, not shipped.
  • ds4-server only — the GGUF uses custom tensor types (MXFP4/MXFP8/IQ2 mix, REAP-pruned expert layout, merged drafter) and won’t load in llama.cpp/ollama/vLLM.
  • Beta. Weights are RAM-resident by design; a model that doesn’t fit is rejected at load rather than silently degraded.

Links

Happy to answer questions on the allocation method or the spec-decode math.

Next up: temperature-matched draft sampling (should push acceptance past the
greedy ceiling) and multi-session serving.

2

Update — v0.2.3 is out. Small but dense release since the last post: DeepSeek-V4-Flash on a single GB10 now prefills ~18% faster, holds three concurrent 1M-context sessions instead of two, and it’s quality-neutral — same tokens, measured. Two changes carry it.

Experts moved to the source’s own numerics (the prefill win). The routed MXFP4 experts now run the CUTLASS block-scaled type-40 W4A8 path — fp4 weights × E4M3 activations, on the sm_120 f8f6f4 tensor cores. The reason this is free quality-wise is the fun part: DeepSeek-V4-Flash’s own config ships expert_dtype: fp4 with activation_scheme: dynamic, fmt: e4m3 — the model was designed to compute its experts in exactly fp4×E4M3. Our previous path quantized activations to Q8_K (a legacy k-quant the source never uses), so W4A8 isn’t a compromise — it’s more faithful to the original than what we shipped before, and it’s faster. Prefill went 365 → ~410–430 t/s.

A couple of notes for anyone attempting the same: it needed a grouped single-projection GEMM so the mixed IQ2+MXFP4 layers batch cleanly. And tensor-core experts only beat CUDA-core dp4a because MXFP4 is already a native tensor-core format (zero dequant) — we measured the same move on the 2-bit IQ2 experts and it’s a net loss, since the 2-bit→8-bit dequant expansion costs more bandwidth than the tensor cores save. So IQ2 stays on dp4a; that’s the wall if you’re chasing “all experts on tensor cores.”

Pre-stored MXFP8_LT layout (the memory win). The MXFP8 attention/shared/head weights were being de-swizzled into cuBLASLt’s layout at runtime and held as a second resident copy — ~6.4 GiB of double-storage. Pre-storing the swizzled layout (a new MXFP8_LT weight type, byte-identical, GGUF size unchanged) lets cuBLASLt read them in place, which frees the copy and lifts the live-session cap from two to three on a warmed box. Decode stayed neutral.

Measured (GB10, greedy, DSpark k=3):

Context Prefill (t/s) Decode structured Decode prose
~2.3k ~390 ~24 ~21
~9.3k ~413 ~26 ~20

Quality: fresh 4-seed tool-eval-bench hardmode run averages ~90/100 (range 87–92); fidelity vs the pre-flip build is neutral (KL 0.007, top-1 100%). Being honest about the format-agnostic reality: sustained prefill lands ~410–430 t/s — the same ~400 band the other GB10 forks reproduce; the four-figure “peak” numbers are cold best-case profiles at large chunk sizes, not sustained.

Grab it with ./download_model.sh v5mx (weights tagged v0.2.3 at https://huggingface.co/twaggs88/DeepSeek-V4-Flash-REAP25-DSpark-ds4-GGUF; DSpark drafter merged in-file, auto-enables), build make cuda-spark CUDA_ARCH=sm_120f.

Next up: batched multi-session decode — one weight sweep over N co-scheduled sessions, ~1.67× aggregate at 3 concurrent (spec stays on when a session runs alone, plain-batched when sharing). That’s what the freed memory is for. Will report back when it lands.

(and if you missed v0.2.2: DSpark draft depth k=5→3, +14–52% decode across shapes.)

3

Thank you for mentioning my work and for sharing the details of your REAP25/PrismaAURA release. Your native MXFP4 expert path is particularly interesting, as is the decision to retain the IQ2 experts on dp4a after measuring the dequantization overhead.

I would be very interested in testing your model. Unfortunately, I cannot download another large model at the moment because of temporary bandwidth limits.

In the meantime, my experimental GB10 fork has progressed considerably. The latest end-to-end results on a single GB10 include:

  • up to 854.26 tok/s on the first 8192-token cold prefill chunk;

  • 787.06 tok/s average on a 13.6k-token cold prompt;

  • 724.69 tok/s average on a 41.7k-token append ending at 55.3k context;

  • approximately 24–25 tok/s DSpark decode at 55k–70k context.

The current pipeline includes token-tile HMMA attention, D2R/MMQ routed-MoE prefill paths, an MXFP4 indexer cache with native SM121 block-scaled MMA, exact Top-512 selection and a separate small-batch target-MoE path to preserve DSpark decode performance.

Since your model uses a mixed IQ2/MXFP4 expert layout, I cannot assume that it will be immediately compatible. However, should you have time to run it against my fork, it would be extremely useful to see which parts work as-is and which adaptations may be required. Combining your model and weight-layout work with the runtime optimizations in my fork could make for a very interesting cross-test.

Latest results and technical details:

Repository:

Thanks again for the citation and for sharing your measurements.

4

Marco — thanks, this made my day. The citation was the least I could do; your indexer and prefill work is a big part of what made this tractable in the first place.

Your prefill numbers are no joke — 787 average on a 13.6k cold prompt is well ahead of our ~410–430 t/s sustained. I’ve put real time into that path, so rather than just concede the gap, here’s my honest read of why it exists:

Prefill on this model is compute-bound, not bandwidth-bound (the experts sit at a couple percent of the memory roofline), so the whole game is filling the tensor cores. We got the MXFP4 experts there — the CUTLASS type-40 W4A8 grouped GEMM runs them ~2.6× faster per layer than dp4a — and we’ve got the expert-major L2-reuse CTA schedule and widened token tiles. That’s what took us to ~410–430. A nice side effect: that W4A8 path (fp4 weights × E4M3 activations) is exactly what DeepSeek’s own config specifies (expert_dtype: fp4, activation_scheme: e4m3), so it landed fidelity-neutral — we were previously diverging from the source by running Q8_K activations.

Two walls stop us short of your numbers, and both are as much choices as physics:

  1. The IQ2 experts can’t leave dp4a. They’re the majority of the routed layers, and as you found, the 2-bit→8-bit dequant needed to reach the tensor cores costs more bandwidth than the tensor cores save (~72 ms/layer vs ~62 for dp4a; E4M3 is also ~2.5% RMS lossy on the IQ2 codebook). So they’re a CUDA-core floor under the whole expert path — MXFP4 is the only format that escapes it.
  2. We haven’t taken the HMMA-attention route — which is exactly the one you did. It’s the biggest single lever left, and it’s the fp16/non-bit-exact path; we’ve kept attention exact and spec-decode acceptance exactly sampled, so we’ve effectively traded ~2× cold prefill for determinism. You also run 8192-token prefill chunks vs our conservative 4096, which fills the small-M expert tiles better.

So honestly, ~410–430 is close to the ceiling of the exact path — closing to your numbers means the HMMA-attention front-end and bigger chunks, i.e. your side of that tradeoff, and it clearly pays on cold prefill. Where we’ve pointed our own energy instead is memory and concurrency: pre-storing the MXFP8 weights swizzled freed ~6.4 GiB of runtime repack, and I’m mid-build on batched multi-session decode (co-scheduled sessions sharing one scratch working set, so concurrency stops being scratch-bound).

A cross-test sounds great — I’d genuinely like to see your HMMA-attention prefill run over our W4A8/IQ2 mix; the two might compose well (your prefill front-end + source-faithful experts + the batching). I’m auditing my rats nest of tools right now, so if you’ve got the source weights you should just be able to run my quantize/REAP pipeline and get my checkpoint out. I’ll try to have that done in the next hour or so.

A note on our REAP: I used eouya2’s REAP25 as a starting point. I fully intend to use @flash3’s RIY on a representative workload of mostly code, very little prose.

Good to be building the same thing from different angles. Cheers.