Optimizing DeepSeek-V4-Flash on a single NVIDIA GB10/GX10 with DSpark speculative decoding

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, ,
1

Hi everyone,

I’m working on an experimental fork of Salvatore Sanfilippo’s ds4 project, focused on running DeepSeek-V4-Flash efficiently on a single NVIDIA GB10/GX10 system.

Original project:

My experimental fork:

The goal of this fork is very specific: improve local inference throughput for DeepSeek-V4-Flash on a single GB10/GX10 machine, without changing the target model distribution.

What I implemented

The main areas of work are:

  • DSpark sidecar loading and GGUF conversion support.
  • True p/q speculative rejection sampling for DSpark drafts.
  • CUDA-side verifier logic for p/q acceptance and residual correction.
  • DSpark-specific CUDA Graph variants for drafter and verifier paths.
  • GB10-oriented Tensor Core tiny-batch experiments.
  • Q8 tiny-batch reuse and Q8/F16 hot-cache profiles.
  • Reproducible run scripts and log analyzers.

Current results

On one NVIDIA GB10 machine, with DeepSeek-V4-Flash plus the DeepSeek-V4-Flash-DSpark sidecar, I measured roughly:

Profile Throughput
Initial CUDA path ~13 t/s
CUDA Graph + fused compressor + Q8/F16 cache ~14.5-14.7 t/s
MTP sidecar experiments ~15.1 t/s
DSpark p/q rejection sampling ~16.8-17.6 t/s
DSpark p/q rejection + GPU verifier + Tensor Core tiny batches ~18.2 t/s weighted decode

Representative final analyzer output:

Fused verifier cycles:     1029
Ordinary/fallback cycles:  0
P/Q rejection cycles:      1029
Verifier acceptance:       53.18%
Mean verifier target rows: 4.956
Mean verifier draft time:  20.572 ms
Mean verifier target time: 148.339 ms
Mean fused cycle:          168.911 ms
Verifier-cycle throughput: 18.376 t/s
Weighted request decode:   18.274 t/s

The important part is that the final path uses true speculative rejection sampling. Draft tokens are sampled from q, the target model defines p, accepted tokens follow min(1, p(x)/q(x)), and rejected drafts are replaced from the positive residual distribution. So the goal is not just “faster approximate decoding”, but preserving the target sampling distribution.

Where I’m looking for feedback

The main bottleneck is still the target verifier step. In the best profile, the fused DSpark cycle is around 169 ms, with about 148 ms spent in the target verification path.

I would be very interested in feedback from CUDA/GB10 developers on:

  1. Whether the tiny-batch Tensor Core strategy is the right direction for this workload.
  2. How to better structure verifier batches for GB10/Blackwell-class hardware.
  3. Whether CUDA Graph update/replay patterns can be improved for this kind of token-by-token speculative decode loop.
  4. Whether there are recommended profiling strategies for this type of workload on GB10, especially when Nsight Compute has trouble with graph/profiled kernels.
  5. Any obvious GB10-specific CUDA optimization opportunities I may be missing.

This is still experimental code, but the full fork, scripts, README and analyzer tools are available here:

Thanks in advance for any suggestions.

2

Very impressive improvements!

How is the prompt processing speed? Still in the 350-400/s range?

In my testing the prompt processing has been a bigger interactive use (OpenCode/similar) bottleneck on established code bases rather than the token generation (although improvements there are surely also welcome).

3

Thank you!

At the moment, the prompt processing speed remains essentially unchanged. So far I’ve focused exclusively on improving generation performance.

In my case I’m using OpenClaw, and while the faster generation has made the experience noticeably better, I think prefillis definitely an area that can still be improved.

I’ll keep experimenting, and if I manage to improve prefill performance as well, I’ll be happy to share the results.

4

I am (well, codex is) trying to optimising PP on the original version.

So far I do have real gain, but I prefer to test stability before opening a PR

./ds4-bench   -m ds4flash.gguf   --prompt-file speed-bench/promessi_sposi.txt   --ctx-start 2048   --ctx-max 65536   --step-incr 2048   --gen-tokens 128
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121)
ds4: CUDA host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings: 80.24 GiB

ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 19.946s
ds4: cuda backend initialized for graph diagnostics
ds4-bench: context buffers 1742.43 MiB (ctx=65665, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=16418)
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes
2048,2048,504.43,128,13.73,52184460
4096,2048,483.11,128,13.82,80373132
6144,2048,474.04,128,13.82,108561804
8192,2048,468.40,128,13.63,136750476
10240,2048,459.51,128,13.57,164939148
12288,2048,455.77,128,13.52,193127820
14336,2048,450.08,128,13.47,221316492
16384,2048,446.39,128,13.51,249505164
18432,2048,441.45,128,13.42,277693836
20480,2048,437.23,128,13.36,305882508
22528,2048,432.11,128,13.29,334071180
24576,2048,427.54,128,13.22,362259852
26624,2048,422.74,128,13.12,390448524
28672,2048,418.40,128,13.05,418637196
30720,2048,415.24,128,12.97,446825868
32768,2048,410.14,128,12.59,475014540
34816,2048,391.67,128,12.50,503203212
36864,2048,388.39,128,12.45,531391884
38912,2048,385.15,128,12.41,559580556
40960,2048,381.05,128,12.38,587769228
43008,2048,376.91,128,12.34,615957900
45056,2048,374.26,128,12.27,644146572
47104,2048,370.82,128,12.22,672335244
49152,2048,366.47,128,12.21,700523916
51200,2048,359.70,128,12.13,728712588
53248,2048,357.41,128,12.05,756901260
55296,2048,354.78,128,12.00,785089932
57344,2048,352.01,128,11.94,813278604
59392,2048,348.95,128,11.90,841467276
61440,2048,345.97,128,11.81,869655948
63488,2048,343.20,128,11.76,897844620
65536,2048,340.20,128,11.74,926033292
5

Update: GB10 launch profile and append-prefill support

A small update on the project.

Tonight I consolidated a dedicated GB10 launch profile for the fork, aimed at sustained single-stream use with OpenClaw and long-running agent conversations.

The current profile includes:

  • the target model copied into device memory at startup;
  • a 12 GiB Q8 → F16 hot cache;
  • Q8 tiny-batch reuse;
  • the DSpark Tensor Core tiny-batch path enabled by default;
  • always-on DSpark drafting when there is a single active GB10 decode stream;
  • a physical context size of 131k tokens, with an 85% advertised-context safety guard;
  • a default 16 GiB disk budget for persistent KV checkpoints.

I have also added an initial append-prefill optimization for long conversations.

Instead of rebuilding the complete prompt state on every request, the server can retain canonical KV checkpoints near long and stable prompt boundaries. When a subsequent request shares the same prefix, it can restore the most suitable checkpoint from disk and process only the newly appended tail.

The goal is not to increase raw prefill throughput in isolation, but to reduce the effective prompt-processing cost of repeated turns in long OpenClaw conversations and established code bases.

This is especially relevant once a conversation grows to tens of thousands of tokens: the unchanged prefix should not need to be processed again from the beginning at every turn.

The implementation is still experimental and I am currently validating checkpoint correctness, hit rates, disk behaviour and real-world latency across long sessions.

The updated code and launch profile are available in the repository:

6

Update: up to 854 prefill tok/s and 24–25 tok/s DSpark decode on one GB10

A few days ago, I was asked whether prompt processing was still in the 350–400 tok/s range.

At that time, the honest answer was yes.

That is no longer the case.

After several iterations on the CUDA prefill pipeline, the latest end-to-end Athena runs produced:

Workload Result
Cold 13.6k-token prompt 787.06 tok/s average
First 8192-token cold chunk 854.26 tok/s
Average through the first 12,288 tokens 834.49 tok/s
41.7k-token append, ending at 55.3k context 724.69 tok/s average
Full chunks inside that append 790.72 → 761.57 tok/s
Append ending at 84.6k context 664.92 tok/s average
DSpark decode at 55k context 25.13 tok/s
DSpark decode at 70k context 24.39 tok/s

These are server-level measurements, not isolated GEMM microbenchmarks. Request averages also include slower partial and tail chunks.

The most important result for me is that the prefill gain no longer comes at the expense of speculative decode.

What changed

The current GB10 pipeline combines:

  • aligned in-place SoA storage for the routed-MoE weights;
  • D2R/MMQ prefill tiers for IQ2_XXS gate/up and Q2_K down projections;
  • token-tile HMMA attention for indexed sparse and dense/raw-mixed prefill;
  • a compact 68-byte MXFP4 indexer cache with native SM121a block-scaled MMA;
  • exact shape-specific Top-512: Radix for large prefill batches, a parallel chunk tree for verifier-sized batches and GVR for serial rows;
  • a dedicated Q8_K small-batch target-MoE path to preserve DSpark decode throughput.

The target GGUF weights and lossless p/q speculative rejection algorithm remain unchanged. Exact Top-K ordering, attention behavior and target-MoE numerical parity are covered by the CUDA regression suite.

The current profile still uses:

  • one NVIDIA GB10;
  • 131,072-token physical context;
  • 85% advertised-context guard;
  • 8192-token prefill chunks;
  • DeepSeek-V4-Flash with the DSpark sidecar;
  • 12 GiB Q8→F16 hot cache.

There is still work to do. My next target is keeping full-chunk prefill closer to 800 tok/s beyond 80k context and further reducing the cost of long tool-driven conversations.

But this is the first version where I feel the three goals are working together:

fast prefill, fast DSpark decode and long-context operation on a single GB10.

Code, build instructions and regression tests:

Feedback and independent GB10 measurements are very welcome.

7

Update: DeepSeek-V4-Flash breaks the 1,000 tok/s prefill barrier on a single NVIDIA GB10

Yesterday I reached an important new milestone with DeepSeek-V4-Flash running entirely on a single NVIDIA GB10.

For the first time, the full server pipeline exceeded 1,000 tokens per second on a complete 8,192-token prefill chunk, reaching:

1,009.78 tok/s

This is not an isolated GEMM benchmark or a synthetic kernel measurement. It is the actual model running through the complete server pipeline.

Some results from the latest tests:

Workload Result
Cold 13,376-token prompt 952.97 tok/s average
Best complete 8,192-token chunk 1,009.78 tok/s
Cold 25,352-token prompt 902.67 tok/s average
Append from 57.8k to 78.2k context 730.56 tok/s average
Complete chunk during the append 898.45 tok/s
Append from 77.2k to 90.5k context 760.77 tok/s average
Complete chunk at long context 891.13 tok/s
DSpark decode at 90.5k context 24.00 tok/s
DSpark decode at 93.5k context 23.46 tok/s

Crossing 1,000 tok/s is exciting, but I believe the most important result is actually the behavior at long context.

Prefill performance degradation is now much more contained. Even while appending tokens at around 77k–90.5k context, the request still averaged 760.77 tok/s, with a complete central chunk reaching 891.13 tok/s.

In other words, this is no longer only a fast short-prompt benchmark. The optimized pipeline continues to perform well as the context grows.

The latest improvements came from further reducing intermediate memory traffic and fusing several operations that were previously executed as separate CUDA stages, including hidden-state expansion, RMS normalization, inverse-RoPE packing and routed-MoE reduction epilogues.

These changes work together with the optimizations already implemented in the project:

  • token-tiled HMMA attention;

  • native SM121a MXFP4 indexer scoring;

  • exact shape-specific Top-512 selection;

  • optimized D2R/MMQ routed-MoE prefill paths;

  • persistent F16 compressed KV;

  • append-prefill and canonical KV checkpoint reuse;

  • DSpark speculative decoding with a dedicated Q8_K small-batch verifier path.

The entire system has now been tested end to end beyond 93k tokens of context, while maintaining stable speculative decoding, tool calling and KV reuse, without sustained system paging.

The current target is becoming increasingly realistic:

near-1,000 tok/s practical prefill, limited degradation at long context and stable speculative decoding on one GB10.

There is still plenty of room to improve, especially around long-context attention, memory scheduling, Tensor Core utilization and speculative-token acceptance.

I would genuinely welcome suggestions, criticism, profiling ideas or alternative CUDA strategies from anyone working on GB10, Blackwell, LLM inference or speculative decoding. Even small observations could point toward the next major optimization.

Code, benchmarks, build instructions and regression tests are available here:

Independent benchmarks on other GB10 systems would also be extremely valuable.

The goal is simple: find out how far a single GB10 can really be pushed.

8

Thank you very much for sharing! I’ve got couple of questions about constrains at the moment:

  1. Why DS4_DSPARK_MAX_KV is set to 65536? Do I understand right - this means that all conext above this number will be at plain decode path?
  2. Can you recomment settings to have 262k context length usage?
9

Thank you!

DS4_DSPARK_MAX_KV=65536 does not limit the target model context. It only limits the DSpark speculative path; above that threshold, generation continues with the normal target-model decode path.

For a 262k context, set:

DS4_CTX=262144 ./run-dspark-server.sh

The current launcher does not explicitly set DS4_DSPARK_MAX_KV, so the internal/default value is used.

I am currently testing a ~263k physical context on my GB10, and so far it appears to be working correctly. I will share memory usage, long-context prefill and decode results once the test is complete.

Suggestions or alternative settings for keeping DSpark effective beyond 65k are very welcome.

10

Update: 836 tok/s prefill at 180K context, 256K default, and an experimental 1M profile on one GB10

I have another long-context update for the single-GB10 DeepSeek-V4-Flash
runtime.

The most important result is not a new short-prompt peak. This time I found and
removed an artificial performance cliff that appeared exactly after 131,072
tokens.

The server now uses a 262,144-token physical context by default, with the same
85% advertised-context guard, 8,192-token prefill chunk, 12 GiB Q8-to-F16 hot
cache and DSpark sidecar.

The 131K cliff

The ratio-4 sparse-attention path has one compressed row for every four context
tokens. The token-tile indexed attention fast path still contained a fixed
n_comp <= 32768 guard, so it stopped being eligible at exactly:

32768 compressed rows * 4 = 131072 context tokens

The chunk ending at 131,072 was still around 860 tok/s. The next chunk dropped
to 648 tok/s because execution fell back to the older indexed-attention path.

That fixed limit is required by the F32 path, which converts the compressed KV
into a preallocated F16 mirror. It is not required by the operational
direct-F16 path: that path already stores compressed KV as F16 and reads only
the exact Top-512 selected rows.

The direct-F16 eligibility check now uses the real row capacity of the
compressed-KV tensor. The F32 mirror path and dense path retain their existing
limits. No new score matrix, mirror, persistent cache or scratch allocation was
added.

Grafico aggiornato
Grafico aggiornato1920×1080 125 KB

Measurement Before Dynamic direct-F16 Change
First chunk after 131K 648.39 tok/s 858.48 tok/s +32.40%
Deep append request average 624.92 tok/s 836.16 tok/s +33.80%

The request averages cover different deep-context intervals
(147.2K → 207.1K before, 127.8K → 180.8K after), so they are not presented as
a position-matched microbenchmark. The chunk-level curve is the stronger
evidence: after the old cutoff, the updated run measured:

858.48, 849.68, 846.14, 842.61, 833.60, 828.20 tok/s

The remaining decline is gradual as the context grows. The abrupt 131K cliff is
gone.

End-to-end operational run

These are server-level measurements from the same real tool-driven workload,
including partial chunks and tails:

Context interval Appended tokens Average prefill
cold → 25.9K 25,881 911.36 tok/s
27.7K → 95.1K 67,316 913.15 tok/s
95.1K → 125.3K 30,204 859.77 tok/s
127.8K → 180.8K 53,017 836.16 tok/s

The last append completed in 63.406 seconds. DSpark then generated 290 tokens at
19.89 tok/s at 180.8K context. Decode is content- and acceptance-dependent, so
I am reporting that number as an observed long-context result rather than a new
decode peak. No CUDA error, OOM or attention fallback appeared.

Memory and resumed prefill

Two supporting changes made the larger context practical:

  • the indexer-score workspace and routed-MoE workspace now share one arena
    because their lifetimes do not overlap, reducing allocated context scratch by
    exactly 1,024 MiB;
  • resumed prefill processes only the unaligned compressor fringe
    incrementally, then replays the aligned body through the batched compressor
    path.

Neither change alters model weights, accessible KV rows, exact Top-512
selection or speculative sampling semantics.

The CUDA regression now executes the direct-F16 attention kernel with 32,769
compressed rows and compares it with the indexed reference:

ds4: CUDA FlashMLA direct-F16 dynamic compressed capacity enabled (rows=32769 tensor-cap=32769)
cuda-regression: token-tile indexed direct-F16 attention above 32K rows rel-rmse=0.000091 max-abs=0.000040
cuda long-context regression: OK

Current launch profiles

The recommended single-GB10 profile is now:

physical context: 262144
advertised context: 85% (222822)
default max input: about 220622 tokens after output reserve
prefill chunk: 8192
Q8 -> F16 hot cache: 12288 MiB
DSpark: enabled, draft depth 5

Start it with:

./run-dspark-server.sh

I also added a separate capacity-first launcher for an experimental 1M physical
context:

./run-dspark-server-1m.sh

The 1M profile uses a 4,096-token chunk, an isolated disk-KV directory and the
same 85% guard. It advertises 891,289 tokens and reserves the output budget from
the accepted input. Chunk 8,192 is intentionally not used there because that
combination exceeded the measured GB10 unified-memory budget. The 1M profile
has passed an initial real run beyond 163K tokens, but I am not claiming a
completed 1M-token session yet.

The repository, build instructions, regression tests and detailed rejected
experiments are here:

Independent GB10 measurements are very welcome, especially beyond 180K context.
The next goal is to preserve this long-context prefill curve while improving
DSpark verifier throughput.

11

Update: 1,000 tok/s prefill, compact Q2 DSpark and up to 1M context on one GB10

The GB10 fork is now packaged so other users can reproduce the latest results
without manually assembling the model and DSpark sidecar.

What started around 350-400 tok/s prefill and 13 tok/s decode now reaches more
than 1,000 tok/s prefill and commonly 24-26 tok/s DSpark decode.

Latest end-to-end results on one NVIDIA GB10:

Workload Result
Short/medium prefill 900-953 tok/s average
Best complete 8,192-token chunk 1,009.78 tok/s
Long append, 27.7K to 95.1K context 913.15 tok/s
Deep append, 127.8K to 180.8K context 836.16 tok/s
DSpark tool-call decode commonly 24-26 tok/s
Clean mixed coding session 20.88 tok/s weighted

The main additions are:

  • a compact Q2 DSpark sidecar, now the default, using about 5.06 GiB less
    unified memory     than Q4;
  • 262K physical context by default with an 85% client guard;
  • an experimental 1M-token context launcher;
  • stable memory behavior in long sessions;
  • sustained temperature observed around 75 C on Athena;
  • a one-command installer that downloads the target, creates the DSpark
    sidecar, runs the CUDA regression and compiles the server.

The target model remains authoritative. DSpark still uses lossless p/q
rejection sampling, so the compact drafter does not replace or approximate the
target distribution.

Install the recommended Q2 configuration:

cd "$HOME" && git clone https://github.com/xangel82/DS4-GB10-GX10-DSpark-CUDA.git && cd "$HOME/DS4-GB10-GX10-DSpark-CUDA" && ./install-gb10.sh --install-deps --dspark q2

Start the standard 256K profile:

./run-dspark-server.sh

Or start the experimental 1M profile:

./run-dspark-server-1m.sh

Models default to $HOME/ds4, but both the installation and runtime paths are
fully configurable.

Code, full installation guide, benchmarks and technical notes:

Independent GB10/GX10 results, especially for Q2 acceptance and the 1M
profile, would be very welcome.

12

DS4 GB10 update: 35 tok/s decode, 1K tok/s prefill, 5 GiB freed

Two strong updates for DeepSeek-V4-Flash on a single NVIDIA GB10:

  • New compact Q2 DSpark sidecar: 5.64 GiB, saving 5.06 GiB UMA versus Q4.
  • Typical tool-call decode: 24-26 tok/s.
  • Lossless HybridLC: up to 34.97 tok/s when suffix retrieval is covered.
  • Fixed the shared-arena fused-D2R dispatch: 960.94 → 1001.94 tok/s on the first 8K block, with 1006.61 tok/s on the next.
  • Greedy hashes unchanged, CUDA parity bit-exact, no extra persistent memory, DSpark verifier unchanged.
  • Default physical context: 256K; experimental profile available up to 1M.

No quality shortcut: the target model remains authoritative and speculative tokens are verified with exact residual correction.

Repository and automated GB10 installer:

13

How does this compare to the version @entrpi put together?

Also, any plans to push back upstream to antirez’s branch?

14

@entrpi @marco.palaferri

**Side-by-side measurements: DS4-GB10-GX10-DSpark-CUDA and Entrpi/ds4 v0.4.2 on the same 1× GB10
**
Disclosure, so my position is clear: I’m an independent user with no affiliation to either project — not a contributor to this repo, and my only interaction with the Entrpi fork has been as a user reporting benchmarks in its thread (one small server patch I wrote for my own deployment was later incorporated upstream there). I run DeepSeek-V4-Flash for my own local agent workloads and my interest is purely in accurate numbers on shared hardware: everything below is measured on my machine with the same methodology applied to both forks, raw data available on request. Corrections to my setup or methodology are welcome — I’ve had my own benchmarking mistakes corrected in the other thread and expect the same standard applies here.

I’ve been benchmarking DeepSeek-V4-Flash serving on a single GB10 (121 GB) and ran both this repo and Entrpi/ds4 v0.4.2 on the same box, same target GGUF (DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix), same client-side methodology (llama-benchy 0.3.3, pp 2048 / tg 128 / --no-cache, one run per point unless noted). Posting the full dataset since cross-fork numbers on identical hardware seem rare.

Setup notes for this repo: built from main (9814007), Q2 sidecar assembled from the official HF shards, cuda-regression CUDA_ARCH=sm_121a passed. One packaging note: install-gb10.sh refuses to run if any ds4-server process exists — including one from a different fork — even with --skip-regression; I ran the download/sidecar/build steps manually to work around it.

Prefill vs depth

image
image1260×690 64.6 KB

depth this repo pp t/s Entrpi v0.4.2 pp t/s
2k 852 597
8k 899 810
32k 938 736
64k 955 585
128k 903 525
176k 886 —
512k (1M profile) 722 249
850k (1M profile) 618 not measurable*

* the Entrpi client path at 850k exceeds the benchy client timeout at that config’s prefill rate.

The README’s 900–953 t/s claim reproduced on my hardware, and the profile is notably flat with depth. The 850k row is the first client-side request I’ve completed at that depth on this box (TTFR 23.7 min). The --advertise-context-pct 85 guard puts the 1M profile’s practical ceiling at 891,289 tokens.

Decode (tg128 windows; short windows at depth carry one-time graph/cache warm-up cost on both forks, so treat deep means as conservative):

depth this repo tg t/s (peak) Entrpi v0.4.2 tg t/s
2k 15.6 (21) 26.3
8k 19.7 / 12.9 repeat (30 / 19) 23.9 (stable across repeats)
32–128k 10.0–12.0 (15–23) 13–15
176k 18.5 (26) —
512k 23.6 (35) ~9–16
850k 13.7 (15) —

Two observations here: shallow-depth decode showed run-to-run spread on this repo (19.7 then 12.9 on identical warm 8k runs — possibly the adaptive K scheduler; happy to collect telemetry logs if useful), while the deep-context behavior is the standout — 23.6 t/s at 512k is on par with the other fork’s shallow decode.

Concurrency

image
image1080×630 42.4 KB

— llama-benchy --concurrency 1 2 4 at 8k depth, aggregate decode peak:

concurrency this repo Entrpi v0.4.2
1 23 21
2 27 30
4 28 45

Consistent with the run script’s single-active-stream design: concurrent requests largely serialize here (per-request peaks stay near single-stream values while others wait), whereas the Entrpi continuous-batching path scales aggregate ~2.1× at c4. Prefill did not scale with concurrency on either fork.

Summary of the trade space as measured on this hardware: this repo’s device-copy weight strategy delivers roughly 1.1–2.9× the prefill throughput at every depth and holds decode remarkably flat into the hundreds-of-thousands of tokens; the Entrpi fork provides higher and more stable shallow-depth decode plus multi-request batching. Which matters more is workload-dependent — single-user long-context sessions vs multi-agent serving.

For context on why these particular axes: my own use is multi-agent serving at mostly 2k–30k depth with 2–6 concurrent requests, so the concurrency and shallow-decode rows are the ones that matter for me — readers running single-user long-context sessions will weigh the depth rows differently.

All raw benchy JSON, serve logs, and the regression output are saved; happy to share any of it or run additional configurations (Q4 sidecar, prefill-fast/lean memory profiles, telemetry runs) if that’s useful to the project.

15

Thank you very much for taking the time to run such a thorough and independent benchmark.

Your comparison is extremely valuable to me, not only because it confirms several of the results I had observed, but especially because it clearly highlights where the branch still needs work.

The decode variability, short-context behaviour and multi-request scaling you identified have given me some very concrete directions for the next development steps. I will focus the upcoming work on those areas.

Detailed feedback like this is exactly what helps an open project improve. I truly appreciate the time, effort and technical care you put into testing it.

Thank you again.

16

I did look into the possibility of integrating some of the work, but the implementations have diverged significantly, especially around DSpark, the verifier and the CUDA execution path. For that reason, a direct merge is not straightforward at the moment.

Regarding antirez’s original project, I have already shared my fork with Salvatore. I hope that, if he finds some of the changes useful and suitable for upstream integration, there may be an opportunity to contribute them back.

17

Thank you @agupta30 for the careful side-by-side and for tagging me two days ago. That tag is what brought this fork to my attention, and I’ve spent some time since reading its code and commit history closely. Your measurements match my own receipts where they overlap. The review turned up things worth sharing, about where this fork’s code comes from and what it adds.

Congratulations, Marco. A working p/q rejection sidecar, HybridLC, and 1,000 t/s prefill on a single GB10 is a lot of ground covered in two weeks. Documenting discarded experiments as carefully as the wins also made my review more interesting.

@Zambonilli asked how this fork compares to mine, and Marco replied that he had looked into integrating some of the work but found the implementations had “diverged significantly.” That’s true of the serving architectures. But the two trees are not independent codebases, and the part that drives the prefill numbers in this thread has not diverged much: it is largely the same code.

The prefill kernel stack in this repo originated in my fork: the routed-MoE D2R/MMQ prefill tiers, the aligned-SoA repack machinery, and the token-tile HMMA attention. Commit 4eb7441 here (July 17) imported my cuda/mmq directory from my then-current v0.2.x public tree, 38 of its 41 files identical, including my internal test and prototype harnesses, and commit 699245d adapted my token-tile attention kernels the same day. At this repo’s current head (9814007) the kernel tree under cuda/ is about 33,000 lines:

lines origin
~17,000 authored in Entrpi/ds4, imported byte identical or lightly adapted (D2R kernels, MMQ adapters and repack machinery, test and prototype harnesses)
~13,000 llama.cpp mmq kernels, vendored through Entrpi/ds4 (pin and per-file inventory in cuda/mmq/VENDOR.md, itself my file)
~3,000 Marco’s additions (the fused gate/up pipeline and token-bound stream-K inside those files, plus his cuda/indexer MXFP4 scorer)

Another ~2,000 lines adapted from my tree wire that stack into ds4_cuda.cu (the token-tile attention port and the MMQ integration glue). Counts are from git blob identity against my repo’s object store; happy to share the method. Marco’s repo acknowledges the lineage at the code level: ds4_cuda.cu carries “Adapted from Entrpi/ds4 commits 47438d7 and 9de3044 (MIT)”, the startup banner prints “CUDA Entrpi batched MMQ MoE prefill enabled”, and the Italian README-GB10.md links my repo and names the specific commits. I appreciate all of that.

The one place I’d ask for a change is the English README, since that’s what most readers of this thread will see. Its “What this fork delivers” list presents the D2R/MMQ prefill and token-tile HMMA attention as fork deliverables, and the “License and attribution” section describes the CUDA modifications as “Copyright (c) 2026 Marco Palaferri” without mentioning that roughly nine tenths of the kernel tree was imported from Entrpi/ds4. Marco, I’d appreciate it if the lineage that’s already in your source comments and Italian docs were stated with the same prominence there: a line in the attribution section distinguishing the imported Entrpi/ds4 prefill stack from your own modifications would settle it.

MIT permits all of this reuse; this is about readers being able to tell who built what. I held my own repo to the same standard just now, after deriving a commit from your stack: my README now carries a license and attribution section spelling out the lineage of every layer in my tree, upstream’s engine, the llama.cpp kernels vendored under cuda/mmq, my fork’s serving and prefill additions, and the pieces adapted from your fork.

Screen Shot 2026-07-29 at 13.08.39 PM
Screen Shot 2026-07-29 at 13.08.39 PM1320×658 114 KB

Measured in ideas rather than lines, the ledger is more even. The foundation mechanisms of the prefill pipeline came from my tree, but there are great updates on top of it which are Marco’s contributions: single-pass fused gate/up with token-bound stream-K, fused epilogues with a runtime self-test, native block-scaled MMA scoring on the packed FP4 mirror, lifting the 131k fast-path ceiling, and context-bucketed adaptive verification. None of that came from my tree. On my side, my branch has just ported two of his epilogue fusions re-derived against my own kernels, and also his fused gate/up prefill pipeline, re-tuned for my serving shapes. Fully prominently attributed. That is the standard I try to hold in both directions: commit credit when I follow a design, a copyright notice in the file when I adapt code. More of his work may yet inspire additional improvements in my v0.5 prefill-performance-focused release.

For anyone weighing the benchmarks: the prefill-at-depth rows in the side-by-side are largely measuring Marco’s real optimizations on a shared kernel base against an older snapshot of that same base. His deep-prefill wins as of that snapshot are genuine. A good part of that gap has since closed on my public branch and will land in the next tagged release once quality evals are done.

Back to Marco’s integration point. The serving divergence is really between upstream and my fork: antirez’s tree serves sessions one at a time with KV checkpoints, and Marco kept that machinery and built his DSpark sidecar and frontier caching on top of it, while my fork fully rebuilt the serving layer into a continuously batched engine. His requests serialize under concurrency because of that inheritance. It also bears directly on the priorities he named after agupta30’s benchmark: decode variability, short-context behaviour and multi-request scaling are serving-layer properties, and they are where most of my fork’s effort went. My tree is roughly 460 fork-side commits since mid-May; only about a quarter of those are the prefill kernel arc he imported, which came together in the final three weeks before v0.1.0. The rest is the engine underneath it: admission and eviction, KV banks, warm starts, graph capture, and the hardening that makes concurrency reliable.

So a wholesale merge of the two forks isn’t the right near-term goal, but the shared kernel base makes narrower collaboration tractable, and I’m open to it. I’d welcome his kernel improvements as PRs against the live base they were built from, with his attribution intact, and I’m happy to help rebase his prefill stack onto my current tree, since his snapshot is from mid-July and the base has moved a lot since. If multi-request scaling is where his focus is going, that rebase is also the shortest route: his kernels would inherit an engine that already scales, instead of waiting for one to be rebuilt inside a session-centric tree.

Zambonilli also asked about plans to push back upstream, so here is where I stand for my fork. Its job is to be an exemplar of what this hardware can actually do: roofline first, everything else after. That is why the releases are shaped the way they are. Each one closes a single ledger, prefill in v0.5, decode after that, then batching efficiency and robustness, and each closes with no unexplained gaps, every remaining millisecond either fixed or attributed to a named cause. Upstream ds4 is intentionally narrow and self-contained, and the way to respect that is not to aim a moving target at it. Once a piece has stopped moving and proven itself at roofline, it becomes a candidate for upstreaming in a shape Salvatore would actually want to maintain.

The eventual catch-up to upstream’s head is nontrivial because of what the serving layer became on my side: a ground-up rebuild into an engine that parallelises efficiently across many concurrent streams and is designed for fast resumption of separate workstreams, which is what you would normally run vLLM or SGLang to get, here self-contained in C/CUDA with none of their runtime overhead. Preserving that layer is the fixed constraint, and it is also why the catch-up comes last in the sequence: each of the passes above still reshapes the engine, and reconciling with upstream once, against its settled form after the kernels are fully optimized, beats redoing that reconciliation after every release. I’ll reach those rooflines one way or another, and contributions are welcome.

18

Thank you for raising this. The Entrpi lineage was documented when the code entered the repository: commit 4eb7441introduced the routed-MoE MMQ backend with its Entrpi/llama.cpp provenance, while 699245d explicitly documented the token-tile HMMA port from Entrpi commits 47438d7 and 9de3044. The top-level README also named the Entrpi MMQ path in its runtime banner.

However, you are right that the later English showcase refactor no longer distinguished the imported foundation from my subsequent additions prominently enough. I have corrected that: the README now maps the Entrpi foundation directly to the additions made in this fork and limits “What this fork adds” to original or substantially adapted work.

Updated v0.5 prefill results are very welcome. I am currently working on the agent and physical multi-session batching path, so comparing the two stacks should be genuinely useful.

19

Thank you both for all the amazing work and such thoughtful responses. The work by all three of you is a testament to the power of open source.

20

Update: physical multi-agent DSpark batching on a single GB10

After @agupta30’s independent benchmark highlighted multi-request scaling as the clearest remaining weakness, I shifted the focus from single-stream peaks to physical multi-session execution.

The latest release introduces a hardware-aware DSpark coordinator that can combine compatible verifier work from independent sessions into physical R=2 and R=3 CUDA cohorts.

What is now implemented:

  • physical batching of compatible QKV, FFN, output, vocabulary-head and CSA/HCA attention work;
  • independent KV frontiers, RNG and rejection state for every session;
  • KV-aware routing that considers both resident prefixes and queued work;
  • cooperative prefill/decode scheduling at safe 8192-token chunk boundaries;
  • two warm resident lanes, with a third activated elastically only when sufficient UMA headroom remains;
  • automatic fallback to R=2 or R=1 when agents diverge into tool calls, prefill or completion.

Measured on one NVIDIA GB10:

Measurement Result
Operational R=2 cohorts 24.88 aggregate t/s, +7.7%
Complete elastic R=3 agent run 24.38 aggregate t/s
Physical R=3 cohorts 35.39 t/s vs 23.15 t/s serialized, +52.8%
Physical R=2 cohorts 21.43 t/s vs 19.04 t/s serialized, +12.6%
Overall speculative acceptance 66.62%

These are aggregate system rates, not per-client rates. The complete R=3 run is lower than the physical-cohort peak because real agents do not remain synchronized: they enter tool calls, prefill different amounts of context and finish at different times.

Approximately 40.5% of the measured verifier work used physical cohorts, so the next objective is clear: increase physical coverage without delaying independent requests.

There is no quality shortcut. The target model remains authoritative, every session preserves its own state, and speculative tokens still use lossless p/q rejection sampling with exact residual correction.

Code and detailed measurements:

A broader thought

@Zambonilli, I agree. This is the spirit in which I approach open source: sharing useful work with the community so that everyone can benefit from it.

Difficult engineering problems are rarely solved by one person or one idea alone. Different experiments, measurements and perspectives help the whole ecosystem move forward. When one idea succeeds, everyone gains from the knowledge, tools and improvements that follow.

I started this project because I wanted DS4 to work well enough to become a genuinely useful daily system on Athena, my single DGX Spark. Buying a second Spark was not financially realistic for me, so instead of scaling out, I began exploring how much more performance could be extracted from the hardware I already had. This became my path toward running and using a frontier-class model locally on one compact machine. If the solutions developed along the way also help other people achieve more with the hardware they can afford, then the work becomes useful far beyond my original goal. if the solutions developed along the way also help other people run larger models more efficiently, then the work becomes useful beyond its original goal.

More broadly, I believe advanced AI should become accessible to everyone, rather than remaining limited to a small number of organizations or only to those who can afford expensive infrastructure and services. Open source can be one of the keys to achieving this: it turns engineering knowledge into something people can study, reproduce, improve and share.

The most valuable outcome is seeing useful ideas travel across projects, get tested in new environments and become better. Every improvement that makes local AI faster, lighter or easier to run allows more people to learn, experiment and build with it.

If even one line of code or one idea from this project helps another implementation gain a single additional token per second, it would be an honor to know that I made a modest contribution to that progress.

The code developed here is available for anyone to use, test and improve, and I am open to any collaboration that helps make advanced AI more accessible and moves the whole ecosystem forward.