Agreed, I feel the same. DGPP for sure, especially with Qwen on 2xDGX is fantastic. I tried it with GLM and it wasn’t as fast for me as some of the other options out there. As for local AI vs. cloud, my cloud AI usage is down by over 90%.
Hi @door.blu!
This is actually a restriction of the checkpoint’s chat template, not dgpp itself persay. The model checkpoint’s chat template only allows a system message as the first message in a request. Additional system messages, even two consecutive ones at the start, cause it to reject the request. DGPP also converts developer messages to system, so those can trigger the same error.
Qwen’s template understands system, user, assistant, and tool. Passing developer through unchanged would produce “Unexpected message role.” DGPP therefore maps it to system, allowing application instructions from clients using developer to reach the model. Qwen template (https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8/blob/main/chat_template.jinja)
That works for a single initial developer message, but exposes Qwen’s ordering restriction:
developer → user
becomes: system → user ✓
system → developer → user
becomes: system → system → user ✗
The conversion preserves the instruction text and message position, but collapses the distinction between the two roles.
So the rationale is sound, but the compatibility handling is incomplete. Succintly: Qwen imposes the restriction, and DGPP’s role conversion can cause otherwise valid client requests to encounter it. Simply removing the conversion would replace this error with an unsupported-role error.
there is nothing better for the Qwen Flash Next. 60-70tok/s reasoning is absurdly fast for the sparks.
Thank you for the prompt reply. Within CC CLI got a workaround via LiteLLM + small python proxy script which implements reordering (full package attached, windows).
anthropic-fix.zip (3.5 KB)
@stephen.douglas.hawkins any thoughts on the short prompt admission. I was just doing some testing on tensorfold and it was admitting short requests in 2.0 - 2.5s. For anyone using an interactive Agent that also spawns sub agents or runs background tasks the delay to submit will be painful.
Or I could just have something jobberwhocked up :D
Hmm. I’d be happy to take a look at it if you’re able to package up your cluster logs (~/dgpp/log/deployment/<deploy_id>) and send them my way! One thought is that perhaps you have a large system prompt?
One thing that dgpp does deliberately is use full fp32 logits during prefill, which means that it lags a little bit behind other other engines when it comes to absolute peak prefill performance. This was a deliberate decision in order to try and achieve peak accuracy. That being said there are some knobs that you can turn to enable bf16 or even w4a4 during prefill. Either way, if you can send over your logs I can try to see if there is anything else suspicious going on. Assuming your system prompt isn’t too long then there may be something worth digging into!
I’ve been running DGPP for actual day-to-day coding work for a while now, so thought I’d share some real-world numbers instead of another benchmark.
My setup is 2x ASUS GX10, connected over the dedicated 200G fabric/RDMA, running RadixArk/Qwen3.8-Flash-Next-NVFP4 as DGPP W2. Beast4 is rank 0/head, Beast3 rank 1. I’m running 262K request context with a 524K KV pool and using ZCode as the coding agent.
This has basically become my daily coding setup now.
I’m usually working on one or two real projects at the same time. These aren’t test repos either. One is a fairly large existing full-stack application, the other is a Python reporting project where the agent is currently building a PySide6 desktop application around an existing reporting engine.
So the workload is lots of repo exploration, code edits, shell commands, tests, screenshots, regression checking, tool calls, etc. Pretty much the messy stuff you get during a normal day of development.
The biggest surprise for me has been how well it behaves when the ZCode sessions get really large.
One session reached about 216K / 262K context before ZCode did its own compaction. I don’t manually compress these sessions, I just let ZCode do its thing. After compacting it carried on working normally.
More importantly, at 200K+ it was still doing useful work. It was remembering project constraints, finding actual regressions, making targeted changes and running/understanding the tests. Obviously that’s not a scientific quality benchmark, but I was half expecting the quality to fall apart at that point and so far it really hasn’t.
The prefix cache is probably my favourite part of this setup.
One snapshot while I had two coding jobs running showed:
- 243 requests
- 29.5M prompt tokens
- 224K generated tokens
- 0 failed
- 0 shed
- 97.9% prefix cache hit
- 29.07M prefix tokens saved
- ~1.79s average TTFT on a cache hit
- ~56s average TTFT on a miss
- 78.3% MTP acceptance
- 3772 / 8192 KV blocks in use
ZCode itself was showing 98.8% average cache hit during one of the long sessions.
A later snapshot was already at roughly 397 requests, 44.5M prompt tokens, 367K output tokens and 485 tool calls, still with zero failed requests.
That’s where I stopped caring quite so much about maximum tok/s :)
For this kind of coding workload you’re sending the same enormous conversation/project history back over and over again with a relatively small new part added each turn. Reusing almost all of that prefix makes a massive difference to how responsive the whole thing feels.
I’m also regularly running two ZCode projects at the same time now. Both GX10s get worked pretty hard, but so far DGPP just keeps going. No queue building up in the snapshots I’ve checked and, more importantly, both agents remain perfectly usable.
Another thing I really like is startup time.
I timed our W2 warm start because the first time I saw it I thought I must have missed something: about 23 seconds from stopped to a usable Qwen endpoint.
My previous SGLang setup on these same machines could take around 9–12 minutes before I was back to work after a restart. That’s absolutely not meant as a scientific SGLang-vs-DGPP comparison — different setup, different configuration, etc. — but from a user’s point of view the difference is huge.
Before, if someone suggested restarting Qwen halfway through a job my reaction was basically “please no”. At 23 seconds I really don’t care anymore :)
Stability has probably impressed me even more. The old setup gave me enough crashes during long coding sessions that I became a bit nervous about very large contexts. With this W2 DGPP setup I’ve had both projects hammering away for hours and it has been boringly stable so far. That’s exactly what I want.
So for my particular use case I think my priorities have changed a bit.
Raw generation speed is nice, obviously, but for an actual coding agent I’d now put prefix caching, TTFT, long-context behaviour and stability ahead of squeezing out another few tok/s.
Basically my test has become:
Can I throw two real software projects at it, let the agents work for hours, allow a session to grow past 200K context, and forget that the inference server is even there?
So far… pretty much yes.
Anyway, thought these numbers might be useful because this is actual daily usage rather than a benchmark run. If there are any particular DGPP metrics/config details that would be useful to see, I’m happy to grab them from the running setup.
That’s awesome! Thanks for the feedback, and I’m glad your experience has been positive!
Like you, I had been using vLLM prior, and eventually got fed up with trying to tinker with recipes that would fail to start after waiting 10 minutes, so one of my number one goals was to try to figure out a way to write an engine that can load the model and reach a ready state as fast as possible. A lot of the time, I’m switching between different models for different use cases, or I’m iterating on a feature that requires running lots of end-to-end CI tests against the engine itself, and that sort of wait time would have been soul-crushing.
Do feel free to drop any feature requests you have - here or on the github and I’ll be glad to add them to the queue!
I do have a decent sized prompt as the agent is a specialized digital twin - but that should slowdown equally across inference engines. Let me poke and see if resizing that alleviates the problem. Is there something in DGPP that make it more problematic so slot in additional calls based on prompt size vs vLLM or TensorFold as both of them have no issues with the background actions getting substantially delayed.
Truthfully I’m just guessing. If you wanna DM me your cluster config and logs I can take a closer look. A longer system prompt shouldn’t have that much of an effect. It is more likely coming down to the concurrency settings in your configuration along with your usage patterns.
Either way, the more real feedback and user experience I can learn from, the better I’m able to make dgpp!
Just did my first deploy with DGPP - the startup speed is simply amazing! Great job @stephen.douglas.hawkins !
What I couldn’t figure out is whether there is an option to set the reported model name (like vllm --served-model-name option). I have so many apps accessing the LLM and changing model names everywhere is a little annoying, so I’m using the same name across different models.
I’ve been using it for a few days, and I can say that I see a lot of potential.
GLM 5.3 loads in 34 seconds, with three nodes fetching the model over NFS from the 4th node. Nothing beats it.
One thing that I noticed is since the engine resizes images, the coordinates that models return when asked, are for the resized images. This behaviour is different from other engines. It would also be useful to have a knob for the image quality.
The biggest issue for me is the cold prefill speed. With GLM 5.3 Flash, FP8, after 200k prefill gets quite slow.
A nice bonus would be audio support for MiMo.
With all said I’m happy that this engine exists.
Hi, I tried using your project, and I have to say - it’s impressive!
However, I ran into an issue: if I have `deploy/cluster_qwen-3.8-flash-next_nvfp4_w2.example.json` running in `c2` mode with a large context, and then I launch one or two more agents with a small context, the master node shuts down. I’ve tried this several times, and the master node specifically has shut down on each occasion.
I also wanted to see GLM 5.3 boot up in half a minute.
Same recipe every run (TP2, 480K context, 2 slots, MTP). The OS page cache was dropped before each run.
Almost there:
| Scenario | Spark02 checkpoint source | Resident weight cache | OS page cache | What each rank does | Time to ready |
|---|---|---|---|---|---|
| Warm | NFS | present on both nodes | dropped | reads its 89.4 GiB cache from local NVMe; checkpoint shards not read | 46.8 s |
| Warm | local NVMe | present on both nodes | dropped | same | 46.9 s |
| Cold | local NVMe | deleted on both nodes | dropped | reads the full 181.7 GiB checkpoint, builds a new cache | 208.9 s |
| Cold | NFS | deleted on both nodes | dropped | rank 0 builds in 188 s; rank 1 reads the checkpoint over NFS in about 330 s (about 564 MiB/s) | 348.5 s |
It’s a huge jump: from vLLM 6-8 min to less than a minute!
Thanks for you work! Testing the stack currently.
NOTE: I had to patch the following hard-coded values:
BusBoundaryReducer/consumer_deadline_s: 60s → 600s (DGPP_BOUNDARY_TIMEOUT_MS/DGPP_CONSUMER_DEADLINE_S)completion_timeout_ms: 120s → 600s (DGPP_BUS_COMPLETION_TIMEOUT_MS)
I was mistaken; this isn’t an issue specific to dgpp. The same situation occurs on velogb10.
In case it helps anyone: Spark shuts down due to a frequency spike; you simply need to lower the clock speed slightly by running:
sudo nvidia-smi -lgc 300,2200
or
sudo nvidia-smi -lgc 300,2300
DGPP project update — September 26–October 3, 2026
Hey folks!
This week brings DeepSeek-V4-Flash support, a new single-Spark Qwen quantization, lower memory use, and faster processing of concurrent DeepSeek prompts. There are also improvements to conversation caching and fixes for multi-turn agent conversations.
Everything below has landed on master since September 26, inclusive, through 4198955.
New features
-
DeepSeek-V4-Flash on two or four Sparks. DGPP now serves
deepseek-ai/DeepSeek-V4-Flash-0731in its original MXFP4/FP8 format, with reasoning, tool calls, and adaptive speculative decoding. Both deployment templates are configured for the model’s 1M-token context limit. Model and serving details -
Qwen AutoRound int4/int8 on one Spark. You can now run
Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTNwith packed int4 experts, an int8 vocabulary head, and MTP depth 3. Setup handles the additional n-gram table dependency. Implementation and measurements -
Performance tuning through deployment files. New expert-processing, n-gram staging, and prefill precision options can be set in deployment JSON and distributed consistently across ranks. The AutoRound template enables two measured prefill speed/precision options; these can change numerical rounding and generated text. Configuration details
Here are the recorded performance numbers for the new model and quantization:
| Model / configuration | Sparks | Single-request decode, tokens/s | Aggregate throughput under load, tokens/s | Cold prefill: ~2K / ~8K / ~32K tokens |
|---|---|---|---|---|
| Qwen AutoRound int4/int8, shipped template | 1 | 45.9–71.5 | 78.0–116.3, four requests | 1.21 / 4.23 / 17.10 s |
| DeepSeek-V4-Flash MXFP4/FP8 | 2 | 40.4–72.8 | 67.5–101.1, four requests | 2.03 / 8.02 / 38.03 s |
| DeepSeek-V4-Flash MXFP4/FP8 | 4 | 65.4–125.9 | 134.0–186.0, six requests | 1.36 / 5.47 / 28.02 s |
These are recorded GB10/Spark-class measurements. Ranges span five prompt classes: prose, code, JSON, maths, and chat. Single-request rates measure engine decoding; aggregate rates include admission and prompt processing. The Qwen row uses the template’s BF16-partial and FP8-prefill options. Benchmark methods and results
Optimizations
-
DeepSeek-V4-Flash processes arriving prompts together. Concurrent prompts can share a forward pass, with decoding and cancellation between chunks. In recorded four-Spark tests, aggregate generation throughput increased from 58.8 to 88.9 tokens/s at two requests, and 56.5 to 120.6 at five. Change
-
Lower memory use for Qwen attention. Workspace now follows the per-request context limit rather than the entire shared cache. One measured configuration saved 4.49 GiB, reducing attention-scoring workspace from 6.49 to 2.00 GiB without reducing shared cache capacity. Thanks, @armapew! #65
-
Lower vocabulary-output memory use across model families. Serving retains the vocabulary rows needed for generation instead of a prompt-sized output matrix. The original single-Spark Qwen comparison reduced this allocation from 3.79 GiB to 7.58 MiB; the change also covers GLM, MiMo, and DeepSeek-V4.1. Thanks, @armapew, for the initial contribution! #67
-
More efficient Qwen attention during prompt processing. Attention scoring skips entries that the prompt cannot use, and selection work is bypassed when every available entry will be used. This reduces GPU work during prefill. Thanks, @armapew! #66
-
Better cache retention for long agent conversations. Short requests no longer create snapshots below the default 1,024-token threshold, reducing eviction of valuable conversation history. Additional snapshots preserve shared system prompts for new conversations and post-compaction turns. Several templates also allocate larger prefix caches. Details
-
Faster Qwen AutoRound prompt processing and generation. Packed expert kernels, reduced vocabulary-head reads, and overlapping n-gram loading reduce latency. The recorded optimization campaign reduced cold prefill from approximately 1.53 / 5.15 / 20.0 seconds to 1.29 / 4.42 / 17.82 seconds at 2K / 8K / 32K tokens, before enabling the template’s additional precision options. Measurements
-
Faster multi-node DeepSeek processing. Prompt computation overlaps inter-node communication, and decode avoids repeated compression and projection work. Six-request batches also retain batched execution when the full speculative block would exceed the decode-row capacity. Implementation
Bugfixes
-
A full token-cache pool no longer crashes serving when an optional snapshot cannot be taken. DGPP skips that snapshot and continues serving, without publishing an invalid cache entry. Thanks, Mark Griffith (@MARKYMARK55)! #75
-
Claude Code conversations through LiteLLM survive subsequent turns. Anthropic-style assistant thinking blocks are accepted and normalized into reasoning history, fixing the HTTP 400 failures they previously caused. Explicit
reasoning_contentretains precedence. Thanks, Mark Griffith! #76 -
More consistent GLM-5.3-Flash output when prompts are grouped. Cache-boundary and vocabulary-projection differences that made grouped short prompts generate different greedy text have been corrected. Change
-
Correct enforcement of context limits. Requests whose budgets exceed the model’s positional ceiling now receive a clear
context_length_exceedederror, even if the shared cache is larger. Speculative verification also stops staging rows beyond that ceiling. Admission fix, verification fix -
More reliable DeepSeek prefill and speculative decoding. Fixes cover short final prefill chunks, request isolation in grouped draft histories, and sampled fallback at reduced verification depths. These address invalid memory writes, reduced draft acceptance, and serving failures. #67, DeepSeek integration
-
--no-mtpworks with templates that configure speculative decoding. Disabling MTP now clears inherited draft-depth and scheduling settings unless explicitly supplied on the command line. Related startup checks catch unsupported batch configurations earlier. Change -
NVFP4 W4A4 prefill is now explicitly opt-in. Following the NVFP4 prefill optimizations covered in the previous update, BF16 activations are again the default. Calibration scales alone no longer enable activation quantization. The W4A4 path remains available through
DGPP_MOE_W4A4=1while the remaining model-quality checks are completed. Change -
Fresh checkouts configure successfully without local benchmark files. Missing optional microbenchmark sources no longer block production builds. Change
-
Accurate decode timing in retirement logs. Prompt-processing time is no longer subtracted twice from reported decode time. Change
And CPU as well,
My settings for CPU
ENABLE=“true”
GOVERNOR=“ondemand”
MAX_SPEED=“2500000”
Gpu on worker nodes can be set to 2400 or 2500 while head node 2300. So that they may have same temp .
Thank you for this project. It took me a bit to get it running, but I got there with Qwen3.8 FN on a single Spark. One “issue” I have is memory usage of my Spark is only 80%. I’m not sure if this will grow with usage, but I’d like to lock in the memory if I can. I did this with VLLM to hold space for KV cache (before shifting to this project). Is there a way to do this with DGPP?
It also took me some time to figure out how to set the default thinking mode to “medium”. With the help of my agent I made a wrapper script and added this environment variable:
KNOBS=‘–default-chat-template-kwargs {“reasoning_effort”:“medium”}’
I believe this did the trick.
it will grow with usage because the ple is on disk and you want as much of it in cache as possible.
Thanks, I figured as much. I did run into issue where I couldn’t run 2 concurrent queries. My agent says “engine.kv_capacity” at 256K was not enough (each had a fairly large running context). I’m increasing to 512K shortly.