DGPP: a GB10-optimized C++/CUDA inference engine

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Hi everyone,

Hi folks,

I have been building an inference engine called DGPP for NVIDIA DGX Spark. It is open source, written in C++ and CUDA, and focused entirely on getting strong real-world performance from GB10.

The project is here: github.com/HawkBearPig/dgpp

DGPP started with a simple question. What happens if the serving engine, CUDA kernels, memory plan, networking, and speculative decode path are all designed around Spark instead of inherited from a general-purpose GPU stack? Specifically, can we get faster single-stream decode speeds, while still supporting concurrency? Obviously concurrent throughput is very important, but for my use case, I’m typically operating with 1 - 4 concurrent sessions maximum, and I wanted to try to optimize for this workload specifically.

The answer so far has been encouraging. On several models, DGPP is running faster than the public vLLM and SGLang results I could find for the same DGX Spark hardware and model family. It also starts quickly, has an OpenAI-compatible API, and includes deployment templates for one, two, and four Spark systems.

Models currently supported

  • Qwen3.8-Flash-Next FP8 on two or four Sparks
  • Qwen3.8-Flash-Next NVFP4 on one or two Sparks
  • GLM-5.3-Flash FP8 on four Sparks
  • GLM-5.3-Flash NVFP4/FP8 on two or four Sparks
  • GLM-4.7 NVFP4 on four Sparks
  • Full GLM-5.3 754B int4/int8 on four Sparks
  • DeepSeek-V4.1-Flash MXFP4/FP8 on four Sparks

Here is the short version of what each quant changes. Activations stay BF16.

Checkpoint Dense and attention weights Expert weights Main exceptions
GLM-5.3-Flash FP8 KDA and kv_b BF16, other DSA and dense weights FP8 Routed and shared experts FP8 BF16 draft, embedding, and head. BF16 cache by default, with FP8 and FP4 options
GLM-5.3-Flash hybrid Same as FP8 Routed experts in layers 3–44 NVFP4, shared experts FP8 BF16 draft, embedding, and head. Four-Spark cache BF16, two-Spark cache FP8
Qwen3.8-Flash-Next FP8 GDN, QSA, and GR BF16 Routed experts FP8, shared experts BF16 FP8 n-gram table. Embedding, head, and KV cache BF16
Qwen3.8-Flash-Next NVFP4 Dense stack encoded to FP8 at load Backbone routed experts NVFP4, MTP experts FP8, shared experts FP8 at load BF16 embedding and KV cache. FP8 head and mapped n-gram table
GLM-4.7 NVFP4 Attention BF16, dense MLP layers 0–2 NVFP4 Routed and shared experts in layers 3–91 NVFP4 Draft experts converted to NVFP4 at load. Embedding, head, and KV cache BF16
Full GLM-5.3 int4/int8 Layers 0–2 BF16, layers 3–77 attention int8 group-64 Routed experts int4 group-64, shared experts int8 group-64 Draft experts converted at load. Embedding and head BF16. BF16, FP8, or FP4 cache
DeepSeek-V4.1-Flash Attention and dense weights FP8 Routed and draft experts MXFP4, shared experts FP8 BF16 embedding and head, FP8 Engram tables, model-native mixed cache

Current performance

These are warm decode results from the current production paths. The ranges come from five prompt classes, including chat, code, prose, JSON, and math. I prefer showing the spread instead of picking the fastest prompt.

Deployment One request Aggregate under load Cold prefill at roughly 2K / 8K / 32K
Qwen3.8-Flash-Next FP8, 4 Sparks 63.2–77.7 tok/s 142.1–167.3 tok/s at C4 Not re-run on the current path
Qwen3.8-Flash-Next FP8, 2 Sparks 41.7–49.6 tok/s 69.3–83.2 tok/s at C4 1.299 / 5.108 / 21.168 s
Qwen3.8-Flash-Next NVFP4, 2 Sparks, mapped n-gram 62.1–74.9 tok/s 119.0–136.9 tok/s at C4 1.241 / 4.870 / 20.286 s
Qwen3.8-Flash-Next NVFP4, 1 Spark 42.6–50.3 tok/s 69.4–83.7 tok/s at C4 Not re-run on the current path
GLM-5.3-Flash NVFP4/FP8, 4 Sparks 50.3–58.5 tok/s 94.5–104.2 tok/s at C4 2.210 / 11.374 / 93.765 s
GLM-5.3-Flash FP8, 4 Sparks 42.2–48.7 tok/s 62.2–67.5 tok/s at C4 2.211 / 9.165 / 58.555 s
GLM-4.7 NVFP4, 4 Sparks 29.5–33.3 tok/s 62.7–68.7 tok/s at C4 2.809 / 16.865 / not measured
Full GLM-5.3 754B, 4 Sparks 25.4–29.2 tok/s 42.3–47.0 tok/s at C4 6.111 / 40.838 / 350.069 s
DeepSeek-V4.1-Flash, 4 Sparks 49.64 aggregate tok/s at C1 108.49 aggregate tok/s at C6 1,383 prompt tok/s on a 2,950-token cold prompt

The DeepSeek prefill result uses a different prompt workload, so it should not be compared directly with the three target-length columns used by the other models.

The full methodology, quality results, prompt lengths, and reproduction commands are in docs/benchmarks.md in the repository.

How that compares with vLLM and SGLang on Spark

Model and hardware DGPP Published Spark result What stands out
Qwen3.8-Flash-Next NVFP4, 1 Spark 42.6–50.3 tok/s at C1 vLLM: 32.5 tok/s median, 21.7 tok/s prose, 43.8 tok/s peak Same source checkpoint. DGPP encodes the dense stack to FP8 at load. Its slowest class is close to the published peak
Qwen3.8-Flash-Next, 4 Sparks FP8: 63.2–77.7 tok/s at C1, 142.1–167.3 tok/s at C4 vLLM NVFP4: 40.5 median, 54.2 peak at C1, 262 tok/s at C6 in the same source above DGPP’s full C1 range is above the published vLLM median and peak. The loaded results use different concurrency
Qwen3.8-Flash-Next, 2 Sparks NVFP4: 62.1–74.9 tok/s at C1, 119.0–136.9 tok/s at C4 SGLang FP8: 36–41 tok/s at C1, 88–98.5 tok/s at C4 (source). vLLM NVFP4: 53.7 median, 63.7 peak at C1, 309 tok/s at C6 in the Qwen source above DGPP is above the published vLLM median across all five C1 classes and above the SGLang result at both C1 and C4. vLLM reports the higher C6 result
GLM-5.3-Flash, 4 Sparks FP8: 42.2–48.7 tok/s at C1. Hybrid: 50.3–58.5 tok/s at C1 vLLM W4A16: 43–58 tok/s at C1 across repeated passes, with 43.0–47.3 tok/s test means and 52.2–59.7 peaks (source) DGPP’s hybrid sits in the upper part of the latest published vLLM band
GLM-4.7 NVFP4, 4 Sparks 29.5–33.3 tok/s at C1, 62.7–68.7 tok/s at C4 SGLang: 24.4 tok/s at C1, 53.2–54.6 tok/s at C4 (forum topic 366325) DGPP is about 21–36% higher at C1 and 15–29% higher at C4

I rechecked these forum results on September 16, 2026. These are separate community runs, not a controlled shootout. Prompt sets, output lengths, checkpoints, quantization, context capacity, KV format, speculative settings, concurrency, and timing scope differ. The vLLM GLM-5.3-Flash result uses a W4A16 checkpoint, a DFlash2 drafter, and a 1M context configuration. The four-Spark vLLM Qwen result uses NVFP4 while DGPP uses FP8. The two-Spark DGPP and vLLM Qwen results both use the NVIDIA NVFP4 checkpoint, but their clients and serving settings differ. The loaded comparison uses C4 for DGPP and C6 for vLLM. The SGLang row uses the larger FP8 checkpoint.

Where the speed comes from

DGPP does not wrap an existing Python serving backend. The scheduler, tokenizer, HTTP server, prefix cache, sampling path, collectives, model loaders, and hot inference path are native C++ and CUDA.

The CUDA work is specific to these models and to GB10. Qwen has a tiled QSA prefill kernel that shares K/V tiles across query heads. Full GLM-5.3 has a packed int4/int8 tensor-core prefill path. DeepSeek has dedicated CSA2, Engram, and DSpark execution. Dense kernels choose their lowering from the active row count instead of forcing every request shape through the same path.

Decode and speculative verification run through captured CUDA graphs. The graphs include the tensor-parallel collectives and batched request rows. DeepSeek can adjust its verification depth from the draft confidence, which avoids paying for draft rows that are unlikely to survive.

For multi-node inference, DGPP talks to libibverbs directly and stripes bulk traffic over the two active RoCE interfaces exposed by the ConnectX-7 topology. The measured aggregate link rate is about 196 Gb/s.

Startup is another area I cared about. DGPP stores a per-rank resident weight image after the first load. Once that image exists, a deployment typically becomes ready in 15 to 30 seconds, depending on the model. The first launch is slower because it builds the image.

Serving and correctness

The server supports OpenAI-style Chat Completions and legacy Completions, including streaming, tool calls, JSON-schema output, reasoning content, logprobs, prefix caching, health checks, and metrics.

Performance changes go through task-level checks. Recent configurations score 38–40/40 on HumanEval, 59–60/60 on GSM8K, and 30/30 on schema extraction. The larger GLM-5.3-Flash campaign uses 164 HumanEval, 300 GSM8K, and 100 extraction items.

If you have one, two, or four Sparks and would like to try it, the getting-started guide covers setup, RoCE discovery, checkpoint download, deployment, and testing.

I’d especially appreciate independent results, bug reports, and comparisons. If anyone else is working on GB10-specific kernels or multi-Spark inference, I’d also be interested in comparing notes. Thanks everyone for taking the time to take a look! Happy to answer any questions, and feel free to raise any bugs here or file an issue on the github!

2

42.6–50.3 tokens/s on a single Spark with the Qwen3.8-Flash-Next NVFP4 model—I’ll definitely give it a try and share my impressions. Thanks!

3

Can you publish your performance results for GLM5.3-Flash on 2 GB10s ?

4

image
image3045×1201 114 KB

Hi! I’m testing DGPP on a single ASUS GX10 with nvidia/Qwen3.8-Flash-Next-NVFP4. Performance has been very impressive so far, especially with MTP2.

I found a few OpenAI API / Hermes compatibility issues that may be worth looking into.

1. Numeric constraints in JSON Schema

A valid schema like:

{"type":"number","minimum":0,"maximum":10}

is rejected by the server before inference with HTTP 400:

a bound is enforced on integers only; this value admits a fraction

minimum / maximum should work for JSON Schema number as well, not only for integer.

2. tools + response_format

DGPP currently rejects requests that use tools together with response_format: json_schema/json_object:

response_format json_object / json_schema cannot be combined with tools
(a turn is either JSON or tool calls)

Because of this, a typical agent workflow breaks:

model -> tool call -> tool result -> final JSON matching the schema

The tool call itself works correctly, but the final structured-output turn gets HTTP 400.

3. Qwen reasoning level

In Hermes Desktop / tool benchmarks, reasoning_effort: "xhigh" does not seem to be translated into Qwen’s reasoning settings.

It would be useful if DGPP’s OpenAI-compatible API could pass through or map reasoning levels, for example:

low    -> low
medium -> medium
high/xhigh -> xhigh

so that reasoning_effort=xhigh from Hermes or from a benchmark actually changes Qwen’s reasoning mode.

I would also strongly recommend running SeraphimSerapis/tool-eval-bench on DGPP. The benchmark is not perfect, but many DGX Spark runtime/model authors use it as one of the first agent/tool-calling tests. For forum users, it gives at least a useful baseline understanding of the engine and model quality.

5

One more small compatibility issue: Prometheus metrics endpoint

DGPP currently exposes metrics at:

/v1/metrics

while the conventional/default Prometheus endpoint is:

/metrics

vLLM, TGI and llama.cpp also expose their Prometheus metrics at /metrics. Prometheus itself defaults metrics_path to /metrics.

/v1/metrics is technically valid, but /metrics would improve drop-in compatibility with existing monitoring tools and vLLM setups. Maybe keep /v1/metrics for backward compatibility and simply add /metrics as an alias.

It would also be consistent with DGPP already exposing /health outside the /v1 API namespace.

6

Hey Siertum, thanks for the feedback! I’ll take a look at the items you raised and push some fixes shortly. Regarding tool-eval-bench - I do actually run this locally. Happy to incorporate it into my testing suite formally if that adds value.

7

This appears to be a Qwen or Nvidia Qwen 3.8 Quant issue as I had the same issue with Nvidia’s QWEN 3.8 Next Flash NVFP4 on a different engine.

8

Great project, thanks for sharing. I’ve been giving dgpp a test drive.

I needed a bit higher concurrency on Qwen 3.8 Flash Next, so i’ve bumped up the vocab-head to 64 rows. I wouldn’t say it was exactly PR ready yet but happy to share back.

9

Did some testing works great love the start speed (qwen 3.8 nvidia single spark using your defaults on git for it) but the vision seems broken in it. everything else worked great and fast.

10

Triplany - Vision is currently not supported, but should be implemented soon. I’m planning to implement this in the next few days.

jontaylor - Understood, I’ll take a look. Thanks!

siertum - Added the suggested metrics endpoint aliases.

11

Hey folks! Pushed another update that addressed some issues/new features:

  • Added GLM-5.3-Flash image inputs, including streaming/MTP support and bitwise numerical parity with the tested reference.
  • Added file uploads, PDF text extraction, and text/code document inputs.
  • Expanded custom tools, regex/Lark grammars, and JSON Schema constraints.
  • Improved OpenAI/OpenCode compatibility, reasoning controls, structured outputs, streaming, and token accounting.
  • Added live prefill progress, a /metrics alias, and configurable HTTP request-size limits.
  • Optimized DSA prefill tile sizing for the current context.
  • Separated release/testing builds, fixed PCRE2 packaging, and expanded validation—107 tests passed, plus live two-node checks.
12

Wow, this thing is fast! I think this is the future of inference engines on Spark.

(Honestly Nvidia should’ve shipped this with Sparks if they were a serious company.)

Wish list:

  • Setup is quite cumbersome, lots of manual steps. I think much of it can be streamlined into a script.
  • Why is default_max_tokens set to 256 everywhere? I thought the whole thing was broken because it always stopped after 1 paragraph. It’s not hard to change, but it’s not mentioned in the setup guide that it needs to be changed. I’d suggest setting the default to something more reasonable, or auto set based on cache size, or something.
  • Vision 🙏️

Also, the name sounds cool, but what does it stand for? DGx.cPP? DGx Prompt Processor? 🙂️

13

Thanks for trying it out, and appreciate the feedback! I agree, the setup could be made a lot simpler. Perhaps a single bootstrap or setup script that covers all of the necessary dependencies. Also, I agree that default_max_tokens set to 256 is not ideal…I’ll have a chat with my agent about that one for sure!

As for the name, I was initially thinking of dgx.cpp, which probably would have been more accurate, but I was worried it was a little too close to llama.cpp, and not original enough. I went with dgpp because it was shorter, but tbh I don’t really know what it stands for. I like DG(x) Prompt Processor though!

14

This is great! I have been working on something very similar (inferencing engine hyperoptimized for Sparks) and wanted to publish as spark.cpp, but just realised this exists! I have got a bunch of speedups you don’t seem to have in place - will PR for sure!

15

Hey guys, just pushed an update with a pretty exciting (at least I think so) optimization!

Lossless bf12

Decode on the Spark is a memory-bandwidth problem. Every generated token streams the active weights out of LPDDR5X, and dgpp’s decode kernels already read them at essentially the memory’s line rate (~250 GB/s). Without quantizing further, the only lever left is to move fewer bytes. In the checkpoints we serve, a large share of what a decode step reads is still native BF16. The linear-attention projections and LM head are over half the bytes of a GLM-5.3-Flash step. The attention projections are about two thirds of GLM-4.7’s. On Qwen3.8-Flash-Next-FP8, roughly 80% of what each Spark reads per token is BF16.

A BF16 value is 1 sign bit, 8 exponent bits and 7 mantissa bits. The sign and mantissa of a trained weight are effectively random and incompressible, but the exponent is not. It carries only ~2.6 bits of entropy, and within any one row of a matrix 99.98% of the weights fall inside a window of 15 consecutive exponents. bf12 exploits exactly that. Each weight is stored in 12 bits instead of 16: one byte holding the sign and mantissa verbatim, plus a 4-bit code giving the exponent as an offset from a per-row base. The sixteenth code is an escape. The rare weight outside its row’s window (2–7 in 10,000 on real checkpoints) keeps its full 16-bit value in a small per-row side table. The few dozen rows with heavy outliers are simply left as BF16. Rows of any width that is a multiple of eight pack at twelve bits with no padding.

It is lossless because nothing is rounded or approximated. It is a re-encoding, not a quantization. Every weight’s original 16 bits are recoverable exactly: sign and mantissa are copied, the exponent is base + code, and anything that doesn’t fit is stored verbatim. It is also bit-exact at the output, which is the stronger claim. The bf12 kernel rebuilds the exact BF16 bits in GPU registers, then performs the same multiply-accumulates in the same order, with the same reduction tree, as our BF16 kernel. The results are identical bit for bit, not merely “within tolerance”. We test that directly:

  • The unit tests compare outputs bitwise at every batch width and row shape, including adversarial matrices full of zeros, subnormals, Inf/NaN and outlier rows.
  • On the cluster, greedy transcripts
    served with bf12 on and off are byte-identical on GLM-5.3-Flash, GLM-4.7, the full GLM-5.3 and Qwen3.8-Flash-Next. The payoff is 25% fewer bytes per step for those matrices.

Single-stream decode gains:

  • GLM-5.3-Flash: +6–7%
  • GLM-4.7: +11–12%
  • full GLM-5.3: +5–6.5%
  • Qwen3.8-Flash-Next-FP8: +6.4% on four Sparks and +9.0% on two

The gain tapers as concurrency rises and the kernels become less bandwidth-bound, to +1–6% at four concurrent requests. It only pays where weights are actually read from DRAM. Weights that are already in cache by the time they are used gain nothing from being smaller and pay for the unpacking. On Qwen that is about a third of its BF16 bytes (the hyper-connection GR weights), so we deliberately leave those as BF16. It is one setting, engine.bf16_weights. bf12+bf16 keeps both forms in memory, so prefill is untouched. bf12 keeps only the 12-bit form: the BF16 bytes are handed back to the system as each layer loads, so the model takes less memory than the original checkpoint. That is what let us restore 160K context on a two-Spark GLM-5.3-Flash and 120K on the full GLM-5.3. In that mode prefill expands the rows it needs back to their exact BF16 bits on the fly. That costs about 1–4% on prompts of 2K tokens and up, and a few tens of milliseconds on a short prompt’s time to first token. Either way the numerics are untouched: same logits, same tokens, fewer bytes.

Model (DGX Spark cluster, MTP depth 1, greedy) Metric BF16 weights (before) bf12+bf16 bf12 alone
GLM-5.3-Flash NVFP4/FP8 (4 Sparks) Decode, 1 stream (tok/s) 55.1 58.9 (+6.9%) same as bf12+bf16
Decode, 4 concurrent (tok/s, aggregate) 105.0 111.8 (+6.4%) same as bf12+bf16
Prefill ~2K / ~8K / ~32K tokens (s) 1.28 / 5.29 / 24.6 unchanged 1.33 / 5.36 / 24.9 (+3.5% / +1.3% / +1.2%)
Memory per node (GiB) 93.9 95.9 93.4
GLM-4.7 NVFP4 (4 Sparks) Decode, 1 stream (tok/s) 32.1 35.8 (+11.4%) same as bf12+bf16
Decode, 4 concurrent (tok/s, aggregate) 68.3 72.0 (+5.4%) same as bf12+bf16
Prefill ~2K / ~8K / ~30K tokens (s) 2.63 / 13.9 / 143.9 unchanged 2.73 / 14.2 / 144.6 (+3.6% / +1.7% / +0.4%)
Memory per node (GiB) 79.2 84.1 77.8
GLM-5.3 (full, int4/int8) (4 Sparks) Decode, 1 stream (tok/s) 28.6 30.2 (+5.9%) same as bf12+bf16
Decode, 4 concurrent (tok/s, aggregate) 46.0 48.2 (+4.9%) same as bf12+bf16
Prefill ~2K / ~8K / ~30K tokens (s) 4.55 / 20.2 / 138.7 unchanged 4.59 / 20.1 / 138.0 (within 1%)
Memory per node at 120K context (GiB) 110.4 111.7 (does not fit; 100K max) 110.1
Qwen3.8-Flash-Next-FP8 (4 Sparks) Decode, 1 stream (tok/s) 73.3 78.0 (+6.4%) n/a (Qwen always keeps both forms)
Decode, 4 concurrent (tok/s, aggregate) 162.2 164.3 (+1.3%) n/a
Prefill ~2K / ~8K / ~32K tokens (s) 1.04 / 3.98 / 16.4 1.04 / 3.95 / 16.3 (unchanged) n/a
Memory per node (GiB) 54.8 56.1 n/a
Qwen3.8-Flash-Next-FP8 (2 Sparks) Decode, 1 stream (tok/s) 47.1 51.4 (+9.0%) n/a
Decode, 4 concurrent (tok/s, aggregate) 96.2 100.6 (+4.6%) n/a
Memory per node (GiB) 98.4 100.8 n/a
16

wo..w nice to see! I am more interested in glm5.3 flash but it’s awesome to see quite fast. keep it up! I will definately try it out.

17

I’ve dropped some PR’s in around cross-build, decode metrics, MTP metrics, FP8 vocabulary head, C16/MTP3, and sparse compaction.

I’ve primarily focussed on optimising for higher concurrency agentic workloads, so the changes are mostly around expanding out concurrency while retaining MTP3 and protecting against some pathological cases where there can be a real mismatch between active rows and batch size.

There are similar warnings for this as prefix chunking, in that reordering rows eventually leads to different choices which result in a divergence vs no compaction of rows.

After these changes my workloads are up 20% on generation rate, while reducing GPU power draw by circa 15%.

18

any plan to support DSV4 for poor man 2 DGX setup? Also opened bunch of PRs around Qwen flash Yarn support. I am keep going back DSV4 and Qwen as of now, Need 1M context for my work, I do have native DSV4 kernel in working but I am not able to crack the speed it works but is very slow compare to open recipes out there.

19

any plan to support DSV4 for poor man 2 DGX setup? Also opened bunch of PRs around Qwen flash Yarn support. I am keep going back DSV4 and Qwen as of now, Need 1M context for my work, I do have native DSV4 kernel in working but I am not able to crack the speed it works but is very slow compare to open recipes out there.

Saw the PRs. Thanks for the contributions! I’ll add support for DSV4 flash in the coming days. I recognize that it’s definitely still a popular choice. Also going to add support for qwen3.8 27b as well.

20

Hey folks. Just wanted to go over some recent changes that landed. A lot of these cover general usability, bug fixes, and even some issues and PRs raised by some of you (thanks to everyone who decided to contribute!).

  • Faster generation with long Qwen contexts. Optimized attention selection reduces the slowdown as conversations and documents grow. On two Sparks running Qwen NVFP4, measured decode-step latency fell 47% at roughly 260K tokens and 60% at roughly 521K tokens, with identical responses in the comparison.

  • Much faster follow-up questions about the same document. Prefix caching now retains an earlier reusable point, allowing a changed question near the end of a long prompt to reuse most of the document’s processing. In the two-Spark Qwen test, time to first token for a changed question over a 260K-token document dropped from 215.5 seconds to 4.47 seconds. Identical repeated prompts retain their deeper cache reuse.

  • Clearer prefix-cache sizing and memory reporting. Startup diagnostics and --memory-plan now show snapshot size, available cache slots, actual allocation, and the separate KV capacity. A new sizing guide explains which setting to increase for a given workload; existing recipe budgets remain unchanged because the tested document-sharing workloads fit. Single-node Qwen planning also now includes speculative-decoding snapshot state.

  • Optional 512K context for Qwen3.8-Flash-Next. YaRN configuration extends the context ceiling to 524,288 tokens, with a supplied two-Spark NVFP4 deployment recipe. Users can work with larger documents by explicitly enabling it and providing sufficient KV capacity; existing configurations retain their previous positional encoding.

  • More reliable speculative decoding under concurrent load. Fixed a bug where one request’s sampling fallback could overwrite draft information still needed by another request in the same batch. Concurrent sampled requests now use the correct verified drafts, preventing this source of incorrect token selection.

  • Tool calls respect declared argument names by default. Tools with declared properties now constrain generated top-level arguments to those names, reducing unexpected parameters passed to applications. Tools intentionally accepting additional names can enable them with additionalProperties: true.

  • Safer handling of duplicate tool arguments. GLM and Qwen tool-call parsers now reject repeated argument names within a call. Malformed calls are preserved as ordinary response content, avoiding ambiguous structured calls where different clients might silently choose different argument values.

  • Guided setup from a fresh checkout. scripts/setup.sh walks through deployment selection, node addresses, networking, prerequisite checks, building, checkpoint preparation, and serving preflight. Repeat runs preserve local settings, and unattended and read-only check modes simplify automation. Passing --start also starts the service.

  • Build Spark binaries on an x86 Linux workstation. A Docker-based cross-compilation workflow produces ARM64/GB10 server binaries without requiring a GPU or NVIDIA driver on the build machine. Developers can move compilation off their Sparks and use the target hardware for execution and testing.

  • Visibility into batching efficiency. The JSON metrics endpoints now expose decode batch capacities, active requests, processed rows, and padding. Operators can see how effectively batches are filled and make better-informed concurrency adjustments.

  • Visibility into speculative-decoding acceptance. Metrics now report attempted and accepted draft tokens, including acceptance at each speculative position and decisions resolved through sampling fallback. This helps operators assess whether their configured speculation depth provides useful work.

  • More accurate decode timing. GPU completion timing is now recorded before host-side metrics bookkeeping, preventing that bookkeeping from inflating measured decode time. This makes performance comparisons and monitoring more trustworthy.

  • More reliable load benchmarks. Timed benchmarks now reconcile published counters with completed requests before evaluating results. Delayed metrics updates no longer produce false “unexpected traffic” failures; reconciliation uses observed completion conditions and a deadline.

  • Resumable, more accurate long-context validation. The YaRN release checker now saves progress and supports --resume, corrects decode timing units, accommodates reasoning and queued long requests, and improves cache-reuse checks and Spark memory reporting. Expensive validation runs can resume completed work and produce more useful results. Invalid YaRN correction settings also receive
    stricter validation.

  • Fewer misleading cluster startup warnings. Nodes now compare effective automatic batch settings, eliminating false configuration-override warnings when their settings agree. Genuine differences still produce warnings, making startup logs easier to act on.

  • Updated benchmarks and deployment documentation. The README replaces previously pending performance entries with measurements across the shipped GLM and Qwen FP8 recipes. Documentation also clarifies FP8 Flash memory requirements and covers the new setup, cross-compilation, long-context, monitoring, and caching features, helping users choose and configure deployments.

Pending / In-Progress:

  • DeepSeek V4 flash support. We already support v4.1, but the smaller v4 is still popular.
  • Qwen3.8-27b support.
  • Vision support for Qwen3.8 Flash Next and DeepSeek v4.1 flash. Vision is already supported for GLM-5.3-Flash.