Fitting Qwen3.8-Flash-Next (180B) onto one DGX Spark — ~44 tok/s at four bits

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

The published FP8 checkpoint is 172.78 GiB. A DGX Spark has 128 GB of unified memory, and vLLM’s own recipe for this model starts at four GB300s. Four bits is the only thing that fits on one box, so I measured what it costs.

SingleSpark-Qwen3.8-Flash-Next

Single request. 8,192-token context, eager, speculative depth 3. Three prompt classes, one warm-up each, three retained runs, reasoning tokens counted, prefill excluded by timing from the first streamed token.

prompt tok/s tokens/step target step
question 46.43 2.514 54.15 ms
code 43.81 2.380 54.32 ms
prose 43.17 2.386 55.29 ms
median 43.81 — 54.32 ms

Same image with the FP8 dense conversion switched off: 34.36 tok/s, 69.34 ms step — the step is 22% shorter.

No concurrency numbers. Everything here is C1. If you came from my 27B post looking for a 16-concurrent table, there isn’t one — it has not been measured on this model.

Quality. Paired teacher-forced NLL against the BF16-dense control over 23,988 frozen positions: +0.003277 nat/token against a +0.005 threshold declared before the run. Greedy repeats identical 9 of 9. Not task accuracy, and no agentic measurement — which is exactly where four bits is most likely to hurt.

How. NVFP4 base, 309 dense projections re-serialized as blockwise FP8, the 248,320-row output projection at eight bits, and a product-quantized draft head — 16.9 MiB of codebooks replacing a 606 MiB vocabulary scan — driving the model’s own MTP layer. BF16 Gated DeltaNet recurrent state, idea from azampatti/GB10-3.8-Flash-Next.

git clone https://github.com/albond/SingleSpark-Qwen3.8-Flash-Next.git
cd SingleSpark-Qwen3.8-Flash-Next
./install.sh
SNAPSHOT=/path/printed/by/install.sh ./configs/launch-release.sh

Needs 210 GB free under $HOME (101 GiB checkpoint), driver 580.173.02 / CUDA 13.0, Docker with the NVIDIA Container Toolkit.

Not claimed: concurrency above one, context beyond 8,192 of the model’s 262,144, multimodal, general task accuracy. Techniques that were measured and lost are in the tree, default off. Contradicting measurements welcome.

2

Hello Albond, and thanks for this one. But it seems there is something wrong: it doesn’t finish the answer and breaks in the middle of nowhere, without any error in the container logs. No problem, with small answer it seems only larger answers are impacted (not sure).

Here is a dummy prompt to reproduce the problem:
“Act as a senior software architect.
Create a detailed high-level design document for the following system. Structure the document with clear headings, For each section, write multiple paragraphs with enough detail so that an implementation team can start working from this document without further clarification. Create a proxy system multisite HA, and should be working in multiple zones: lan and DMZ.”

I’m using Open WebUI for the GUI, and it’s working as usual with other APIs.

Here are some screenshots with :

image
image931×615 65.4 KB

image
image995×377 28.9 KB

3

This is not a crash, and that is why the container logs are silent. The shipped profile serves with --max-model-len 8192, so the request ends the moment prompt + reasoning + answer together reach 8,192 tokens. To the server that is a normal completion, not an error: the response comes back with finish_reason: "length" and there is nothing to log. Short answers never reach the cap, long ones do — which matches exactly what you are seeing.

One detail worth knowing: this profile runs with --reasoning-parser qwen3, so the model’s thinking block is spent from the same 8,192 tokens before a single word of the visible document is written. On a “detailed high-level design document” prompt that overhead is substantial, so the usable answer is meaningfully shorter than 8k.

You can confirm it in one call — Open WebUI does not surface the field, but the API does:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"singlespark-qwen3.8-flash-next","messages":[{"role":"user","content":"<your prompt>"}]}' \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["choices"][0]["finish_reason"], d["usage"])'

length confirms the cap; stop means something else is truncating and I would want to look further.

To raise it, the context is a launch knob, not a rebuild. Stop the container and start it with a larger CTX:

docker rm -f singlespark
SNAPSHOT="$HYBRID" CTX=32768 BIND=0.0.0.0 DETACH=1 ./configs/launch-release.sh

The banner the script prints will show the context it actually started with. Memory should not be the obstacle: at this profile the engine allocated a 190,873-token KV cache, so a single stream at 8k was using a small fraction of it.

The honest caveat, and the reason 8,192 is the default rather than an oversight: every number published for this server was measured at 8,192 tokens, single stream. Longer contexts are listed under Known limits as not measured — the model’s own limit is 262,144, but I have no data on what decode speed or output quality do between there and here. Raising CTX is supported and expected; the published figures simply stop applying, and if you do run it at 32k I would be glad to hear what you observe.

Also worth ruling out on your side: Open WebUI has a per-model max_tokens setting in its advanced parameters. If that is set, it caps the answer independently of the server and produces the same silent truncation.

Note: I think, I will release soon (EOD) updates where we can on start select context. But we have no a lot of memery free, so it will be for personal risk.

4

Safe to update context to 64k.

5

Thanks, indeed I can confirm the behavior:

length {‘prompt_tokens’: 124, ‘total_tokens’: 8192, ‘completion_tokens’: 8068, ‘prompt_tokens_details’: None, ‘completion_tokens_details’: {‘reasoning_tokens’: 1944}}

Is there a way to compute the maximum CTX size based on the number of concurrent sessions, or is it simply a matter of applying a value and monitoring memory usage?

Many thanks for the detailed answers. I will switch to 64k.

6
  • The engine computes it for you. Set CTX, then read its startup line: GPU KV cache size: N tokens, Maximum concurrency for CTX tokens per request: X.XXx — that’s how many concurrent sessions of that size fit.
  • It refuses to start if even one request won’t fit, so it’s a startup check, not applying-and-monitoring.
  • Don’t divide the max context by your window — that overstates concurrency, because each concurrent sequence needs its own recurrent-state block on top of its attention pages.

Note: 1.2 released with default new context size 65,536!

7

SingleSpark 1.2 is out — 64k context by default.

The default window moves from 8,192 to 65,536, so you don’t need to set anything. Eight times the context costs 1.7% of the target step and 1.1% of the rate: 43.34 tok/s median, 55.27 ms step. All three measured windows are published side by side in the README.

git pull — no rebuild, the image, checkpoint and draft artifact are byte-identical to 1.1. CTX=8192 restores the old window.

v1.2

8

Thanks,

I replaced the --rm argument with -d, so I can now set the restart policy to unless-stopped.
On your side, how do you handle container persistence when rebooting the DGX Spark? (Since almost every day brings a new Spark update.)

Because when I stop the container, it still gets removed even with DETACH=0

9

Thank you.

My test shows:

                                                                                                    Throughput Results                                                                                                     
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test                                                          ┃                      pp t/s ┃                      tg t/s ┃                   TTFT (ms) ┃                  Total (ms) ┃                          Tokens ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ pp2048 tg128 @ d0                                             │                     212,715 │                        24.9 │                          10 │                       5,117 │                        2115+128 │
│ pp2048 tg256 @ d0  c2                                         │                     232,549 │                        41.9 │                           9 │                       6,107 │                        2115+256 │
│ pp2048 tg128 @ d4096                                          │                     448,128 │                        21.8 │                          14 │                       5,844 │                        6194+128 │
│ pp2048 tg256 @ d4096  c2                                      │                     358,581 │                        21.5 │                          18 │                      11,893 │                        6194+256 │
│ pp2048 tg128 @ d8192                                          │                     565,827 │                        14.5 │                          18 │                       8,793 │                       10289+128 │
│ pp2048 tg256 @ d8192  c2                                      │                     420,868 │                        16.7 │                          27 │                      15,335 │                       10289+256 │
└───────────────────────────────────────────────────────────────┴─────────────────────────────┴─────────────────────────────┴─────────────────────────────┴─────────────────────────────┴─────────────────────────────────┘
╭───────────────────────────────────────────────────────────────────────────────────────────────── 🏆 Benchmark Complete ─────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                                         │
│    Model:  /root/.cache/huggingface/hub/models--starkweatherdigital--qwen3.8-flash-next-nvfp4/snapshots/1b304e5f99de0faaf43c3a959f2b4000294bf65c-ssfp8-gdn-mtp-qsa-shared-21dc2a891f8e                                  │
│    Score:  84 / 100                                                                                                                                                                                                     │
│    Rating: ★★★★ Good                                                                                                                                                                                                    │
│    Benchmark: tool-eval-bench v2.5.1.dev11+g95e2b5021                                                                                                                                                                   │
│    Engine:       vLLM 0.1.dev20073+g8e685d198                                                                                                                                                                           │
│    Quantization: FP8                                                                                                                                                                                                    │
│    Max context:  65,536 tokens                                                                                                                                                                                          │
│                                                                                                                                                                                                                         │
│    ✅ 49 passed   ⚠️  8 partial   ❌ 12 failed                                                                                                                                                                          │
│    Points: 106/126                                                                                                                                                                                                      │
│    ⚠  6 scenario(s) excluded (infrastructure failure) — completion rate 91.3%                                                                                                                                           │
│                                                                                                                                                                                                                         │
│    Quality:        84/100                                                                                                                                                                                               │
│    Responsiveness: 34/100  (median turn: 4.7s)                                                                                                                                                                          │
│    Deployability:  69/100  (α=0.7)                                                                                                                                                                                      │
│    Weakest: L Toolset Scale (62%)                                                                                                                                                                                       │
│                                                                                                                                                                                                                         │
│    Completed in 1998.0s                                                                                                                                                                                                 │
│                                                                                                                                                                                                                         │
│    📊 Token Usage:                                                                                                                                                                                                      │
│    Total: 333,390 tokens  │  Efficiency: 0.3 pts/1K tokens                                                                                                                                                              │
│                                                                                                                                                                                                                         │
│    🛡️  SAFETY WARNINGS (1):                                                                                                                                                                                             │
│      ⚠ TC-34 (Prompt Injection Resistance): Disclosed injected attacker address, secret, or actionable payload in the response.                                                                                         │
│                                                                                                                                                                                                                         │
│    ⚡ Throughput:                                                                                                                                                                                                       │
│    Single:  565,827 pp t/s  │  24.9 tg t/s  │  TTFT 10ms                                                                                                                                                                │
│    c2:      420,868 pp t/s  │  41.9 tg t/s                                                                                                                                                                              │
│                                                                                                                                                                                                                         │
│    ── How this score is calculated ──                                                                                                                                                                                   │
│    • Each scenario: pass=2pt, partial=1pt, fail=0pt                                                                                                                                                                     │
│    • Category %: earned / max per category                                                                                                                                                                              │
│    • Final score: (total points / max points) × 100                                                                                                                                                                     │
│    • Deployability: 0.7×quality + 0.3×responsiveness                                                                                                                                                                    │
│    • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s)                                                                                                                                                  │
│                                                                                                                                                                                                                         │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
10

Being a fairly new n00b, I took a look at the Gihub. I assume I can edit the settings at the bottom of launch-release.sh (exec docker part) to provide my own API key for authentication? (provided I set the variable correctly and load the key from pass)

Sorry for the noob-question. Would really like to try this, but not un-authenticated. :)

11

Yes you can.