@entrpi First of all, thank you for your great works!
Most ways to get a 250B-class model onto a 128 GB box remove something — prune
experts, drop layers, distill. I wanted to know whether one could be made to fit
with nothing removed, and then actually served.
It fits. K-EXAONE-236B-A23B — LG AI Research’s 237B MoE — now runs on a
single DGX Spark with all 128 of 128 routed experts present in all 47 MoE
layers, the shared expert, the dense layer 0, and the original MTP block.
781 tensors, identical to the BF16 source. No pruning, no expert dropping,
no layer truncation, no distillation. The only thing that changed is how many
bits each tensor is stored in, assigned by what the tensor does rather than by
a global bit budget: router, norms, attention, shared expert and dense layer 0
stay at 8-bit or F32, and the compression comes almost entirely from the routed
experts, which hold ~64% of the parameters.
441.63 GiB of BF16 → 85.56 GiB, 5.16×. And it serves at the model’s full
262,144-token context on one GB10, at 103.95 GiB of 121.6 GiB resident.
hf download Baekpica/K-EXAONE-236B-A23B-Mixed-Quant-GGUF \
--include 'K-EXAONE-236B-A23B-MXQ-IQ2XXS-Q3K-Q4Edge-Q8Dense-MTPQ8-v1-*.gguf' \
--local-dir ./gguf
git clone https://github.com/Baekpica/ds4 && cd ds4
git checkout 920427ac124078af021a0736792d2115b1d00bc2
make cuda-spark # forces CUDA_ARCH=sm_121
./ds4-server -m ../gguf/K-EXAONE-236B-A23B-MXQ-...-v1-00001-of-00003.gguf \
--cuda -c 262144 --host 0.0.0.0 --port 8001
Point -m at the first shard; it finds the other two. Cold start to
listening is about 4 minutes, dominated by a one-time aligned repack.
Highlights
The whole context fits with the model resident. 84.48 GiB of weights,
12.30 GiB of KV at 262,144 tokens, 1.60 GiB of workspace. The KV budget is what
makes it affordable: K-EXAONE runs an LLLG sliding-window schedule, so only 12
of the 48 layers hold the full context and KV costs 48 KiB/token instead of the
192 KiB/token a fully global GQA stack would need.
Multi-turn chat does not re-pay its prefill. A chat client replays the
assistant’s previous reply as text, and re-tokenizing it does not reproduce the
token IDs the model sampled — so an is-a-prefix KV test fails on a continuation
that shares 98.6% of its tokens. Resuming at the divergence point instead:
| Turn | Prompt tokens | Time to first token | Tokens reused |
|---|---|---|---|
| 1 — cold, ~7K document + question | 6,978 | 165.7 s | 0 |
| 2 — same history + the reply + a follow-up | 7,083 | 5.9 s | 6,992 |
| 3 — a different document, cold | 6,725 | 137.0 s | 0 |
24× on turn 2, and turns 1 and 3 unchanged — an unrelated prompt is not
falsely matched onto a live session.
The MTP block actually executes. blk.48 ships inside the same GGUF; no
separate draft model. llama.cpp stores those tensors and ignores them. Every
draft is verified against the target’s own argmax and committed only on an exact
token-ID match. Honest result, though: it is a net loss on this hardware —
see caveats.
OpenAI-compatible, and checked. 16/16 validation checks pass: streaming,
stream_options.include_usage, thinking-mode reasoning_content separation
from content, stop reasons, no state bleed across concurrent requests, greedy
reproducibility. Long-prompt recall was checked with a planted needle at 10%,
50% and 90% depth of a 7K-token document — 3/3.
Measured throughput
Greedy, thinking off, one cold prompt per measurement over
/v1/chat/completions with streaming.
| Prompt tokens | Prefill t/s | Decode t/s | TTFT |
|---|---|---|---|
| 1,451 | 53.0 | 10.51 | 27.4 s |
| 3,941 | 51.6 | 9.05 | 76.4 s |
| 8,222 | 47.9 | 7.38 | 171.6 s |
| 16,376 | 42.3 | 5.42 | 387.1 s |
Both curves fit cleanly and were validated predictively before extrapolating —
decode predicted 3.58 t/s at 32K against 3.56 measured, cold prefill predicted
1311 s to 40,960 tokens against 1308.8 measured.
This is not a speed post. It is a “the whole model is here, and it serves”
post. Read the caveats before planning around it.
Caveats
- 256K is a memory result, not a throughput result. The context is
allocated and resident, but a cold 262K prompt would take about 8 hours to
prefill and decodes below 1 tok/s at that depth. Useful working depth on one
GB10 today is roughly 2K–32K. - MTP is slower than plain decode here, so it ships off by default with an
automatic loss quench. The limit is the cost of the two-row verify pass, not
draft quality — the same shape @entrpi documented for DSpark on DeepSeek-V4-Flash,
and for the same reason: the draft rows route to disjoint experts, so a shared
weight sweep reads the union and saves nothing. - Concurrency does not help yet. Aggregate decode is flat from 1 to 8
streams (11.12 → 10.80 tok/s). Partly inherent to top-8-of-128 routing, partly
engine-side. IQ2_XXSon routed gate/up is aggressive. Expect degradation relative to
Q4_K_Mon tasks that lean on rarely-activated experts.- Quality numbers in the model card were measured on llama.cpp, not on this
serving path. 32 fixtures against the officialQ8_0build. - Pin ds4 at or after
920427a. Earlier commits sized the sliding-window KV
ring to the attention window alone while prefill ran 2,048-token chunks, so
all but the last row of each chunk attended over overwritten slots. 36 of 48
layers are sliding, and long-prompt comprehension was badly degraded. Short
prompts were unaffected, which is exactly why it survived an API test suite.
Repos
- Weights —
Baekpica/K-EXAONE-236B-A23B-Mixed-Quant-GGUF
(85.56 GiB, 3 shards, model card with the full recipe and every measurement) - Engine —
Baekpica/ds4,
branchfeature/exaone-model-loader— theexaone-moemodel family: GQA with
per-head QK-norm, the LLLG sliding-window schedule, sigmoid/top-8 routing,
sm_121kernels, and theblk.48MTP graph - Converter and reports —
Baekpica/k-exaone-mixed-ds4
— recipe, per-tensor verification, imatrix coverage, raw benchmark records