1x Spark tuned DSpark for DeepSeek-V4-Flash: 35 tok/s, 800+ prefill, and fast multi-agent serving

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
123

Thank you, this log is exactly what I needed, and your observation about the 131072 boundary was a genuinely good catch. Here is where I got with it today.

I rebuilt your session shape on my second box: same context size, same two-bank config, a conversation that gets rewritten by history compression the way your harness does it. My driver replays your exact sequence: a truncate at 67k with a 1.5k suffix, tiny-suffix truncates at 85k and 122k, growth between each, then the crash shape, a truncate whose cut sits just under 131072 while the conversation’s committed length sits just over it, with a 7-8k suffix, on a bank that had already been truncated six times. I ran that against both v0.5.0 (your version) and v0.5.2, with speculation active and a second stream decoding during the admits. Every variant served cleanly, and a memory checker pass over the shorter shapes came back clean too.

The one thing in your log I cannot recreate quickly is time: your drafter mlock release at 09:48. That line comes from the adaptive drafter residency logic that only ever existed in v0.5.0. It unlocks the drafter weights under sustained memory pressure, which means for the rest of your session those pages were evictable, and a deep truncate replay is exactly the kind of page-cache surge that would evict and then refault them mid-serving. I removed that whole mechanism in v0.5.1 for unrelated reasons. Since you moved to v0.5.1 you changed two things at once, the version and FORK_PARTIAL=0, so the fix could be either one.

So here is my ask for the regression box you kindly offered: run stock v0.5.2 with partial forks left ON (just drop the FORK_PARTIAL setting, everything else as before) under the same Hermes workload. If it survives the timescale that used to trip v0.5.0 within hours, the residency removal was the fix and I will close this with a regression gate I have already built from your session shape (it now replays your four-truncate sequence including the 131072 straddle on every release). If it still trips, grab the same log window and I will ship you a diagnostic build with verbose truncate-path logging next.

124

On it. Box is on stock v0.5.2 as of this morning, partial forks back on, same config otherwise. Normal agent workload running, will report either way.

125

I am a little spoiled by the dual spark but I spun up your version today—it loaded but had issues with hermes. The thinking is leaking. but I will take a closer look to see if its my doing.

Also, any chance you add a stop server option on your script? I could only stop it by killing it via PID.

Has anyone tried data parallel with two sparks for throughput?

126

It tripped. Stock v0.5.2, partial forks on, lasted about 13 hours. And this time the log has the missing piece, there’s a client-gone abort between the truncate and the cuBLAS failure:

0802 22:03:50 ds4-server: partial truncate admit bank=1 cut=38832 suffix=22192
ds4: cont-dspark yield-quench bank=0 pos=40830 spec_steps=12 debt=4.92 yewma=1.68 -> spec off for this seq (terminal)
ds4: cont pending admission aborted (bank 1, client gone, 27648/61024 prefilled)
ds4: cuBLAS f16 matmul failed: status 14
ds4: CUDA synchronize failed: an illegal memory access was encountered

So the shape looks like: truncate admit starts its replay, the client times out or disconnects partway (27648/61024 here, agent clients do this constantly on multi minute replays), the disconnect handling aborts the pending admission, and the next kernel touching that bank gets the illegal access. The bank was already truncated in place so the abort leaves it half replayed. The second bank was actively decoding both times.

I think that’s why your driver can’t reproduce it, it serves every admission to completion. Try killing the client connection around halfway through a truncate replay with a second stream decoding.

This also rewrites my earlier posts: the truncates that survived here all had fast replays, the two that crashed were the long ones. The 131072 straddle was probably a coincidence, sorry for that noise. And the residency theory is out, v0.5.2 has no residency mechanism and crashed anyway. Yesterday’s v0.5.0 log has no abort line but your v0.5.2 notes say v0.5.0 handled disconnects differently, so the connection between the two crashes is yours to judge.

Back on FORK_PARTIAL=0 and stable. Full boot-to-crash log saved, happy to run the diagnostic build.

127

Addendum, it got more interesting. 15 minutes after going back to FORK_PARTIAL=0 on v0.5.2 it crashed again, and this time on a plain warm admit, no truncate involved:

0802 22:18:22 ds4-server: warm admit bank=0 cached=54784 suffix=10398
ds4: cuBLAS f16 matmul failed: status 14
ds4: CUDA synchronize failed: an illegal memory access was encountered

No client-gone line either. So the truncate path may just be one way in, the common shape across all three of our crashes is a large suffix prefill onto a live bank. We run lazy session graphs if that helps triangulate against the Codex/LAZY_GRAPH lead. Watchdog caught it and the full log is saved. v0.5.1 with the same guard ran a full day clean, for what that’s worth.

128

Thanks for the two crash logs, they turned out to be the key to something I have been hunting for a while. An update and a couple of questions.

First: I reproduced the crash class. The shape that finally triggered it was a sequence of interrupted work: a long generation on a bank, interrupted, then a fresh admission replaying a large suffix onto the bank field. After enough of those cycles one replay dies with the same signature you posted, a cuBLAS or kernel launch failure reporting an illegal memory access. The error surfaces at whatever kernel checks next, so the named kernel in the log is the victim, not the culprit. Root cause is still open; I have a repeatable harness now and I am narrowing it.

What v0.5.4 changes for you today:

  1. A failed admission no longer kills the whole continuous batch. It aborts that one job with a loud line, the other tenants keep serving, and the bank keeps its committed prefix.
  2. Interrupted work now leaves checkpoints. An aborted replay keeps what it committed and re-announces it, so your retry resumes from the interruption point instead of re-prefilling from zero. In your crash-1 shape the retry previously re-paid the whole truncate replay.
  3. Disconnect detection recognizes reset-style closes. A client that gives up mid-stream with unread bytes sends RST, not a clean close, and v0.5.2/v0.5.3 only noticed that on a later failed write.

The questions, based on what the repro needed:

  • In the window before each crash, were there requests whose generation was interrupted, timed out, or truncated? Long responses that got cut off, client timeouts, cancels. My repro needs interrupted long generations in the mix; if your 13 hours of stable v0.5.1 traffic had the same interruptions, that is a strong version fence and I want to know it.
  • Your watchdog-log offer: yes please, and specifically the segment from the last healthy request on the crashing bank to the crash line.
  • Does your client stream, and does it sit behind a proxy? The RST detail above changes what the server can see about disconnects in each case.

I will post again when the root cause lands. The containment plus checkpoints are in v0.5.4 either way.

129

Just gonna leave it here in case those haven’t been fixed yet.

@entrpi, those made 5.3 stable for multiple long parallel agentic-like runs in a stress test.

Bugfixes for:

  1. CUDA error: an illegal memory access was encountered
    call: cudaFreeAsync(ptr, t_ds4_pool_stream)
    in: free at cuda/mmq/ds4_ggml_stubs.cu:136
    zsh: abort (core dumped)

  2. illegal memory access / cudaFreeAsync, but at much shallower depth (~40–60k tokens, well under what bugs 1/2 needed) and with zero comp-cache budget

BUGFIXES.md.txt (26.7 KB)

Bonus would be this prompt:
write 10 SECRET_KEY = '....' for django, per line, no prose, no comments
it makes current model generate infinitely very often (VLLM Qwen 122B Q4_K_XL works fine for random comparison)

130

I had problems with the previous version, it would run out of memory even with a relatively simple opencode task on a private project.

Today I took a chance, and ran an all day planning session (over 10 hours) with the new version in OpenCode. Single user.

No problems-ds4 rocks! 🦾

Thanks for all your hard work!

131

Answers to your three questions, plus v0.5.4 field results you’ll want.

  1. Yes, interruptions everywhere, and the version fence looks strong. My client (hermes-agent) streams and aborts/retries on its own timeouts constantly. Counting interruption events in its logs (stream aborts, client timeouts, cancels): v0.5.1 with FORK_PARTIAL=0, Aug 1 13:00 to Aug 2 09:10, had 8 of them across 20 hours with zero crashes. v0.5.2 had clusters at 09:32 and 11:12-11:13, then two more at 21:52 and 21:55, and crash #1 hit at 22:03:50. More churn during recovery, crash #2 at 22:18:22. Same traffic shape both days, only the version moved.

  2. Segments, last healthy request on the crashing bank to the crash line:

crash 1 (bank 1, forks on, v0.5.2):
0802 22:01:01 ds4-server: warm admit bank=1 cached=59410 suffix=249
0802 22:03:36 ds4-server: warm admit bank=1 cached=60730 suffix=225
0802 22:03:50 ds4-server: partial truncate admit bank=1 cut=38832 suffix=22192
ds4: cont pending admission aborted (bank 1, client gone, 27648/61024 prefilled)
ds4: cuBLAS f16 matmul failed: status 14

crash 2 (bank 0, FORK_PARTIAL=0, v0.5.2):
0802 22:17:20 ds4-server: warm admit bank=0 cached=49890 suffix=30
ds4: cont-dspark yield-quench bank=0 pos=49926 spec_steps=5 debt=4.80 yewma=1.68 -> spec off for this seq (terminal)
ds4: CONT_MTP_ACCEPT(DSpark) D=4 steps=909 emit=910 drafts=5 hits=1 accept=20.0% tok/step=1.00
0802 22:18:10 ds4-server: warm admit bank=0 cached=50830 suffix=1483
0802 22:18:16 ds4-server: warm admit bank=0 cached=52422 suffix=2264
0802 22:18:22 ds4-server: warm admit bank=0 cached=54784 suffix=10398
ds4: cuBLAS f16 matmul failed: status 14

Client aborts at 21:52 and 21:55 sat right before crash 1, and note the 909-step quenched generation before crash 2. Fits your interrupted-work shape.

  1. Streaming, no proxy. Plain streaming chat completions over direct LAN HTTP, python openai client, so its timeout aborts are mid-stream give-ups. Your RST read likely applies to me.

Now the v0.5.4 part: upgraded this morning, same guard, and it tripped twice. First at 11:48, about 2.5 hours in. Your containment fired exactly as designed:

0803 11:44:47 ds4-server: warm admit bank=0 cached=211352 suffix=75
0803 11:47:25 ds4-server: warm admit bank=0 cached=215592 suffix=50
ds4: CUDA end commands failed: an illegal memory access was encountered
ds4: cont admission chunk FAILED (bank 1, 37888/86179 prefilled) -- admission aborted, bank retains 37888 committed tokens, batch continues

But the context was already poisoned so “batch continues” didn’t hold, every later alloc failed and deep prompts got the serial refusal. Then five minutes after my watchdog restarted it, it went again with a third signature, CUDA compressor store launch failed. Three crashes, three different named kernels, which supports your victim-not-culprit read.

One pattern worth having: all three of today’s crashes happened under deep bank pressure, both banks had spent the preceding 20 minutes evicting and readmitting 180k to 216k committed records. Every stable stretch I’ve ever had was with banks under about 135k. I’ve capped context to 131072 on both sides as the field mitigation and it’s been quiet since, so whatever the root cause is, its probability seems to scale with bank depth. Standing by for v0.5.5, I can put 256k back on and hammer it same day.

132

Correction to my last post: the 131072 cap did not hold. Crash #4 tonight at 21:53, same illegal-access class, and this time the banks were nowhere near deep, prompts in flight were 75k and 88k with ctx capped at 131072 all day:

0803 21:53:08 ds4-server: deep-serial guard: refusing serial fallback for 88100-token prompt (max 65536)
ds4: cuBLAS f16 matmul failed: status 14

So scratch the depth-scaling idea from my last post, it was coincidence from the morning crashes. The honest summary of everything from my box: v0.5.1 ran 20 hours of this exact traffic with zero crashes, v0.5.2 and v0.5.4 both crash within hours regardless of FORK_PARTIAL, bank depth, or context size. Whatever this is, it arrived between v0.5.1 and v0.5.2, and my money is on the disconnect/abort rework interacting with in-flight bank work since my client aborts streams constantly.

Riding it out on v0.5.4 with the watchdog (4 automatic recoveries so far, about a minute each). Still good for same-day testing on the v0.5.5 candidate.

133

I upgraded from 0.53 to 0.55, and now the model is looping in OpenCode.
Maybe the start/end of the thinking is not well recognized.

I load AGENTS.md, then the Plan file for the refactoring (300 lines).
A prompt like “Write me a table in Jira format of the Ubiquitous language for XXX” takes 17 seconds with 0.53. With 0.55 it loops forever.

I start ds4 with: DS4_SERVER_COALESCE_MAX=2 DS4_BATCH_FIT_HEADROOM_MB=6272 ds4-serve --host 0.0.0.0 --ctx 245760 --tokens 32768 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 16384

OpenCode (1.18.13) is using anthropic: “npm”: “@ai-sdkai-sdk/anthropic”.
temperature, top_p, reasoning_effort, chat_template_kwargs is NOT set in opencode.json

134

Anyone get structured outputs working through the openai chat completions endpoint in python? I’m passing a pydantic model to response_format.

135

Closing the loop on the crash reports: v0.5.5 fixed it on my box. 28 hours on the full config that used to die, 256k ctx, partial forks back on, no FORK_PARTIAL guard, real agent traffic with all the usual timeouts and retries. 236 requests, zero failures, zero illegal memory access lines, and 9 partial truncate admits served clean including cuts at 144k-155k and one deep cut with a 21k suffix replay, which is the exact shape that started all this. Watchdog hasn’t fired once since the upgrade. Whatever that stream512 verdict race was, it was the thing. Thanks for chasing it down, the turnaround on this fork is something else.

136

Can you share run command? You’re using --kv-disk-dir?

Thx in advance.

137

for me, this setup works:
DS4_CONT_PREFILL_CHUNK=8192 ~/code/ds4/ds4-server --cuda -m ~/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf --dspark ~/gguf/DSpark-drafter-Q2K-Q8-0731.gguf -c 262144 --kv-disk-dir ~/.cache/ds4-kv --kv-disk-space-mb 32768 --host 0.0.0.0 --port 8000
Inference via Open WebUI in a RAG setup and via PI as agent; also recompiled with cuda-spark target

138

Overwhelmed by the outpouring of support from the big v0.5 launch a week ago, the day after DeepSeek V4 Flash 07-31 shipped. I’ve spent the whole week shipping the top priority updates from everyone’s feedback, and the engine is really in a better place thanks to your detailed reports and suggestions. Six releases in seven days, latest release is 0.5.6, and every single one includes work on your suggestions and reports.

What that looked like in practice:

The crash a few of you hit under heavy agent load? Root-caused to a timing window a few billionths of a second wide and fixed, with a reproducer that shows 5 crashes in 16 runs before, 0 after, and identical output token for token. One of you bisected it to the right kernel family before I found the race. That’s the community this project has now.

The one I’m most excited about: now you can point Claude Code or Codex straight at your Spark and they just work. All four APIs now get the full batching engine, streaming, thinking, tool calls and all. I watched Claude Code run a real tool loop against the box before writing this. Its second turn reused nineteen thousand tokens of already ingested context and paid for 42 new ones. Agent loops are nearly free after the first turn.

And a dozen or more quality of life fixes you asked for, if you want to read the full changelog: https://github.com/Entrpi/ds4/blob/batched-serving/CHANGELOG.md

Now is a great time to upgrade, from any version:

curl -sSL https://enia.cc/ds4 | bash -s – --start

139

@kafej666 sure, mine’s about as stock as it gets on v0.5.6:

DS4_SERVER_COALESCE_MAX=4 ds4-serve --host 0.0.0.0 -c 262144 --no-mtp

ds4-serve picks up the IQ2XXS 0731 gguf and the matching DSpark drafter by default so I don’t pass -m or --dspark. And no, I’m not using --kv-disk-dir. I tried disk KV earlier in the week and hit the illegal-access crash on the restore path that GaelicThndr also reported, so I left it off, and since v0.5.5 fixed the main crash I haven’t needed it. COALESCE_MAX=4 gives me 4 banks at 256k with headroom on the 128GB box, and I dropped the DS4_BATCH_VMM_BUDGET_MB guard I used to run once v0.5.1’s trim-on-evict landed.

One extra since it came up in the crash thread: I run a one-line cron watchdog that restarts ds4-server if “illegal memory access” shows up in the log tail, and on boot since there’s no autostart. Happy to paste it if useful.

140

Thx for sharing, “–no-mtp” that speed sacrifice is really worth that +quality in your reviews?

Yes, elapse share.

141

Nah, no speed hit. --no-mtp only kills MTP, and the speed comes from the DSpark drafter which stays on. I’m seeing 3-4 tok/step at ~82% accept right now with it set, so speculation is fully running. 0731 doesn’t ship an MTP head anyway, the only MTP file is the old pre-0731 one and entrpi measured that as break-even on 0731 while still costing memory. So it’s not a speed-for-quality trade, I keep full DSpark speed and just skip the wasted RAM. Both are lossless anyway so neither touches quality, that’s all on the model and the effort level.

142

Some numbers from my box (single Spark, 128GB, v0.5.6, IQ2XXS 0731 + DSpark, 256k, COALESCE_MAX=4):

Prefill lands around 900 tok/s on a cold big prompt. On decode, speculation runs around 80% draft accept, 3 to 4 tokens per step. Right now I’m seeing anywhere from 92% accept / 4.2 tok/step down to 72% / 2.9 depending on how predictable the output is. I haven’t run a clean single-stream tok/s decode benchmark so I won’t quote one, but the tok/step is what makes it feel fast. Nothing scientific, just what scrolls by during real agent work.

And since I offered it earlier, here’s the watchdog. One cron line every minute. It restarts ds4-server if the process dies, if the CUDA context gets poisoned (illegal memory access in the log), or if the API goes unresponsive on a mature process. Also covers reboots since there’s no autostart, and it saves the crash log before restarting so I don’t lose it:

# crontab -e
* * * * * /home/you/ds4-watchdog.sh

#!/usr/bin/env bash
# ds4-server watchdog: run from cron every minute.
LOG="$HOME/ds4-watchdog.log"
SERVER_LOG="$HOME/ds4-server.log"
STAMP="$HOME/.ds4-watchdog-last-restart"
note() { echo "$(date '+%F %T') $*" >> "$LOG"; }

# don't restart more than once per 3 min
if [ -f "$STAMP" ] && [ $(( $(date +%s) - $(stat -c %Y "$STAMP") )) -lt 180 ]; then exit 0; fi

restart() {
  note "RESTART: $1"
  if grep -q 'illegal memory access' "$SERVER_LOG" 2>/dev/null; then
    cp "$SERVER_LOG" "$HOME/ds4-server.log.crash-$(date +%Y%m%d-%H%M%S)"
  fi
  pkill -x ds4-server 2>/dev/null; sleep 3
  touch "$STAMP"
  nohup "$HOME/start-deepseek.sh" > "$SERVER_LOG" 2>&1 < /dev/null &
  disown
}

PID=$(pgrep -x ds4-server | head -1)
[ -z "$PID" ] && { restart 'process not running'; exit 0; }

# process alive but CUDA is dead
if tail -n 100 "$SERVER_LOG" 2>/dev/null | grep -q 'illegal memory access'; then
  restart 'poisoned CUDA context'; exit 0
fi

# dead API on a mature process (300s grace for model load)
AGE=$(ps -o etimes= -p "$PID" | tr -d ' ')
if [ "${AGE:-0}" -gt 300 ] && ! curl -s -m 10 http://localhost:8000/v1/models >/dev/null 2>&1; then
  restart "API unresponsive (age ${AGE}s)"
fi
exit 0

start-deepseek.sh is just my launch wrapper (the DS4_SERVER_COALESCE_MAX=4 ds4-serve … --no-mtp line from earlier). Since v0.5.5 it basically never fires, but it saved me a lot during the crash week.