1x Spark: DeepSeek-V4-Flash-0731 @ 1,000 tok/s prefill, 59 tok/s multi-agent serving

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
41

Hi @entrpi I tested the v0.5.3 max-reasoning candidate on my single DGX Spark with DeepSeek V4 Flash 0731:

--ctx 393216
--tokens 16384
--threads 10

I found two issues that may be relevant to DS4 or the Q2 quant.

1. Long reasoning sometimes becomes repetitive

In standalone high/max-reasoning tests, some responses repeatedly revisited the same reasoning without reaching a final answer, eventually exhausting the output-token budget. An earlier repetition guard reduced the symptom, but I’m trying to determine whether the underlying behavior comes from the Q2 quant, prompting or runtime.

Have you observed repetitive reasoning or output-budget exhaustion with this quant under high/max effort?

2. What happens after a client timeout?

During a long agent session, large context-compression requests to DS4 repeatedly exceeded the client’s 300-second timeout:

20:50:51 compression started: ~95,970 tokens
20:55:51 request timed out after 300 seconds

21:02:32 compression started: ~97,193 tokens
21:07:32 request timed out after 300 seconds

21:13:59 compression started: ~97,440 tokens
21:18:59 request timed out after 300 seconds

Normal shorter requests still completed between these timeouts. Although the retries were initiated by the client, I want to understand the DS4 behavior:

  • Does DS4 immediately cancel generation when the client disconnects or times out?
  • Could an abandoned generation continue consuming compute?
  • Is there a logging flag that shows request IDs, cancellations and final stop reasons?
  • Would you recommend a lower output limit for summarization requests while retaining 16,384 tokens for normal agent work?

During inspection, DS4 remained at approximately 95% GPU utilization and 116,762 MiB memory. If you tell me which DS4 diagnostics to enable, I can reproduce this and provide request-level logs.

42

This is something that I noted earlier in the thread. I’m pretty sure it’s this 731 version of DSV4F that’s the culprit. I’m starting to wonder if it was just “benchmaxxed” and only tests better for those tasks. Either that, or maybe this particular Q2 quant is just done more poorly. Personally I would wonder if moving back to the pre-731 quant that was used originally would be the right call. I noticed it converged on a solution more quickly and deliberately, on my task, it was run out of reasoning tokens probably 50% of the time now.

But this doesn’t have much to do with Entrpi’s work with getting the thing actually running quickly, which works great. I’m wondering if we can just move the old model from Antirez DSV4F back into the same location and have it use that thing instead, with all of the performance gains of the 731 version.

Either that or maybe we could somehow get the Q3 unsloth version running here instead. Should be similar size?

43

Old model still works fine, yes. There’s also a version that’s mostly q2 experts and has q4 in final layers which takes another 10GB, so eats into KV budget, but will still run on Sparks okay. I haven’t tested that yet, but it’s on the backlog.

44

Oh I thought that’s what we were using here already. Hmm… I guess that makes more sense then. When I originally used antirez’s installer, this is the version I downloaded:

./download_model.sh q2-q4-imatrix  # 96/128 GB RAM machines, q2 with last 6 layers q4

So is what we are using right now just a pure q2 quant mostly, not this version? I haven’t yet tested whether the q2/q4 hybrid version of post 731 DSV4F is similarly challenged (assuming it exists).

45

It’s been a long time since I last used DeepSeek. I stopped because the R1 distill or DS coder models left a pretty bad impression on me. The tooling wasn’t great, but more importantly, the reasoning quality was disappointing. I don’t really care how well it scores on benchmarks—what I experienced in practice was that it would frequently get trapped in repetitive reasoning loops until it exhausted the token budget without producing a useful answer.

Is this issue still present in the latest version?

46

On the timeouts, your four questions in order.

Does ds4 cancel generation when the client disconnects? Yes, since v0.5.2, on by default. Queued requests are reaped before any engine work is spent, admission prefills probe the client socket between chunks, and both decode paths probe every token/step. An orderly close is seen within about one decode step. The one gap: a peer that vanishes without closing the socket (hard network drop, some proxies) is only caught on streaming writes; a non-streaming request from a silently vanished client runs to its budget.

Do abandoned generations keep consuming compute? Not if the client closed, which a normal client-side timeout does. You can verify in your server log; the three abort paths each print a line:

serial decode aborted at N tokens: client disconnected

cont pending admission aborted

reaped queued request: client disconnected

If your 20:50 to 21:19 window has the timeouts but none of these lines, please tell me, that would mean your client held the socket open after giving up and I would want to reproduce that.

Request logging? Three surfaces: --trace FILE writes a full per-request record (request json, cache decision, token stream), GET /metrics and GET /v1/stats expose live counters, and every response carries a timings block with ttft, prefill and decode times, and cached vs computed token counts.

What limits for summarization? The arithmetic at your shape answers this. A ~96k cold prefill is roughly two minutes on a Spark. Your 16384 output budget at deep-context decode rates is another ten-plus minutes if the model uses all of it, and a high-effort reasoning loop will. No 300 second client timeout can hold that. For compression calls I would cap max_tokens at 2000 to 4000, set reasoning_effort low or off, and either stream (first token arrives right after prefill, and a broken pipe then cancels generation immediately) or raise the client timeout to 600+ seconds.

One more thing your timeline shows: on v0.5.3 an aborted reasoning request retires its context bank without a reusable record, so each of your three retries re-paid the full ~96k prefill. v0.5.4 fixes this class: an aborted request keeps its committed prefix and re-announces it, so a retry resumes from where the abort landed instead of starting over.

The 95% GPU utilization and 116 GiB resident you observed are both normal, that is the model plus context banks plus active decode from your other requests, not a leak.

If you can share whether your client streams and what sits between it and the server (proxy or direct), plus the log window above, I can close the loop on the disconnect question.

47

v0.5.4 is out

Every change traces to a report in these threads; thanks all, especially GaelicThndr and Hawi.

  • Reasoning effort works on every box now. Below 384K context the server silently collapsed every level to low, so on a single Spark high and max never did anything. Explicit request values are now honored at any --ctx, and --reasoning-effort sets the boot default for clients that never send the field.
  • Interrupted work leaves checkpoints. An aborted or timed-out request keeps what it prefilled and a retry resumes from there instead of re-ingesting the whole prompt. This was the sting in Hawi’s timeout report: three retries of a ~96k summarization, each re-paying the full prefill.
  • A failed prompt-ingest chunk aborts that one request, not the whole continuous batch. I reproduced the illegal-memory-access crash from the DSpark thread on my own box. Containment ships today; root cause is harnessed and should be fixed in v0.5.5 next.
  • A drafter failure disarms speculation and serving continues plain. Output cannot change; only speed can.
  • --mem-floor-gb (default 4) gates cache growth against live free memory, whatever the box has lost since boot.
  • Mid-stream disconnects that arrive as resets are now caught at the liveness probe, not at a later failed write.
  • Batched responses report real cached_tokens in usage (it was always 0 before; at 240k a warm continuation now shows 246,598).

Upgrade is the usual installer run or a pull and rebuild. If you are on v0.5.3 the daily update check will tell you on its own. Everything ships default-on with env kill switches.

48

Thanks for all the hard work @entrpi !

Has anyone tried to run the fork from inside of docker? I have it fully functional but I am not getting the speeds I would have expected. My prefill speed is average 370 t/s when I input a 12K lorum ipsum. These numbers seem similair as the upstream, but I am quite sure I have git cloned the repo from entrpi.

decoding is the same story, only getting around 15 average.

This is my compose.yml:

services:
  ds4:
    build:
      context: .
    container_name: ds4
    restart: unless-stopped
    ports:
      - 8000:8000
    volumes:
      - ./models:/models:ro
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - CTX=262144

This my Dockerfile:

ARG CUDA_ARCH=sm_121
ARG CUDA_VERSION=13.3.1
ARG UBUNTU_VERSION=24.04

# -----------------------------------------------------------------------------
# Builder
# -----------------------------------------------------------------------------
FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu${UBUNTU_VERSION} AS builder

ARG CUDA_ARCH

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        python3 \
        make \
        g++ && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /build

# Entrpi/ds4
# ----------
COPY ./ds4 .
RUN make cuda CUDA_ARCH="${CUDA_ARCH}" -j"$(nproc)"

# xangel82/DS4-GB10-GX10-DSpark-CUDA
# ----------------------------------
# COPY ./DS4-GB10-GX10-DSpark-CUDA .
# RUN make -B -j"$(nproc)" cuda-spark-graph-sm121

# -----------------------------------------------------------------------------
# Runtime
# -----------------------------------------------------------------------------
FROM nvidia/cuda:${CUDA_VERSION}-runtime-ubuntu${UBUNTU_VERSION}

ENV DEBIAN_FRONTEND=noninteractive

COPY --from=builder /build/ds4-server /usr/local/bin/
COPY entrypoint.sh /entrypoint.sh

RUN chmod +x /entrypoint.sh

ENV MODEL=/models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf
ENV DSPARK_MODEL=/models/DSpark-drafter-Q2K-Q8-0731.gguf

ENV HOST=0.0.0.0
ENV PORT=8000
ENV CTX=32768

EXPOSE 8000

ENTRYPOINT ["/entrypoint.sh"]

And this my entrypoint:

#!/bin/sh
export DS4_CONT_MTP_MODE=2
export DS4_CONT_DSPARK=1
export DS4_DSPARK_MODEL="${DSPARK_MODEL}"

exec ds4-server \
    --cuda \
    --model "${MODEL}" \
    --ctx "${CTX}" \
    --host "${HOST}" \
    --port "${PORT}"

I have the repo cloned locally, which I pull every so often before I am building a new image.

These are my logs I’m getting from the container:

ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings
ds4: CUDA loading model tensors into device cache
ds4: CUDA loading model tensors 16.02 GiB cached
ds4: CUDA prepared model tensor mappings 16.37 GiB
ds4: CUDA loading model tensors 32.06 GiB cached
ds4: CUDA prepared model tensor mappings 32.07 GiB
ds4: CUDA loading model tensors 48.02 GiB cached
ds4: CUDA prepared model tensor mappings 48.43 GiB
ds4: CUDA loading model tensors 64.06 GiB cached
ds4: CUDA prepared model tensor mappings 64.13 GiB
ds4: CUDA loading model tensors 80.04 GiB cached
ds4: CUDA prepared model tensor mappings 80.24 GiB
ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 20.131s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 3.72 GiB (raw 0.36 + compressed 3.36) + buffers 2.00 GiB + resident model 80.76 GiB = 86.48 GiB planned
ds4: memory detail: ctx=262144 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=65538 backend=cuda
0803 07:44:26 ds4-server: context buffers 5855.75 MiB (ctx=262144, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=65538)
0803 07:44:27 ds4-server: listening on http://0.0.0.0:8000
0803 07:44:53 ds4-server: chat ctx=0..12804:12804 prompt start
0803 07:44:53 ds4-server: chat ctx=0..12804:12804 prefill chunk 0/12804 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.000s
0803 07:45:04 ds4-server: chat ctx=0..12804:12804 prefill chunk 4096/12804 (32.0%) chunk=379.34 t/s avg=379.33 t/s 10.798s
0803 07:45:15 ds4-server: chat ctx=0..12804:12804 prefill chunk 8192/12804 (64.0%) chunk=368.63 t/s avg=373.90 t/s 21.909s
0803 07:45:27 ds4-server: chat ctx=0..12804:12804 prefill chunk 12288/12804 (96.0%) chunk=359.52 t/s avg=368.98 t/s 33.302s
0803 07:45:28 ds4-server: chat ctx=0..12804:12804 prefill chunk 12804/12804 (100.0%) chunk=337.17 t/s avg=367.58 t/s 34.833s
0803 07:45:28 ds4-server: chat ctx=0..12804:12804 prompt done 34.833s
0803 07:45:31 ds4-server: chat ctx=12804..12854:50 gen=50 THINKING decoding chunk=14.72 t/s avg=14.72 t/s 3.397s
0803 07:45:35 ds4-server: chat ctx=12854..12904:50 gen=100 THINKING decoding chunk=14.83 t/s avg=14.78 t/s 6.768s
0803 07:45:38 ds4-server: chat ctx=12904..12954:50 gen=150 decoding chunk=14.83 t/s avg=14.80 t/s 10.138s
0803 07:45:42 ds4-server: chat ctx=12954..13004:50 gen=200 decoding chunk=14.84 t/s avg=14.81 t/s 13.507s
0803 07:45:44 ds4-server: chat ctx=13004..13042:38 gen=238 decoding chunk=14.85 t/s avg=14.81 t/s 16.067s
0803 07:45:44 ds4-server: thinking live checkpoint remembered ctx=0..12804:12804 live=13042 visible=44465
0803 07:45:44 ds4-server: chat ctx=0..12804:12804 gen=238 finish=stop 50.899s

Anyone ideas what I’m missing? I am not close to the 1k encode I expected, and the same for the decode.

Thanks in advance :D

49

thank you, I did just try to upgrade and noticed I can no longer run with bigger context than around 90k.

With 130k etc. it fails with following message in log: 0803 09:34:31 ds4-server: serial right-size: no graph fits (prompt=15178 need_min=16202 boot -c 130000); refusing 503

in Pi Agent:
Error: 503: {“message”:“Server is temporarily at capacity for a 15178-token serial request (no session graph fits beside the batch banks); retry shortly”,“type”:“invalid_request_error”}

Error: Retry failed after 3 attempts: 503: {“message”:“Server is temporarily at capacity for a 15178-token serial request (no session graph fits beside the batch banks); retry
shortly”,“type”:“invalid_request_error”}

50

Hi @entrpi

First of all, thank you for your excellent work on DS4. I’m currently running it directly on a single Asus Ascent GX10 with latest update using your installer (v0.5.3)

My setup is essentially:

#!/usr/bin/env bash
set -euo pipefail

BASE="/home/dgx001/apps/ai/ds4-0731"

mkdir -p "$BASE"

export DS4_SRC_DIR="$BASE/code/ds4"
export DS4_GGUF_DIR="$BASE/models"

echo "Installing DS4 into:"
echo "  Source : $DS4_SRC_DIR"
echo "  Models : $DS4_GGUF_DIR"

curl -fsSL https://raw.githubusercontent.com/Entrpi/ds4-on-spark/main/install.sh \
| bash -s --

mkdir -p "$BASE/logs"

nohup ds4-serve \
    --host 0.0.0.0 \
    --port 8888 \
    -c 262144 \
    > "$BASE/logs/server.log" 2>&1 &

echo "Started."

sleep 2

pgrep -a ds4-server

I’m currently using it with the Kilo extension in VS Code and running real-world coding tasks on an actual project to evaluate the model.

At runtime, the system typically uses around 118 GB out of 122 GB of RAM, while the GPU stays around 35 W (~95% utilization).

I have a few questions:

  1. Is there any way to limit the maximum concurrent sessions (CCU) to 4? I’d like to trade concurrency for a larger context window (393,216 tokens), but I couldn’t find any CLI option for that.

  2. During inference, I constantly observe disk reads between roughly 150 MiB/s and 400 MiB/s. Why is this happening? I assumed everything would already be resident in RAM after startup. Is this expected, and does it have any impact on prefill or decode performance?

  3. I couldn’t find any runtime statistics showing the actual prefill or decode throughput (tokens/s) for my workload. Is there any way to monitor these metrics? The Kilo extension in VS Code doesn’t expose them. Subjectively, reasoning and text generation feel quite slow—only a few tokens per second—even though my prompts are usually only around 30k–60k tokens, well below the configured 262k context window.

Thank you again for all the work you’ve put into this project. I really appreciate it and would be grateful for any insights you can share.

server.log (8.9 KB)

P/s: Also, after about 40 minutes of running with Kilo, the RAM usage gradually climbs to nearly the full 122 GB, at which point the entire GB10 crashes. The system becomes completely unresponsive and disconnects from SSH, and the only way to recover is to perform a hard reset using the physical reset button on the device. 😄

51

Hi @Entrpi,

I wanted to share some feedback after upgrading to v0.5.4 and testing it on a single Asus GX10 with Kilo (VS Code) against a real production codebase.

After upgrading to v0.5.4, I also set:

DS4_SERVER_COALESCE_MAX=4

My server is running with:

  • Context: 262,144

  • Single GX10

  • Kilo as the OpenAI-compatible client

  • Real coding/refactoring tasks (not synthetic benchmarks)

Before limiting the coalesce value, my observations were:

  • Startup memory was around 118 / 122 GB.

  • During longer coding sessions, RAM would eventually grow to almost 122 GB, causing the whole machine to OOM and requiring a hard reset using the physical reset button.

  • During inference, the SSD was continuously reading around 150–500 MiB/s.

  • GPU power was typically only 25–35 W.

  • Generation and reasoning felt much slower than expected.

After upgrading to v0.5.4 and setting DS4_SERVER_COALESCE_MAX=4, the behavior changed noticeably:

  • Startup memory dropped to about 105 / 122 GB.

  • The server has now been running continuously for about 30 minutes.

  • After two large refactoring tasks, the active conversation has grown to more than 150k context tokens.

  • Tool calling has been very reliable throughout the session.

  • RAM has gradually increased to only 109 / 122 GB and is still completely stable at the moment.

  • SSD reads are now essentially 0 MiB/s during inference, instead of continuously reading 150–500 MiB/s as before.

  • GPU power is much higher now, usually around 60–65 W, suggesting the GPU is staying busy instead of waiting.

  • Subjectively, generation and reasoning are significantly faster than before.

So far, this configuration looks much healthier for a single-user coding workflow on the GX10.

I’ll continue running longer real-world sessions and keep you updated with any new observations. Hopefully these real-world results are useful for further optimizing DS4 on a single GX10.

Thanks again for all the work you’ve put into DS4.
newlog:
server.log (32.3 KB)

52

I don’t know if this may be of interest to anyone, but I could run @entrpi’s version of DS4 with Jetson Thor.

Tested with Entrpi/ds4 v0.5.4, the 0731 IQ2XXS model and matching DSpark drafter, compiled only for Thor’s sm_110.

Performance

Prompt Prefill Decode
2.4K 488 tok/s 23–24 tok/s
21K 465 tok/s 23 tok/s
63K 431 tok/s 22 tok/s
105K 403 tok/s 16.5 tok/s
125K 389 tok/s 16.6 tok/s
244K 328 tok/s 12.9 tok/s
480K 250 tok/s 10.5 tok/s

DS4’s original streaming Top-512 selector caused Xid 13 errors on Thor at long context. Disabling it fixed stability but reduced performance.

I replaced it with a bounded, atomics-free Thor selector. It matched the safe implementation and passed tests up to 480K without Xid, OOM or restarts.

At approximately 105K context:

  • Safe fallback: 364 tok/s
  • New Thor selector: 403 tok/s
  • Improvement: 10.6%

Using 8K prefill chunks at 256K context provided another ~4% improvement. The stable 512K configuration remains two request banks, 4K chunks and an 8 GiB memory floor.

The complete implementation is on my NemoClaw-Thor fork in commit 7e1f590.

git clone https://github.com/pastoriomarco/NemoClaw-Thor.git
cd NemoClaw-Thor
./serving/start-ds4.sh start
53

Curious to get your feedback on how you are liking Deepseek overall and if you are getting decent results?

For my background, I’ve been testing against a large production code base, running an asp.net framework to blazor .net core migration/upgrade.
DS4 seems to be very solid on my Ascent, my prompt processing is usually over 1000 tok/s and my generation is consistently ~18 tok/s and it’s very stable. Initially this seems acceptable, but in practice it’s not very usable. I have a good example page that is partially migrated but still has visual and functional issues when comparing to the legacy site. I’ve been testing this single migration improvement task repeatedly trying to tune DS4 (very little difference from the stock settings so I reverted back to defaults).
Now, the issue I have is when I compare the deepseek results/performance to Qwen 3.6 27b nvfp4. I’ve run this task on each with minor prompt/instruction changes and found the following :
Deepseek initially had more accurate results but at a much slower speed. For example it fixed one migration issue but took nearly 30 minutes (in the end the only change was flipping true to false on one line).
On the same task, Qwen initially fixed 3 issues and one fix wasn’t correct, but it took about 9 minutes.
After some prompt, safety gate, and instruction changes Qwen now consistently fixes all 3 issues in under 9 minutes.
I’m just finding that deepseek is too slow for a small quality bump over 27b, just curious if you are finding the same or if you’re getting better overall performance from deepseek.

54

Running DeepSeek-V4-Flash 0731 (IQ2XXS + DSpark drafter) stock DGX Spark via the ds4-on-spark installer, v0.5.4 pinned. A few findings:

Throughput on the box, defaults plus the two knobs below: OpenAI chat completions do about 27-30 tok/s single-stream at low reasoning effort, about 22 at high or max. 12 concurrent chat completions aggregate to roughly 62 tok/s on the continuous batch lane. A 31.8K-token prompt prefilled at about 1000 tok/s. The Anthropic /v1/messages path is serial-only in this build (around 14 tok/s single, no aggregation under load), and /v1/responses behaves the same. Only OpenAI chat/completions and completions get the fast lane.

The memory wall is the thing to plan around. At 262144 context with the 16 GiB serial headroom, the box ends up with about 1 GiB available. The 4 GiB memory floor then rejects every batch admission, everything falls back to serial, and the deep-serial guard 503s any prompt over the default 64K (DS4_SERVER_SERIAL_MAX_TOKENS). Deepest refusal we saw was 131K tokens. Dropping context to 196608 plus a clean reboot fixed it: 8 GiB available, batch max_seq 16 to 23, zero 503s since, and decode barely moved (the 262k to 192k tax is about 1-2 tok/s).

Where I landed: ctx 196608, headroom 16 GiB, DS4_SERVER_SERIAL_MAX_TOKENS=131072 so deep prompts fall back and serve slowly instead of dying, reasoning-effort default high (A/B: low 26.6 tok/s, high and max both around 22). Two installer quirks to flag: it defaults to ctx 32768 and binds 127.0.0.1, so pass --host 0.0.0.0. Also, stock v0.5.4 has no auth at all, no --api-key flag, so put something in front of it if the box is reachable beyond your LAN.

Bottom line: it does what the thread claims, and the only real friction is the memory floor at big context. 192K is the point where it just works.

55

Adding data: same crash class on 1x Spark, agentic (Hermes) traffic, v0.4.2 through v0.5.3

Running Entrpi/ds4-on-spark under an agentic client (Hermes, tool-calling, retries on 503, background review process hitting the same endpoint) rather than direct chat. Confirmed across three engine versions tonight — v0.4.2, v0.5.2, v0.5.3 — same failure class each time: CUDA_ERROR_OUT_OF_MEMORY on fork admit, or the box thrashing hard enough (vmstat bi 150k-400k) that SSH itself became unreachable for 20-40 min until the request timed out.

One clean-boot run (24.91 GiB free at start, max_seq 29, comp-cache budget 6.78 GiB — best allocation I got all night) survived 9 hours before OOMing at fork 11:

07:34:58 fork src=0 dst=1 cached=27331
07:42:35 fork src=7 dst=8 cached=64700
09:42:35 fork src=7 dst=9 (new fork, 2hrs later, branches from 7 again — not 8)
12:48:15 fork src=7 dst=10 (3hrs later, same source)
13:49:09 fork src=7 dst=11 → CUDA_ERROR_OUT_OF_MEMORY

Notable: every fork after #8 branched from src=7 again, never reusing 8/9/10. Concurrency was confirmed at 1 the whole time (served=1 x 50, zero served=2), so this wasn’t a load problem — banks just weren’t being reclaimed. Reading ds4_server.c, freeing is tied to on_done firing on request completion (line ~12861/11918). My working theory, given @scope_cr33p’s note about the disconnect/abort rework: a client that retries-then-abandons on a 503 (which Hermes does 3x with backoff) may leave the original request’s socket in a state where on_done never fires, so its bank sits resident forever.

Also tried --kv-disk-dir + --kv-disk-space-mb 32768 and --warm-weights in combination — that run got worse (bi spiked to 336k-421k), so warm-weights forcing the full 80GB mmap resident on top of the 78.71GB repacked artifacts seems to hurt rather than help. Haven’t isolated disk-KV alone yet.

@scope_cr33p — did v0.5.1 predate the aligned-artifacts repack, or did it just handle the abort case differently? Tempted to try DS4_REF=v0.5.1 next but wanted to ask before burning another few hours if you already know.

56

@MrKen0 nope, v0.5.1 has the same repack. My v0.5.0 boot log already shows the full aligned-artifacts build (474 artifacts, 78.71 GiB) and v0.5.1 shipped byte-identical serving vs v0.5.0. The thing that changed in v0.5.2 is disconnect handling. Before that a cancelled request just ran to completion, after it the server aborts mid-flight when the client bails. That’s the fence on my box. v0.5.1 took 20 hours of Hermes traffic with all the same timeouts and retries, zero crashes. v0.5.2 and v0.5.4 both die within hours no matter what I set. So ya, v0.5.1 is worth your hours.

Heads up though, your failure looks different than mine. I get context corruption (illegal memory access, server stays up but poisoned). Yours is a real OOM from banks not getting reclaimed, and you’re running max_seq 29 with forks. I run COALESCE_MAX=2 and FORK_PARTIAL=0 so partial forks are off entirely. If it’s an on_done leak like you think, v0.5.1 might have it too and my 2 bank setup just can’t accumulate enough to notice. So don’t read my 20 stable hours as proof for your shape.

My stability stack if you want it: FORK_PARTIAL=0, COALESCE_MAX=2, and a cron watchdog that restarts when illegal memory access shows up in the log tail (saves the crash log first). Happy to share the script. entrpi says root cause is v0.5.5, I’m on standby to test it.

57

@scope_cr33p Appreciate the detail, that distinction matters. True my failure isn’t proven by your 20 hours different mechanism, and COALESCE_MAX=2 may just be capping you below the threshold where it’d show up.

Before I burn another session on v0.5.1, one question, does COALESCE_MAX=2 reduce how many banks can be live concurrently, or does it cap something else (max forks total, max parallel branches)? I’m running agentic traffic with a background review process hitting the endpoint independently of the main session, so I need to know if capping that value also caps how much real concurrent context I can hold, vs just how many banks pile up over a long session.

If it’s the latter (caps accumulation, not concurrent capacity), I’ll try v0.5.1 + FORK_PARTIAL=0 + COALESCE_MAX=2 as a bounded test and report back with fork counts over a few hours rather than assuming it’s fixed. Would take the watchdog script too if you’re willing to share, sounds like the sane way to run this until v0.5.5 lands regardless of which theory is right.

Thanks for actually checking your own boot log instead of just going on vibes.

58

I’m on v0.5.4 with the same agentic traffic class and see the same memory-loss signature: box down to 1 GiB available, every batch admission rejected on the memory floor, and the log literally says ‘the box lost memory since boot’. Hit 68 deep-serial refusals in about four minutes before I stepped in. So the leak is still present in 0.5.4, which means the abort/disconnect rework you asked about predates it and does not close this.

I didn’t root-cause it either. Bought headroom instead: ctx 196608 plus a clean reboot took it from 1 GiB to 8 GiB available, banks got reclaimed, and the batch lane runs again. Workaround, not a fix, but it supports the ‘give the box slack and it survives for hours instead of minutes’ reading.

Your fork-7 pattern lines up with refused jobs never firing on_done. I just wired a banks_live-over-time card into monitoring to watch for exactly that creep.

59

that “lost memory since boot” line is exactly what I’d expect to see given the on_done theory — good to know it’s not v0.5.2-specific, kills that idea cleanly

the 196608 number is useful, that’s a gap neither @scope_cr33p nor I actually tested, we jumped straight to 131072 or 262144. good middle ground if it buys you 1 to 8 GiB just from ctx alone

at this point feels like 3 of us independently landing on the same shape: banks that outlive their request, no root cause yet, “give it slack + reboot” as the only real mitigation. hope that’s enough for entrpi to actually reproduce it for 0.5.5

banks_live-over-time is smart, wish I’d had that running instead of manually grepping fork admit lines all night. mind sharing what you’re using for the monitoring card? might set up the same thing if I go back in for another round

60

‘lost memory since boot’ line, yeah, that’s the tell. I didn’t have the on_done lens when I hit it, I just watched the box eat memory until admissions died. Your read fits what I saw.

196608 was the gap nobody had touched, which is exactly why I tried it (well, not really, I happen to try 192k by habit…) 131072 felt way too small (might as well run QWEN 122B again), 262144 sat on the redline, the middle was a good target. The ctx drop plus a clean reboot took it from 1 GiB to 8 GiB available and decode barely moved.

The monitoring card is nothing fancy: Prometheus scraping ds4’s /metrics, ds4_banks_live, ds4_banks_total and ds4_kv_pages_resident are all gauges it already exports. Grafana stat with a sparkline, thresholds so it flips red when live creeps. Screenshot here, but you can’t see the sparkline as it’s idle at 0 at the bottom of the card.

image
image1920×1047 239 KB