Adding data: same crash class on 1x Spark, agentic (Hermes) traffic, v0.4.2 through v0.5.3
Running Entrpi/ds4-on-spark under an agentic client (Hermes, tool-calling, retries on 503, background review process hitting the same endpoint) rather than direct chat. Confirmed across three engine versions tonight — v0.4.2, v0.5.2, v0.5.3 — same failure class each time: CUDA_ERROR_OUT_OF_MEMORY on fork admit, or the box thrashing hard enough (vmstat bi 150k-400k) that SSH itself became unreachable for 20-40 min until the request timed out.
One clean-boot run (24.91 GiB free at start, max_seq 29, comp-cache budget 6.78 GiB — best allocation I got all night) survived 9 hours before OOMing at fork 11:
07:34:58 fork src=0 dst=1 cached=27331
07:42:35 fork src=7 dst=8 cached=64700
09:42:35 fork src=7 dst=9 (new fork, 2hrs later, branches from 7 again — not 8)
12:48:15 fork src=7 dst=10 (3hrs later, same source)
13:49:09 fork src=7 dst=11 → CUDA_ERROR_OUT_OF_MEMORY
Notable: every fork after #8 branched from src=7 again, never reusing 8/9/10. Concurrency was confirmed at 1 the whole time (served=1 x 50, zero served=2), so this wasn’t a load problem — banks just weren’t being reclaimed. Reading ds4_server.c, freeing is tied to on_done firing on request completion (line ~12861/11918). My working theory, given @scope_cr33p’s note about the disconnect/abort rework: a client that retries-then-abandons on a 503 (which Hermes does 3x with backoff) may leave the original request’s socket in a state where on_done never fires, so its bank sits resident forever.
Also tried --kv-disk-dir + --kv-disk-space-mb 32768 and --warm-weights in combination — that run got worse (bi spiked to 336k-421k), so warm-weights forcing the full 80GB mmap resident on top of the 78.71GB repacked artifacts seems to hurt rather than help. Haven’t isolated disk-KV alone yet.
@scope_cr33p — did v0.5.1 predate the aligned-artifacts repack, or did it just handle the abort case differently? Tempted to try DS4_REF=v0.5.1 next but wanted to ask before burning another few hours if you already know.