DeepSeek-V4.1-Flash-1017 (EXL3 2.9bpw) on 2x DGX Spark with TensorFold: 57 tok/s text, 92 tok/s code, 2.1-2.3x the vLLM kit

Yeah, you are right. He is using version 1.0.2, in which this problem does not yet exist.

I’m just gonna leave it here:

My prediction last week from the research I did was 60%-70% performance of sparks.

@0rand Below is my current zig flash17 build, it was created by cursor grok 4.7 at 28 oct 2026

0rand/deepseek-v41-tensorfold-spark,commit 055c4aefa8794f1876e2894d666693f3784bb0fa

jayleaton/DeepSeek-V4.1-Flash-1017-EXL3-2.8bpw

Image tensorfold-dsv41-zig:local, built on spark1 from engine/zig at the
checkout commit and loaded on spark2. Both nodes have image id
sha256:e0d7b0b88cce0c435f76b1c4482887b2cda09b0029949cc8a41257b59ac6aa1f.
scripts/check-zig-image.sh passed (NCCL, mlx5, cuBLAS / cuBLASLt 13.6.0.2).

vendor/TensorFold was initialized at c464617. The three patches in
patches/ were applied under build/python-source. That tree was used only
to generate assets. The running server is the Zig image, not that Python tree.

Engram was packed with scripts/pack_engram.py from the 1017 raw shards and
flash17/config.json:

The stack is overall working and I got 97 on tool bench with max effort. It has an annoying connection errors (I think zig restarts itself), every time it happens, reasonix will stop, “continue” to continue the work.
can you tell me your version?

below is my customized extra request body in reasonix:

{
"chat_template_kwargs": {
"reasoning_effort": "max",
"thinking": true
}
}


That tells us nothing. If stack restarts then most likely it’s memory guard:

  1. Did you build from main-0rand branch?
  2. Did you lower the memory guard values to 1gb from 4gb as described in readme and in sample env?
  3. Did disable early oom?

Was working fine for a week without restarts and reboots. Absolute stable beast.

@0rand is it something for your repo? Serve compute in the engine for every chat by default, pin its release engine, and ship the studies behind it by jayleaton · Pull Request #87 · jayleaton/deepseek-v41-tensorfold-spark · GitHub

Intersteting from what I understand it hide thinking and internal tool calls to do some caps on data already in context. To save session context? So it’s not reinjected back. But it also means replies will be just assistant message and you may wait for a long time for reply to arrive having no idea what is going on. Not sure it is very useful, can be done via proxy as well. Worth looking into, but so far jayleaton version diverged a lot and merging with current main is not feasible, better start from scratch and reimolement. Bth not sure I want to do that. Some else can. I am supper happy with the current stack as is and may wait until next release, be it 4.2 or 5 as it is rumored

My primary dual GB10 setup is currently running DSV4 Flash 0731. The cluster was previously running Nemotron 3 Super. I used Qwen 3.8:27b running on my 5090 to help with the current build. Basically I staged everything and did the planning with Nemotron, then switched to Qwen once the cluster was down. Qwen did great at following-through and getting the cluster stable with the new model.

I use Hermes agent with everything so all the context and session data stays consistent, then just swap out the LLM like a lobotomy, heh.

Cheers!

cold start of flare’s glm 5.3 flash with cached per rank artefacts on nodes is 60 seconds (can be faster), zig’s ds v4.1 ±120s; if you have prefix cache offload you can swap between them fast enough, that it is as fast as without prefix offload on larger context.

@0rand I’m being told I should be reporting this, so since I’m using your recipe I figured I’d post it here.

Two days ago my worker spark floated an error in my logs; Xid 13 subtype, “SM Cga Exception: CTA Not Present”, raised under the inference process (tf-dsv41-m1). I’m being told this points at a kernel in that inference stack and I should send to the recipe owner.

jason@spark2:~$ sudo journalctl -k -b 0 --no-pager | grep -n ‘NVRM: Xid’
1451:Nov 10 11:28:11 spark2 kernel: NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Cga Exception on (GPC 2, TPC 0, SM 0): CTA Not Present
1452:Nov 10 11:28:11 spark2 kernel: NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x5257b0=0x0 0x5257b4=0x20 0x5257a8=0x1c81fb60 0x5257ac=0x1174
1453:Nov 10 11:28:11 spark2 kernel: NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Cga Exception on (GPC 2, TPC 0, SM 1): CTA Not Present
1454:Nov 10 11:28:11 spark2 kernel: NVRM: Xid (PCI:000f:01:00): 13, pid=1496813, name=tf-dsv41-m1, Graphics Exception: channel 0x00000002, Class 0000cec0, Offset 00000510, Data 001c62ac

spark2 details:
== OS ==
PRETTY_NAME=“Ubuntu 24.04.5 LTS”
VERSION_ID=“24.04”
VERSION_CODENAME=noble
DGX_NAME=“DGX Spark”
DGX_PRETTY_NAME=“NVIDIA DGX Spark”
DGX_SWBUILD_VERSION=“7.5.0”
DGX_OTA_VERSION=“7.6.0”
Linux 7.0.0-1024-nvidia aarch64 GNU/Linux
Static hostname: spark2
Icon name: computer-server
Chassis: server 🖳
Machine ID: 482288dd556b4387862a9cfd993bbe20
Boot ID: ecce7d8c24ad4b31ab1f39f4046574fc
Operating System: Ubuntu 24.04.5 LTS

== Board / firmware ==
NVIDIA_DGX_Spark
Manufacturer: NVIDIA
Product Name: P4242
Serial Number: XXXX

== CPU / memory ==
Architecture: aarch64
CPU(s): 20
Vendor ID: ARM
Model name: Cortex-X925
Thread(s) per core: 1
Core(s) per socket: 10
Socket(s): 1
CPU(s) scaling MHz: 100%
Model name: Cortex-A725
Thread(s) per core: 1
Core(s) per socket: 10
Socket(s): 1
CPU(s) scaling MHz: 100%
total used free shared buff/cache available
Mem: 123Gi 111Gi 4.3Gi 1.3Gi 9.4Gi 11Gi

== GPU / driver ==
name, driver_version, vbios_version, memory.total [MiB], compute_cap, uuid
NVIDIA GB10, 580.178.04, 9A.0B.34.00.00, [N/A], 12.1, GPU-b87b4d7f-2de0-ed4b-2c99-6e71449c058a
NVRM version: NVIDIA UNIX Open Kernel Module for aarch64 580.178.04 Release Build (dvs-builder@U22-I3-G08-04-4) Tue Jul 7 13:36:23 UTC 2026
GCC version: gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1)
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0

Honestly have no idea, sounds extremely alien

Long post - but a bit of info

PSA: a raised vm.min_free_kbytes from an older recipe makes MemAvailable under-read, and everything keyed on it misfires ( I spent a long time due to this troubleshooting)

Setup: four GB10 nodes on a switchless ring (two DGX Spark 4 TB, two ASUS GX10 1 TB), run as two TP=2 pairs: DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on the Zig TensorFold engine, and GLM-5.3-Flash NVFP4. We run MiaAI-Lab’s 2× recipe (now at c7f2b85), not Elevatormusic’s kit, but this is host-side and applies to either.

Cause. MemAvailable ≈ MemFree − watermark reserve + part of the page cache, and the reserve is about 1.5 × min_free_kbytes. Our DSv4.1 nodes still had 5 GiB set from albond’s Qwen3.5 recipe in June (right for vLLM then; our fault for never revisiting it). That hides about 7.5 GiB. The GLM pair never had the setting, which made it a handy control.

Test: grep -E 'MemFree|MemAvailable' /proc/meminfo; cat /proc/sys/vm/min_free_kbytes. If MemAvailable is several GiB under MemFree, that’s your watermarks. On the kernel’s own default (about 45 MB on these boxes) it reads true.

What it broke for us

  • The recipe’s load check: the default 12.2 GiB reserve wouldn’t load with ~119 GiB free, so we lowered it to 7.5, against the recipe’s own warning.
  • The recipe’s memguard: killed the rank at once, so we turned it off.
  • Production: the watermark took nearly all the ~8 GiB left after load, the page cache had nowhere to live, and under live traffic the head sat at kswapd 90% / PSI full 11% with MemFree showing a healthy 7.8 GiB. Reproduced three times at 5 GiB; gone at 2 GiB.

Where we ended up: stock watermarks, recipe defaults. Both DSv4.1 nodes are back on the kernel’s own value, confirmed after a reboot (if you clear this, check the original sysctl file too, or it comes back at the next boot). The reserve is back at the recipe’s 12.2 GiB. Head MemAvailable at stock, MiB:

Shape Head Worker
Idle 7,868 9,657
Two 500K sessions in the pool 7,379 9,300
One session to 298K over 10 turns 7,083 8,893
Four 500K sessions with KV_SHARD=1 (2.1M pool) 7,009 8,794

PSI stayed at 0.00 in every case. The sharded four-session fill is the telling one: at the 2 GiB setting it left the head at 0.4 to 0.7 GiB, and at stock it leaves 7 GiB. Sharding puts all the extra on the head; the worker doesn’t move.

Our guard now keys on PSI (5% full for 30 s) with MemAvailable floors at 2.5 / 1.5 GiB as a backstop, which works now that the number reads true. The recipe’s memguard (2.5 GiB on MemAvailable) should be usable again at stock too; we haven’t evaluated it yet. A 24-hour soak at these settings is running as I write this.

Same trap elsewhere. 0rand’s main-miaai-checkpoint tuning withholds about 2.2 GiB (1 GiB + watermark_scale_factor=50) and its earlyoom acts at 1 GiB of MemAvailable. On our head that stopped a healthy engine at ~3.2 GiB of real free memory, twice. @0rand, the receipt is yours if useful.

Not symptoms: heavy disk reads during prefill (Engram is O_DIRECT here: 20.5 GiB for one 106K prefill), and MemFree sitting near the high watermark.

Numbers. tool-eval-bench 2.7.1.dev11, --seed 42 --hardmode, thinking on, 4 engine slots on both pairs:

high max tokens at high
DSv4.1, Zig engine (31457d0 / c7f2b85) 89 / 90 92 / 92 ~76K
DSv4.1, 0rand branch A (engine 1.0.0) 92 95 ~61K
GLM-5.3-Flash NVFP4, temp 0.3 92 92 ~38K

The Zig build returns 400 on response_format; that’s the gap between the DSv4.1 rows. The max-effort failure set was byte-identical across the two Zig checkouts. Both models pass the sleeper-injection case and both fail the safety gate.

Single-stream decode, thinking off: DSv4.1 71 tok/s prose / 106 code; GLM 31 to 32 / 37 to 39. DSv4.1 totals 129 tok/s at four streams. Net: clearly faster for one person waiting, roughly level when agents share a pair.

@Elevatormusic, prefill on our pair, cold, single request: 106K in 52.2 s (~2,035 tok/s), 500K in 255 s (~1,960), 600K in 309.6 s (~1,940), needles at 5/50/95% all found. Different build from yours.

Next up. MiaAI-Lab’s four-Spark TensorFold recipe on the same ring: TP=4 with DeepSeek’s native FP8/MXFP4 weights, 16 sessions each holding a full 1M context, and structured output. It doesn’t claim a faster single stream than our pair (65.8 / 117.4 tok/s prose / code by the author’s numbers); the draw is sessions, headroom and unquantized weights. All four nodes are now on stock watermarks, so the ranks run on matching hosts. First-light numbers will go here, with the same bench command. If anyone has run it on a mixed DGX Spark / GX10 ring, I’d like to hear how the 1 TB workers coped.

If anyone has anything they want me to try before running the migration to tp4 let me know.

I’ve been using @0rand’s stack for 1017 to replace 0731 for like more than a week, I think it is pretty good so far, I don’t regret the upgrade. I use max effort with reasonix.

I was using Deepseek-v41-1017-exl3-2.9bpw(EXL3,avg_bits2.9) before, but there's an error:

mtp.0.ffn.gate.bias: f16 {128}, expected f32 storage of {128}
UnexpectedTensor

I am not sure it was fixed, at the time to work with 0rand’s stack (28 Oct 2026)

you have to use 2.8bpw: jayleaton/DeepSeek-V4.1-Flash-1017-EXL3-2.8bpw

are you using @0rand’s stack 0rand/deepseek-v41-tensorfold-spark?

It’s actually a tiny patch that is needed to make 0rand/jayleaton’s tensorfold build work with Mia’s weights.

I’ll share that patch here:

diff --git a/engine/zig/zig/src/families/deepseek_v41/names.zig b/engine/zig/zig/src/families/deepseek_v41/names.zig
index 216feeb..1cfb3b6 100644
--- a/engine/zig/zig/src/families/deepseek_v41/names.zig
+++ b/engine/zig/zig/src/families/deepseek_v41/names.zig
@@ -83,7 +83,7 @@ pub fn expertName(block: []const u8, e: u32, p: ExpertProj, buf: []u8) ![]const
 pub const Store = enum {
     /// norms: bf16 in the pack (fp32 in the forward), fp32 accepted
     norm,
-    /// fp32 only (sinks, hc_*, router bias)
+    /// fp32, or fp16 widened exactly to fp32 (sinks, hc_*, router bias)
     f32,
     /// fp16 or bf16, rounded to bf16 once (router gate, indexer weights_proj, Markov heads)
     half,
@@ -91,7 +91,7 @@ pub const Store = enum {
     pub fn accepts(s: Store, d: DType) bool {
         return switch (s) {
             .norm => d == .bf16 or d == .f32,
-            .f32 => d == .f32,
+            .f32 => d == .f32 or d == .f16,
             .half => d == .bf16 or d == .f16,
         };
     }
diff --git a/engine/zig/zig/src/families/deepseek_v41/plan.zig b/engine/zig/zig/src/families/deepseek_v41/plan.zig
index 277e9f9..a65c579 100644
--- a/engine/zig/zig/src/families/deepseek_v41/plan.zig
+++ b/engine/zig/zig/src/families/deepseek_v41/plan.zig
@@ -253,7 +253,7 @@ const Builder = struct {
             const info = try b.pack.need(name);
             var n: u64 = 1;
             for (info.dims()) |d| n *= d;
-            if (info.dtype != .f32 and info.dtype != .bf16 or n != b.c.hidden + b.c.dspark_markov_rank) return error.UnexpectedTensor;
+            if (info.dtype != .f32 and info.dtype != .bf16 and info.dtype != .f16 or n != b.c.hidden + b.c.dspark_markov_rank) return error.UnexpectedTensor;
             try b.mark(name);
             confidence = .{ .name = try b.dupe(name), .info = info, .slice = .whole, .bytes = info.nbytes };
         }

I’ve been using Mia’s weights with this patch and have been very happy.

No, MiaAI-Lab’s 2× recipe at c7f2b85 on Mia’s 2.9bpw. Thanks @wolttam for the patch; I’ll try 0rand’s Zig build with it once our soak finishes. It should be finishing up shortly.

Actually, that was quicker than I thought -

Follow-up: the 24 h soak at stock watermarks passed. 8,869 requests of mixed long sessions plus short requests, no guard kills, no PSI trips, no restarts.

If you set your own memory floors:

  • Set floors from where trouble starts, not from where normal running sits: We derived a floor of 6.25 GiB MemAvailable from an hour of synthetic tests. Run in shadow alongside the real guard, it would have fired 4,577 times in a day. Real traffic sat about 1 GiB lower than any synthetic shape. The floors we kept, 2.5 / 1.5 GiB with PSI as the main trigger, never fired.
  • The head’s engine memory creeps up, then stops: Ours climbed about 0.3 GiB over the first few rounds of load. A restart dropped it 0.7 GiB, eight more rounds brought it back to the same level, and it then stayed flat for ten hours. The allocator is keeping its high-water mark, not a leak, so it’s nothing to restart for.