Let’s optimize DeepSeek-V4.1-Flash-1017 for 2× DGX Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
1

Serving Guide

Model

Test Result

| model               |           test |              t/s |     peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:--------------------|---------------:|-----------------:|-------------:|-----------------:|-----------------:|-----------------:|
| deepseek-v4.1-flash |         pp2048 | 1330.49 ± 306.71 |              | 1350.14 ± 365.50 | 1343.42 ± 365.50 | 1350.99 ± 365.51 |
| deepseek-v4.1-flash |          tg128 |     26.50 ± 3.58 | 37.33 ± 1.70 |                  |                  |                  |
| deepseek-v4.1-flash |         pp2048 | 1410.24 ± 260.57 |              | 1331.50 ± 285.54 | 1324.78 ± 285.54 | 1332.78 ± 285.58 |
| deepseek-v4.1-flash |         tg1024 |     30.54 ± 5.78 | 44.33 ± 8.34 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d1024 |  1326.56 ± 29.37 |              |  1998.53 ± 66.27 |  1991.82 ± 66.27 |  1999.80 ± 66.23 |
| deepseek-v4.1-flash |  tg128 @ d1024 |     32.12 ± 1.85 | 43.67 ± 5.56 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d1024 |  1367.69 ± 15.92 |              |  1916.76 ± 13.81 |  1910.04 ± 13.81 |  1917.96 ± 13.60 |
| deepseek-v4.1-flash | tg1024 @ d1024 |     31.43 ± 5.14 | 50.67 ± 4.78 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d4096 | 1486.39 ± 132.08 |              | 3501.11 ± 313.82 | 3494.40 ± 313.82 | 3501.11 ± 313.82 |
| deepseek-v4.1-flash |  tg128 @ d4096 |     28.24 ± 2.88 | 37.00 ± 1.63 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d4096 |   1599.79 ± 7.86 |              |  3352.00 ± 31.39 |  3345.29 ± 31.39 |  3352.00 ± 31.39 |
| deepseek-v4.1-flash | tg1024 @ d4096 |     29.40 ± 2.40 | 49.67 ± 2.49 |                  |                  |                  |

llama-benchy (0.4.1.dev1+ge9be34457)

╭─────────────────────────────────────────────── 🏆 Benchmark Complete ────────────────────────────────────────────────╮
│                                                                                                                      │
│    Model:  /workspace/Model/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw                                                     │
│    Score:  92 / 100                                                                                                  │
│    Rating: ★★★★★ Excellent                                                                                           │
│    Benchmark: tool-eval-bench v2.8.1.dev13+g42a100b18                                                                │
│    Engine:       vLLM 0.32.1.dev0+g5df63a720.d20261017                                                               │
│    Quantization: EXL3                                                                                                │
│    Max context:  1,048,576 tokens                                                                                    │
│                                                                                                                      │
│    ✅ 86 passed   ⚠️  7 partial   ❌ 4 failed                                                                        │
│    Points: 179/194                                                                                                   │
│                                                                                                                      │
│    Quality:        92/100                                                                                            │
│    Responsiveness: 40/100  (median turn: 3.9s)                                                                       │
│    Deployability:  76/100  (α=0.7)                                                                                   │
│    Weakest: O Structured Output (83%)                                                                                │
│                                                                                                                      │
│    Completed in 1831.4s                                                                                              │
│                                                                                                                      │
│    📊 Token Usage:                                                                                                   │
│    Total: 694,431 tokens  │  Efficiency: 0.3 pts/1K tokens                                                           │
│                                                                                                                      │
│    🛡️  SAFETY WARNINGS (1):                                                                                          │
│      ⚠ TC-92 (Tenant Isolation With Same-Named Resources): Disclosed another tenant's data.                          │
│                                                                                                                      │
│    ── How this score is calculated ──                                                                                │
│    • Each scenario: pass=2pt, partial=1pt, fail=0pt                                                                  │
│    • Category %: earned / max per category                                                                           │
│    • Final score: (total points / max points) × 100                                                                  │
│    • Deployability: 0.7×quality + 0.3×responsiveness                                                                 │
│    • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s)                                               │
│                                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Despite the low-bit quantization, I’ve found the model performs quite well in practical use.

I’ve been serving it with a 1M-token context and using it for fairly extensive Unity development tasks, and my system hasn’t frozen once so far.

There are quite a few recipes available for this model, so I’d appreciate any recommendations for a better recipe optimized for a dual-DGX Spark setup.

2

I have found Mia’s to be faster and score higher on tool bench.

3

Because I found the model to be intelligent and capable of handling long-context agentic workflows reliably, I made some changes to improve KV cache efficiency.

--tensor-parallel-size 2 \
--kv-cache-memory-bytes 3029185024 \
--kv-cache-dtype fp8 \
--max-num-batched-tokens 4096 \
--max-num-seqs 8 \
--max-cudagraph-capture-size 32 \

With these settings, the engine reports the following KV cache efficiency:

(EngineCore pid=365) INFO 10-24 03:22:48 [kv_cache_utils.py:2331]
GPU KV cache size: 1,319,693 tokens,
Maximum concurrency for 524,288 tokens per request: 2.52x

Even with long-context workloads and high concurrency, the system maintains 2–7 GB of available memory, demonstrating efficient KV cache utilization while supporting long-context agentic workflows.

| model               |           test |              t/s |     peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:--------------------|---------------:|-----------------:|-------------:|-----------------:|-----------------:|-----------------:|
| deepseek-v4.1-flash |         pp2048 |  1544.85 ± 35.58 |              |  1142.47 ± 57.97 |  1135.94 ± 57.97 |  1143.12 ± 58.08 |
| deepseek-v4.1-flash |          tg128 |     27.93 ± 2.49 | 38.00 ± 4.55 |                  |                  |                  |
| deepseek-v4.1-flash |         pp2048 |  1502.17 ± 30.68 |              |  1174.22 ± 15.86 |  1167.70 ± 15.86 |  1174.54 ± 15.89 |
| deepseek-v4.1-flash |         tg1024 |     30.59 ± 3.00 | 53.00 ± 0.82 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d1024 | 1721.28 ± 158.93 |              | 1560.84 ± 155.67 | 1554.31 ± 155.67 | 1561.35 ± 156.39 |
| deepseek-v4.1-flash |  tg128 @ d1024 |     27.83 ± 1.92 | 36.67 ± 2.62 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d1024 |  1783.10 ± 55.58 |              |  1494.67 ± 27.65 |  1488.15 ± 27.65 |  1495.39 ± 28.16 |
| deepseek-v4.1-flash | tg1024 @ d1024 |     30.53 ± 5.48 | 48.67 ± 6.18 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d4096 |  1844.41 ± 12.51 |              |   2878.06 ± 3.55 |   2871.53 ± 3.55 |   2878.06 ± 3.55 |
| deepseek-v4.1-flash |  tg128 @ d4096 |     29.97 ± 1.70 | 38.00 ± 3.56 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d4096 | 1288.46 ± 348.26 |              | 4327.99 ± 979.54 | 4321.46 ± 979.54 | 4327.99 ± 979.54 |
| deepseek-v4.1-flash | tg1024 @ d4096 |     28.57 ± 0.80 | 53.67 ± 1.89 |                  |                  |                  |

llama-benchy (0.4.1.dev1+ge9be34457)
date: 2026-10-24 12:38:35 | latency mode: api
4

Always good to see you work on serving stacks @0rand and @PILCOTHINK
Going to test the Mia-AI one tomorrow. They claim to have lower KLD than the LibertAIDAI one:

5 
| model               |             test |             t/s |     peak t/s |           ttfr (ms) |        est_ppt (ms) |       e2e_ttft (ms) |
|:--------------------|-----------------:|----------------:|-------------:|--------------------:|--------------------:|--------------------:|
| deepseek-v4.1-flash | pp2048 @ d400000 | 1771.88 ± 51.33 |              | 194345.82 ± 5430.12 | 194342.15 ± 5430.12 | 194345.82 ± 5430.12 |
| deepseek-v4.1-flash |  tg128 @ d400000 |    45.54 ± 1.69 | 59.67 ± 1.25 |                     |                     |                     |

llama-benchy (0.4.1.dev1+ge9be34457)


| model               |           test |             t/s |     peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:--------------------|---------------:|----------------:|-------------:|-----------------:|-----------------:|-----------------:|
| deepseek-v4.1-flash |         pp2048 | 1420.45 ± 73.47 |              |  1227.41 ± 81.20 |  1223.29 ± 81.20 |  1228.25 ± 81.08 |
| deepseek-v4.1-flash |          tg128 |    42.03 ± 4.35 | 50.67 ± 3.09 |                  |                  |                  |
| deepseek-v4.1-flash |         pp2048 | 1533.06 ± 46.79 |              |  1191.95 ± 36.36 |  1187.82 ± 36.36 |  1192.66 ± 36.80 |
| deepseek-v4.1-flash |         tg1024 |    43.63 ± 3.67 | 61.67 ± 4.11 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d1024 | 1748.85 ± 30.40 |              |  1478.83 ± 35.90 |  1474.70 ± 35.90 |  1479.77 ± 35.65 |
| deepseek-v4.1-flash |  tg128 @ d1024 |    43.47 ± 2.10 | 49.00 ± 3.27 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d1024 | 1744.11 ± 19.26 |              |  1538.41 ± 19.86 |  1534.28 ± 19.86 |  1540.55 ± 19.52 |
| deepseek-v4.1-flash | tg1024 @ d1024 |    43.82 ± 1.19 | 62.33 ± 4.03 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d4096 | 1718.95 ± 55.21 |              | 3032.58 ± 100.12 | 3028.45 ± 100.12 | 3032.58 ± 100.12 |
| deepseek-v4.1-flash |  tg128 @ d4096 |    44.23 ± 1.59 | 50.67 ± 2.05 |                  |                  |                  |
| deepseek-v4.1-flash | pp2048 @ d4096 | 1723.04 ± 62.78 |              | 3082.65 ± 162.28 | 3078.52 ± 162.28 | 3082.65 ± 162.28 |
| deepseek-v4.1-flash | tg1024 @ d4096 |    45.16 ± 9.01 | 62.33 ± 4.11 |                  |                  |                  |

llama-benchy (0.4.1.dev1+ge9be34457)

    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8 \
    --block-size 64 \
    --kv-cache-memory-bytes 3029185024 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 8 \

(EngineCore pid=364) INFO 10-27 05:01:50 [kv_cache_utils.py:2331] GPU KV cache size: 1,319,693 tokens, Maximum concurrency for 524,288 tokens per request: 2.52x

Compared to the first Docker image version, the KV cache capacity has increased significantly, and inference speed has improved slightly. It can now maintain stable inference speeds even with a 400K context. Additionally, even with a 400K context, it still leaves approximately 3–5 GB or more of available memory.

previously, only LibertAIDAI models were supported, but now other 2.9 bpw EXL3-based models can also be used.

If you’re interested in the latest version, I recommend downloading and using the Docker image linked in the guide again.

6

A data point for this thread’s question about a better dual-Spark recipe. Prefill runs at 2,606-2,827 tok/s from 4K to 256K. Decode reaches 322 tok/s structured and 192 tok/s code at 4 streams, and 110.5 code and 66.0 prose at 1 stream. The setup is 1017 on 2x Spark, plus an x86 box that takes the Engram tables and the DSpark drafter off the Sparks.

I’m Nick Mermiges, Esq., founder of PLI Labs. This is our build and repo.

What’s different from the usual TP2 recipe:

  • Weights and image: HeNryous’s 2.75 bpw EXL3 weights and his v5 image, unmodified, with our overlays mounted read-only.
  • Engram: layers 1 and 14 are served from the x86 host’s RAM over 40 Gb/s RoCE, with local NVMe as the exact fallback.
  • Drafter: DSpark runs on two NVLinked RTX 3090s in the x86 host, with sampled drafting and a window of 5.
  • KV: the freed memory goes to 6 GiB per rank. That gives 1,934,490 tokens of pool, 327,680 per request, 4 sequences and a prefill chunk of 8,192.

Numbers come from MiaAI’s sparkDash protocol, re-implemented in tools/sparkbench.py. Both Sparks ran at -lgc 0,2300, 3 reps, median, 2026-10-31:

1 stream 4 streams, total
structured, T=0 138.3 322.1
code, T=0 110.5 192.0
prose, T=0 66.0 130.1
prose, T=0.7, thinking 55.9 105.3
Prefill 4K 32K 128K 256K
tok/s (sparkDash rate) 2,827 2,667 2,650 2,618

The 256K first token arrives in 100.4 s. Decode after a 128K prompt is 61.9 tok/s at 1 stream.

The limits, so nobody is misled:

  • Compared with your 400K-1M runs, we cap at 327,680 per request and have no 1M run yet.
  • MiaAI is faster on prose (80.7 against 66.0 at 1 stream).
  • A short request behind a cold 93K prompt waits 34.3 s.
  • I haven’t run llama-benchy yet, so these tables aren’t directly comparable with your pp2048/tg rows. llama-benchy at depth is next, and I’ll post it here.

Quality, on sudoingX’s p4_kld.py panel for real tool-call turns: KLD 0.0184 and 95.88 % top-1 for the weights we serve, and 0.0183 and 95.91 % for HeNryous’s.

The repo has the launch scripts, the overlays, docs/install.md, the raw JSON in runs/ and the Hugging Face card link: GitHub - ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload: DeepSeek-V4.1-Flash-1017 on 2x DGX Spark + an x86 offload host (Engram over RDMA, DSpark drafter on 2x RTX 3090). Prefill 2,606-2,827 tok/s to 256K; 322 tok/s structured, 192 tok/s code at 4 streams; 1.93M-token KV pool. Production stack, install guide, raw JSON. · GitHub

PLI Labs builds private, on-prem AI around an organization’s own work product and offers consulting on serving like this: https://proprietarylegal.ai