LMCache on DGX Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I launched Deepseek V4 Flash on 4 DGX Spark. I want to launch LMCache on the fifth DGX Spark via a 200 Gbps interface. Does anyone have experience? Is it possible to do this? Is there a ready-made Docker with LMCache for DGX?

2

The internal economy of Anthropic/OpenAI (their prompt caching is the same thing), but self-hosted with infinite TTL — they give you 5 min/1 h and charge you for the cache-read; you save forever and the read costs bandwidth from your NAS.

3

And if I need to reduce queues when a group of programmers works simultaneously ? As for me, this is beneficial. They have a large context that takes a long time to load during communication.

4

Thanks for the question. You’re looking to extend your Deepseek V4 Flash deployment (4 DGX Spark units) with LMCache on a fifth unit via 200 Gbps—that’s a solid use case for reducing queue latency under concurrent load.

To point you toward the right solution (ready-made Docker or configuration steps), I need a few specifics:

  1. Current deployment configuration: How are your 4 DGX Spark instances running Deepseek V4 Flash connected? (e.g., network topology, which interfaces are in use, any load balancing setup?)

  2. LMCache errors or blockers: Have you encountered any specific errors or performance issues when attempting to set up LMCache on the fifth DGX Spark?

  3. Queue behavior under load: You mentioned reducing queues when multiple programmers work simultaneously—what queue behavior are you currently observing, and what is your target queue depth or latency?

  4. LMCache version/image: Are you using a specific LMCache version or Docker image, or are you looking for recommendations on which to use?

Once I have these details, I can either point you to a ready-made Docker setup or walk through the configuration steps.

5

Here’s the LMCache blocker summary, ready to share:

LMCache on GLM-5.2 (4× GB10, vLLM 0.23.1rc1.dev197, TP=4, fp8_ds_mla + DSA sparse-MLA)

Setup that worked mechanically: LMCache MP (multiprocess) connector, per-node GPU-attached sidecars (lmcache server, CUDA IPC, --ipc=host), RESP L2 tier (Valkey) on a separate x86 host over 100GbE. Dual-group DSA layout was recognized correctly (group 0: 22 indexer layers @132B/token; group 1: 79 MLA layers @656B/token ≈ 54.7KB/token). Stores flowed GPU→L1→L2 fine, chunks survived full stack restarts, and restore speed was excellent when it fired: “Retrieved 32000 tokens in 0.344 seconds” (~99.4% external hit rate) vs ~102s to recompute the same prefill.

Blocker #1 — the showstopper (L1 allocation refuses to evict): a restore must stage the entire session’s chunks through free L1 (125 × 14,012,416-byte blocks for a 32k-token session). The batched allocation does not trigger synchronous L1 eviction — if L1 is warm with other sessions’ LRU-evictable chunks, it fails outright:

LMCache WARNING: Failed to batched allocate 125 memory blocks of size 14012416
because no enough memory is available (short by 57 blocks) (memory_management.py:1492)

Result: zero hits reported to vLLM, silent full recompute. Under any realistic multi-session workload L1 is always warm, so the L2 tier’s contents are effectively unreachable. On unified-memory GB10s we can’t afford session-sized L1 (only ~4-6GB spare per node), so this kills the feature on this hardware class. Reproduced on both v0.5.0 and HEAD (d1b6371).

Blocker #2 — restore correctness unverified/suspect: the one config where restores did complete (v0.5.0, 5GB freshly-empty L1), restored-KV generation diverged from computed-KV at temperature 0 (‘ready’ vs corpus gibberish) — suspected layer-group ordering or TP-rank-dedup issue in the DSA dual-group path (Valkey held exactly one chunk copy for 4 TP ranks). Couldn’t re-verify on HEAD because of blocker #1.

Blocker #3 — classic connector is a trap for this model class: LMCacheConnectorV1 (non-MP) initializes its engine on rank 0 only under MLA/TP, but vLLM fires start_load_kv on all ranks → ranks 1-3 hit a bare assert self.lmcache_engine is not None (vllm_v1_adapter.py:791), eject from the forward, and rank 0 deadlocks in the abandoned NCCL collective → engine death via sample_tokens RPC timeout. Very expensive failure mode for what should be a config error. The classic adapter also has no KV-cache-group support at all (no DSA indexer cache handling), so it’s architecturally wrong for these models regardless.

Minor gotchas worth passing along: PyPI sdist builds as version 0.0.0 (install from git tag with SETUPTOOLS_SCM_PRETEND_VERSION); no aarch64 wheels (source build of c_ops required — it does compile fine for sm121); lmcache server HTTP API defaults to port 8080 (collides with vLLM); sidecar restart under a live vLLM strands the KV registrations and wedges the engine; expandable_segments:True is rejected with any KV connector (correctly, by vLLM’s validator)

Environment

  • LMCache HEAD d1b6371 (2026-07-01), source-built aarch64 (SETUPTOOLS_SCM_PRETEND_VERSION);
    also reproduced on v0.5.0
  • vLLM 0.23.1rc1.dev197, LMCacheMPConnector, kv_role kv_both
  • Model: GLM-5.2 (glm_moe_dsa) Int4-Int8Mix, TP=4 across 4x NVIDIA GB10 (sm121, unified
    memory 128GB/node), kv_cache_dtype fp8_ds_mla, MTP speculative (k=4), sparse-MLA (DSA)
  • Sidecar per node: lmcache server --l1-size-gb 3 --eviction-policy LRU --max-workers 4 --port 6555 --l2-adapter '{"type":"resp","host":"10.0.0.10","port":6379,...}'
  • L2: Valkey on a separate host over 100GbE

What works

  • Dual-group DSA registration is correct: group0 = 22 indexer layers @132B/token, group1 =
    79 MLA layers @656B/token; stores flow GPU→L1→L2 (Valkey holds all chunks, verified).
  • When L1 is large enough AND free, restore is excellent: “Retrieved 32000 tokens in
    0.344 seconds” (v0.5.0 run with 5GB L1 freshly restarted).

Bug

Restore of a 32k-token session (125 chunks x 14,012,416 B) fails whenever L1 has
insufficient FREE blocks, even though (a) all chunks are present in the RESP L2 and
(b) L1 is full of evictable LRU chunks from other sessions:

LMCache WARNING: Failed to batched allocate 125 memory blocks of size 14012416
because no enough memory is available (short by 57 blocks) (memory_management.py:1492)

Result: zero external hits reported to vLLM, full prefill recompute (~102s vs 0.34s).
The batched allocation does not trigger synchronous L1 eviction (the watermark-based
L2EvictionController is async and does not run in the allocation path), and there is no
partial-restore fallback. Under any realistic multi-session workload L1 is always warm,
so L2 contents are effectively unreachable and the cache only ever serves L1-resident
sessions.

Expected

Either (1) the restore allocation path evicts LRU L1 blocks (that are not pinned/in-flight)
to make room, or (2) restore proceeds partially/streamed in smaller batches, or (3) at
minimum document that l1-size must exceed the largest session to be restorable.

Secondary observation (separate issue if useful)

On v0.5.0 with a successful full restore (5GB L1), restored-KV generations diverged from
computed-KV generations at temperature 0 on this DSA dual-group model (‘ready’ vs corpus
gibberish) — possible layer-group ordering or rank-dedup issue. Could not re-verify on
HEAD because restores never succeeded there (this bug). Happy to re-test once the
allocator issue is fixed.

Also noted

The classic LMCacheConnectorV1 path hard-crashes this model separately:
vllm_v1_adapter.py:791 assert self.lmcache_engine is not None fires on TP ranks 1-3
(engine initialized on rank 0 only under MLA), desyncing the TP group into a collective
deadlock. MP connector avoids this; classic adapter also lacks KV-group support for the
DSA indexer cache, so this is presumably WAI, but the bare assert produces a very
expensive failure mode (engine hang, not an error).

6

  1. There are 8 devices connected via Mikrotik CRS804 DDQ. Interfaces in spark-vllm-docker (ETH_IF=enp1s0f1np1, IB_IF=rocep1s0f1, roceP2p1s0f1). Deepseek V4 Flash uses the original recipe from spark-vllm-docker (modified --tensor-parallel-size 4 and --max-model-len 1,000,000 and launched with Ray).

  2. Initially I wanted to deploy LMCache via Docker, but I followed the repository GitHub - umianta/lmcache-vllm: Scripts and benchmarks for running [LMCache](https://lmcache.ai) KV-cache offloading alongside [vLLM](https://vllm.ai). Tested on NVIDIA DGX Spark but works on any CUDA-capable machine · GitHub and installed it directly into the system on the 5th Spark.

  3. Currently, when 3 or more people work simultaneously (VSCode + KiloCode), queues appear. I preliminarily think the problem is the batching speed on large contexts. Therefore, the task is to use an external cache to store the chat session (something like shared memory).

  4. LMCache v0.5.0 is installed on the 5th DGX Spark. Ideally, I would still move it to Docker (I haven’t done it because I don’t have much experience with it).

7

Very interesting experience. I’m currently downloading nvidia/GLM-5.2-NVFP4. I think I’ll run it on 8 devices and if there are enough resources, I’ll deploy LMCache for it as well (play around with gpu-memory-utilization for GLM)

8

Prefix Aware Routing — production-stack It’s interesting to add this service by service in front of the nodes.