I made an LLM engine called TideLM.
TideLM serves DeepSeek flash v4.1 with native MXFP4 quant.
It keeps the routed experts your traffic actually uses resident (10,000 of 15,360), learns which ones from the router’s demand while it serves, and reads the heavy misses from NVMe during decoding. A missing expert never stalls a token: the router uses the next-best resident one. It also supports GLM 5.3 (Full, not Flash) EXL3.
On a pair of Sparks, tensor-parallel:
| Prose | Code | 8K prompt, first token | Context | |
|---|---|---|---|---|
| DeepSeek-V4.1-Flash-1017, MXFP4 | 31.8 tok/s | 49.0 tok/s | 4.8 s | 1M |
| GLM-5.3, EXL3 3.0 bpw | 17.2 tok/s | 20.8 tok/s | 19.5 s | 256K |
(Medians of three runs, 1,024 tokens at temperature 0.6.)
The engine is in Rust and ships as one Docker image; scripts/tide.sh start <profile> brings up both nodes.
If you’re comparing with the EXL3 recipes: they decode faster, since their 3-bit experts are about 30% smaller to read. TideLM’s point is running the weights DeepSeek released.
It’s built on 0xBakeer’s deepseek-v41-flash-spark, and the RoCE all-gather comes from TensorFold.
If you run it on your pair, I’d like to hear your numbers and what breaks.