Could DeepSeek-V4.1-Flash run at its original precision on 4× DGX Spark by offloading Engram to NVMe?
I am not a developer, so this is only an idea for people with the technical skills to evaluate.
DeepSeek-V4.1-Flash is over 500 GB, so its official weights cannot simply fit into the combined 512 GB unified memory of four DGX Sparks. However, around 196B parameters belong to the Engram lookup tables rather than the main computational backbone.
Could the original FP8 Engram tables be sharded across the NVMe drives of four Sparks, while keeping the backbone resident in unified memory?
Qwen3.8-Flash-Next already uses a similar idea: its large n-gram table can be offloaded to host memory or streamed from SSD because only a small number of rows are accessed for each token.
A possible layout might be:
Keep the original backbone weights in the unified memory of 4× DGX Spark.
Split the original FP8 Engram tables across the four local NVMe drives.
Use the remaining unified memory as a page cache.
Give the earlier Engram layer higher cache priority.
Prefetch lookup rows asynchronously.
Keep all original weights unchanged, with no additional quantization or expert pruning.
The goal would not necessarily be maximum speed. Even 20–35 tokens/s could be useful if the model retained its original precision and agentic capabilities.
I do not have the technical ability to implement or validate this. Is this architecture realistically possible, and would anyone familiar with vLLM, SGLang, DwarfStar, or DGX Spark be interested in trying it?
Yes, this is very possible, but software stack is key.
I really am hoping for code-smart folks here to get this on int4 and 2 sparks…if Alex is right @0rand there is a chance this can load respectably in 2 boxes with original weights as well. But I’ll settle for half the precision, knowing DS is amazing even w some brain damage.
Funnily enough I’ve just been testing Mia’s 1 spark recipe on my 21 year old son’s new spark. He’s joined the spark owning club (he’s at university but has spent the last year working as a machine learning engineer on his year in industry).
I’m quite surprised how good it is - 25 t/s on prose, 300k+ context. And seems pretty smart, too. Much better than I thought - though I suspect the either Qwen flash next or 3.8 27b will be a better fit for his coding heavy workload.
Just get a CRS504 and 100G QSFP28 DAC’s. If you feel short changed with bandwidth you can add a second CRS504 later and run them in parallel for the full 200G.
Currently working on 2-bit version for my own research/curiosity. Looking very likely that 4 x Sparks will be able to run this and Engram wont be an issue one way or another. Also looking likely that Dual Spark owners, can at least play around with it at 2 bit. Will share once its working
DeepSeek has a reference implementation of the model in HF; I’ve been using it to wire up engram serving from disk.
Numbers: TP=4, 47.2 GiB of engram per box on NVMe (2 shards of 23.6 GiB), 600 s at batch 8, 369 tok/s over 27,700 steps, zero prefetch misses.
engram offload and 3-bit quant of experts should let it server on 2 sparks. I have a 4x cluster so I’m shooting for a native serve with no quant. Will post again if/when I get it working.
I say posts like this should be deleted immediately. Otherwise Spark prices are gonna double every week!
We should only share our benchmark results in DMs 😄
Definitely not a question for me. I have misread it being int8 quantized 305B, in reality its in4 packed into int8 by pairs. So - it can be quantized but it will be 2-bit, not 4-bit. Probably not for me.
We have 0731 and Vision Exp and Qwen 4 is rumored to come in Q3. There will be good models for 2xSparks one way or another and we already have them. No need to chase every shiny object every day, IMO. Dirt cheap on API, btw.
i agree , 2BIt is not for me too. ANd V4 Vision EXP is working so well , i wont need anything for a while .
I will get back to GLM 5.3 after tht pasky !!!!!!!!!!!!!! repeat problem had been fixed and we will still have equivalent model in 2x spark.
Using it right now (to see what the fuzz about) - I do believe it is the same model (deepseek-flash tag, no version) that was served yesterday as -expires-on-0910 and the same as vision-exp that was silently changed to upcoming 4.1 a week ago or so (suddenly sped up 2x+ times)