I managed to get DeepSeek v4 Flash (IQ2XXS quantized) running on a single GB10!
(or rather, I managed to get an LLM to hack on it until it worked… can’t really take too much credit myself here.)
So far I’ve gone from not running at all to ~4t/s → ~8t/s → ~15t/s gen (at short context lengths).
Prefill is also slow at ~80t/s… but hey, it runs!
It’s still holding a CPU core at 100% and the GPU (according to nvidia-smi) at ~90% most of the time, so I’m expecting to get this running even faster within the upcoming days (and at this point I can let it just work on itself, hah).
Currently running it with these arguments:
`llama-server -hf antirez/deepseek-v4-gguf -c 524288 -np 1 --cont-batching -ngl all -fa 1 --batch-size 4096 --direct-io --no-mmap --jinja --fit off -t 10`
though I have had it OOM once so some tweaking is still needed.
Update: oops, looks like the version I pushed crashes on an assert… will update as soon as I can. fixed.
This implementation only works with the DeepSeek V4 Flash GGUFs published for this project. It is not a general GGUF loader, and arbitrary DeepSeek/GGUF files will not have the tensor layout, quantization mix, metadata, or optional MTP state expected by the engine. The 2 bit quantizations provided here are not a joke: they behave well, work under coding agents, call tools in a reliable way. The 2 bit quants use a very asymmetrical quantization: only the routed MoE experts are quantized, up/gate at IQ2_XXS , down at Q2_K . They are the majority of all the model space: the other components (shared experts, projections, routing) are left untouched to guarantee quality.
I’m using DeepSeek V4 Flash on Spark. It’s very accurate, but it’s also extremely slow — in many cases, the thinking phase can take up to 20 minutes.
Qwen 3.5 models, even though they’re less accurate, usually reach a solution much faster by retrying.
Qwen 3.6 dense model without DFlash (I am having trouble to use flash) is slow too, but still consistently faster than DeepSeek V4 Flash.
Think I’ve used it with 500k at the most. I kind of switched to antirez/ds4 some days ago.
Planning to see if I can somehow get this working across two nodes in a few weeks, got another GB10 on the way.
Thanks, this is what also I achieve meanwhile, based on the antirez/ds4 Repository.
I now published my approach to build and run all the binaries based on a CUDA devel container to avoid the need of installing the toolchains entirely on the host.
Another important finding was, to use the model “mapped” (not copied) to the unified memory and “warmed”. The price tag is a slight degradation of the token decode speed to ~ 13.6 tok/s but allowing to use the entire 1M token context on a single Spark.
You find that repo here: GitHub - hhackbarth/ds4-ops · GitHub
It uses the antirez/ds4 Repo as sub-module.
Any chance we’ll see the updates in the repo soon? :)
I was playing with the current iteration of github, but kept hitting illegal memory access errors with sufficiently large contexts (100_000+). antirez repo is working reliably at 256_000, but would love to see that speed increase
I’ve been working on this a little. I’m trying out swapping the Q4_K and Q8_K with MXFP4 and MXFP8 that are native to the stock weights. It also lets me use native CUTLASS to fully leverage the tensor cores. So far it’s looking really good. Prefill rates are way up, but I’ve not gotten started on decode yet. Everything I’m doing completely breaks compatibility with ROCm and metal, so my fork will be CUDA only…
