Kindling Spark OS - ~4GB+ free RAM

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hey all,

tl;dr: for all sparks - an extra 4-8GB for your models, better decode perf, better connectx perf.

We’ve been working on some ultra-tuned GLM 5.3 Flash recipes for sparks ( GitHub - kindlingai/glm-5.3-flash-gx10: GLM-5.3-Flash (NVFP4) at TP=4 across 4x ASUS GX10 / GB10 - vLLM + mentat instead of Ray, 524k context, DFlash2. Minimal working recipe. · GitHub ) and have been investigating how to squeeze out the last bit of performance from sparks.

The result of that research is Kindling Spark OS: it’s a barebones, lightly tuned OS that bootstraps itself from the DGX Spark OS and adds itself as a boot option. We claw back about ~4GB of headroom for models, and I’m seeing up to ~10% more decode in some benchmarks over the stock OS I was running on the ASUS model hardware.

There’s two big changes: one, a 64k page kernel. This frees up 2GB on its own. The second is a small daemon named “dispram” that lets a container map 2GB from the unused framebuffer RAM directly into its KV. You’ll need a small vLLM or other LLM server patch to get that framebuffer 2GB back. And because its such a barebones install, I’m seeing a stable ~8GB (!) of headroom that goes fully to KV in the GLM 5.3 Flash recipe.

We’ve also been seeing better ConnectX perf in some synthetic benchmarks, but the reason for that isn’t clear (maybe the kernel?). Hopefully we’ll be able to track that down, otherwise it’s just a “nice bonus”.

This is compatible with all the sparks - same image works on the Nvidia founder’s edition, Asus GX-10, etc.

Most containers will be compatible as long as they don’t try to use mmap to load their weights from safetensors files. There’s a bug I need to file in the nvidia kernel driver repo about this: most vllm recipes just need to use the “eager” safetensors loading strategy.

The GLM 5.3 Flash repo from kindlingai has support baked in to get the extra 2GB of KV, but that’s the only one so far. It should be easy enough for a recipe to add optional support for it!

3

Some more details - this works with all the models that I was serving before. I did have to update a bunch of them specifically to add --safetensors-load-strategy eager

The 2GB of framebuffer RAM is super easy to integrate into any container. We’ve turned that into a little helper library ( kindling-spark-os/dispram at main · kindlingai/kindling-spark-os · GitHub ) that takes a few minutes to integrate. By default it stuffs the extra 2GB into your KV, since that’s usually the most useful place to put it.

You don’t lose access to the DGX OS install - enter-dgx-os puts you into the old system where most of the tools still work, or you can even reboot back into the old OS.

4

I’m following along! Thank you for your effort and contribution!

5

@matthew16 Thank you for sharing this! I’m excited to test it out. Out of curiosity, do you happen to know how much performance can be contributed to this aspect?

  • FP8 and NVFP4 copies of the dense layers the checkpoint leaves in bf16. This is the only change with a measured accuracy cost: about 1% NLL on prose.

The reason I’m asking is because my preference would be to run the original NVIDIA NVFP4 quant without further alterations, so comparing that to my current vLLM setup is probably the key benchmark for actual utility.

6

Following up on my own question, I have come to the conclusion that the bulk of the performance gains when comparing a similar setup (e.g., MTP with eugr to this version with MTP; and a similar comparison using DFlash2) are due to the additional quantization. Just FYI for anyone else interested.