Something to explore:
The stepfun-ai GitHub repo has more in-depth info about running on the DGX Spark, and there are two options we could experiment with which aren’t included in my above config.
From this file:
“When testing long context (e.g. 256K), OOM may occur. The build flag -DGGML_CUDA_FORCE_MMQ=ON, environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1, and runtime parameters -ctk q8_0 -ctv q8_0 can help mitigate memory issues.”
My Docker config above has the Int8 KV cache runtime flags already set, but is not using -DGGML_CUDA_FORCE_MMQ=ON in the llama.cpp build, nor the environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
From what I understand forcing MMQ is mostly something for legacy hardware, but can result in lower RAM use (at a substantial performance penalty). It would probably be worth trying this, as they clearly benched contexts up to 262k on the Spark.
I’m less convinced GGML_CUDA_ENABLE_UNIFIED_MEMORY=1will be helpful. It’s designed to overflow to system RAM if GPU VRAM is exhausted. The memory is already unified on the GB10, and I don’t think there is really a fallback - unless this allows it to thrash instead of crash.