Kimi K2.5 viable on a DGX cluster?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I’m looking into the viability of running Kimi K2.5 locally and considering a DGX Spark cluster to do it. Would 4 DGX Sparks be able to run this model and, if so, what quant would work? It looks like 2-bit is about 375GB for the model alone, which looking at a 512GB memory pool I’m thinking the math adds up for it to work. Also, if it does work, what kind of TPS would I probably be seeing? I ask this because just because it loads doesn’t necessarily mean it will run at a usable speed. Thanks for any input and advice!

2

I tried but so far it did not work. (on 8x node)

3

At 2bits quantisation it may work on 4 only Sparks, memory wise. Yet, this is a bit aggressive quant, and makes me wonder whether the advantage offered by Kimi K2.5 over alternative models would be lost at such level of quantisation.

For comparison, you could run Qwen3.5-397B-A17B at 4bits on a cluster of 2 Sparks. Also, running Kimi K2.5 at 4bits quants would require a cluster of 8 Sparks, because 4bits quants would not fit a 4 units cluster, and, in order to leverage tensor parallel performance, the next level of parallelism would be 8 units cluster.

4

Hmm, that doesn’t sound encouraging considering 4 would be my cap haha.

5

Yeah I know I wouldn’t be able to run it at 4-bit, which also has me wondering how much I’m dumbing the model down going down to 2-bit and whether that would kill any advantage of using the model, or in fact even make it worse than a smaller model, which is I think the point you were getting at. Right now I have two RTX 6000 (fully separate machines), so I was just starting to look into the viability of running something larger than ~120B.

6

Can you share your experience on Qwen3.5-397B on 4x or 8x node setup?

7

I would be more than happy to share it, unfortunately I don’t have so many Sparks at my disposal. The reflection above resulted from a quick “back of envelope” calculation. At 4bits quants Qwen3.5-397B runs on 2 Sparks already.

8

I’ve got a two-node setup and it runs quite well, actually, over 20 t/s generation with the Intel autoround 4-bit quant

9

What context can you get out of it?

10

I think you can get the max if you set gpu memo utilization to 90%, use kv cache fp8 and have very little else running on the boxes. There is another thread here with some actual benchmarks.

11

This will begrudgingly fit on a 4x DGX Spark cluster: QuantTrio/Kimi-K2.5-E304.

Just know that you must mod a fix for TF5 to get Kimi-K2.5 to work… this is how I did it; just make sure you update the vars (SNAP_DIR, SPARK_NODES, and SPARK_USER):

SNAP_DIR="$HOME/.cache/huggingface/hub/models--QuantTrio--Kimi-K2.5-E304/snapshots"
REV="$(ls -1t "$SNAP_DIR" | head -n 1)"
echo "REV=$REV"

for ip in ${SPARK_NODES//,/ }; do
  echo "== Fast patch on ${ip} =="

  ssh -o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=10 -o ServerAliveCountMax=2 \
    "$SPARK_USER@$ip" \
    "docker exec vllm_node bash -lc 'set -euo pipefail
REV=\"$REV\"

F_MOD=\"/root/.cache/huggingface/modules/transformers_modules/_\${REV}/tokenization_kimi.py\"
F_SNAP=\"/root/.cache/huggingface/hub/models--QuantTrio--Kimi-K2.5-E304/snapshots/\${REV}/tokenization_kimi.py\"

patched=0
for f in \"\$F_MOD\" \"\$F_SNAP\"; do
  [[ -f \"\$f\" ]] || continue

  if grep -q \"from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode\" \"\$f\"; then
    echo \"[patch] patching \$f\"
    sed -i \"s|from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode|from transformers.convert_slow_tokenizer import bytes_to_unicode|g\" \"\$f\"
    patched=\$((patched+1))
  fi
done

# Verify only the same two files
for f in \"\$F_MOD\" \"\$F_SNAP\"; do
  [[ -f \"\$f\" ]] || continue
  if grep -q \"tokenization_gpt2 import bytes_to_unicode\" \"\$f\"; then
    echo \"[patch] ERROR: old import still present in \$f\" >&2
    exit 1
  fi
done

echo \"[patch] OK patched_files=\$patched\"'"
done

Serving commands:

./launch-cluster.sh \
  --nodes "$SPARK_NODES" \
  --name vllm_node \
  --eth-if "$FABRIC_IF" \
  --ib-if "$IB_IF" \
  -t "vllm-node-tf5" \
  vllm serve "$HF_MODEL" \
    --revision "$REV" \
    --served-model-name "$HF_MODEL" \
    --distributed-executor-backend ray \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.6 \
    --max-model-len auto \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 4 \
    --mm-encoder-tp-mode data \
    --enable-auto-tool-choice \
    --max-parallel-loading-workers 1 \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
	--trust-remote-code \
    --host 0.0.0.0 \
    --port 8000

Just warning you that it is a slow load and you must not have extra processes running.

12

thanks for that, can you share tok/s?