zai-org/GLM5.3 on x8 Spark : 23 tok/s decode, 64K x C4, tool-eval-bench 95 using eugr's repo

[hypermac.6502]

This is probably a pretty rare setup, so I’m not sure how useful this will be for everyone, but here’s an example of running inference with the original zai-org/GLM-5.3 model (756 GiB) as-is, without additional quantization.

Model: zai-org/GLM5.3
vLLM: Eugr’s spark-vllm-docker + mod
DGX Spark: x8 (TP=8)
tool-eval-bench: v2.7.1

Even with 8× DGX Sparks, I can only get about 11.0 GiB of KV cache per node, so the practical context size is limited:

GPU KV cache size: 218,112 tokens
Maximum concurrency for 65,536 tokens per request: 3.33x

But the most important point for me is this:

zai-org/GLM-5.3 can actually run in a private home setup.

That alone is pretty amazing.

Huge thanks to NVIDIA for making the DGX Spark possible, to @eugr_nv for making spark-vllm-docker available, and to everyone in this forum who shares information and contributes to the discussions.

This community has made setups like this possible. 🔥

Thank you!

[sjug]

It’s not quite zai-org/GLM-5.3, it’s the NVFP4 quant of it.
I doubt it’s as rare as you may think.

Keep on optimizing and get those prefill numbers up!

[hypermac.6502]

NVFP4??

I downloaded this one.

zai-org/GLM-5.3 · Hugging Face

[sjug]

Ah I see, that’s why the KV cache is so low, got it.
Post up your recipe.

[hypermac.6502]

this!

# Recipe: zai-org/GLM-5.3
# GLM 5.3 using the B12X-optimized serving stack

recipe_version: "1"
name: zai-org/GLM-5.3
description: vLLM serving zai-org/GLM-5.3

model: zai-org/GLM-5.3
#container: vllm-node-b12x
container: vllm-node-b12x:n128fix2

build_args:
  - --exp-b12x

cluster_only: true

mods:
 - mods/glm53-fix-chat-template

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 8
  pipeline_parallel: 1
  decode_context_parallel: 1
  block_size: 256
  max_model_len: 65536
  max_num_seqs: 4
  max_num_batched_tokens: 4096
  num_speculative_tokens: 3
  max_cudagraph_capture_size: 16
  gpu_memory_utilization: 0.88


env:
  CUTE_DSL_ARCH: "sm_121a"
  VLLM_ENABLE_ROCE_ALLREDUCE: "1"
  VLLM_ENABLE_PCIE_ALLREDUCE: "0"
  VLLM_ROCE_ALLREDUCE_MAX_SIZE: "2MB"
  VLLM_WORKER_MULTIPROC_METHOD: "spawn"
  VLLM_USE_BREAKABLE_CUDAGRAPH: "1"
  B12X_POLICY_MODE: "auto"

command: |
  vllm serve zai-org/GLM-5.3 \
    --host {host} \
    --port {port} \
    --pipeline-parallel-size {pipeline_parallel} \
    --decode-context-parallel-size {decode_context_parallel} \
    --mamba-cache-mode align \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --block-size {block_size} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --tensor-parallel-size {tensor_parallel} \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --async-scheduling \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --load-format b12x \
    --attention-backend B12X \
    --trust-remote-code \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --max-cudagraph-capture-size {max_cudagraph_capture_size} \
    --compilation-config '{{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}}' \
    --override-generation-config '{{"temperature": 1.0, "top_p": 0.95, "top_k": 20, "presence_penalty" : 0.0, "repetition_penalty" : 1.0, "min_p": 0.0}}' \
    --default-chat-template-kwargs '{{"thinking":true,"reasoning_effort":"max"}}'

[hypermac.6502]

This one is the nvidia/GLM-5.3-NVFP4 weights version.

Model: nvidia/GLM-5.3-NVFP4
vLLM: Eugr’s spark-vllm-docker + mod
DGX Spark: x8 (TP=8)
tool-eval-bench: v2.7.1

GPU KV cache size: 446,208 tokens,
Maximum concurrency for 262,144 tokens per request: 1.70x

[jetspark]

I’m waiting for GLM-5.5 to see the TC-68 finally PASS