# zai-org/GLM5.3 on x8 Spark : 23 tok/s decode, 64K x C4, tool-eval-bench 95 using eugr's repo

**URL:** <https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447>\
**Category:** DGX Spark / GB10\
**Created:** [October 9, 2026, 3:54pm UTC](https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447 "2026-10-09T15:54:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![hypermac.6502](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hypermac.6502/32/513319_2.png) [@hypermac.6502](https://forums.developer.nvidia.com/u/hypermac.6502)\
**Post date:** [October 9, 2026, 3:54pm UTC](https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447/1 "2026-10-09T15:54:26Z")

</div>

This is probably a pretty rare setup, so I’m not sure how useful this will be for everyone, but here’s an example of running inference with the original `zai-org/GLM-5.3` model (756 GiB) as-is, without additional quantization.

Model: zai-org/GLM5.3  
vLLM: Eugr’s spark-vllm-docker + mod  
DGX Spark: x8 (TP=8)  
tool-eval-bench: v2.7.1

Even with 8× DGX Sparks, I can only get about 11.0 GiB of KV cache per node, so the practical context size is limited:

```auto
GPU KV cache size: 218,112 tokens
Maximum concurrency for 65,536 tokens per request: 3.33x

```

But the most important point for me is this:

**`zai-org/GLM-5.3` can actually run in a private home setup.**

That alone is pretty amazing.

Huge thanks to NVIDIA for making the DGX Spark possible, to [@eugr\_nv](https://forums.developer.nvidia.com/u/eugr_nv) for making `spark-vllm-docker` available, and to everyone in this forum who shares information and contributes to the discussions.

This community has made setups like this possible. 🔥

Thank you!

 ![Screenshot from 2026-10-10 00-16-55](https://global.discourse-cdn.com/nvidia/original/4X/0/e/7/0e7cb7779a3cd9f10208a9955372616b76f38f05.png)

 ![Screenshot from 2026-10-10 00-17-13](https://global.discourse-cdn.com/nvidia/original/4X/b/1/e/b1e1d13c0f9995e23511ea663244341ad452042b.jpeg)

 ![Screenshot from 2026-10-10 00-18-18](https://global.discourse-cdn.com/nvidia/original/4X/7/2/4/724fd61d5c5ed3f09e0fb56810ac251b6e812958.jpeg)

 ![Screenshot from 2026-10-10 00-18-34](https://global.discourse-cdn.com/nvidia/original/4X/d/7/b/d7b3f1600489e804a03a226bb72f0bdb4344b536.png)

---

<div class="post-metadata">

**Author:** ![sjug](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sjug](https://forums.developer.nvidia.com/u/sjug)\
**Post date:** [October 9, 2026, 3:57pm UTC](https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447/2 "2026-10-09T15:57:16Z")

</div>

It’s not quite **`zai-org/GLM-5.3`** , it’s the NVFP4 quant of it.  
I doubt it’s as rare as you may think.

Keep on optimizing and get those prefill numbers up!

---

<div class="post-metadata">

**Author:** ![hypermac.6502](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hypermac.6502/32/513319_2.png) [@hypermac.6502](https://forums.developer.nvidia.com/u/hypermac.6502)\
**Post date:** [October 9, 2026, 4:00pm UTC](https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447/3 "2026-10-09T16:00:08Z")

</div>

NVFP4??

I downloaded this one.

> **[zai-org/GLM-5.3 · Hugging Face](https://huggingface.co/zai-org/GLM-5.3)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![sjug](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sjug](https://forums.developer.nvidia.com/u/sjug)\
**Post date:** [October 9, 2026, 4:03pm UTC](https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447/4 "2026-10-09T16:03:56Z")

</div>

Ah I see, that’s why the KV cache is so low, got it.  
Post up your recipe.

---

<div class="post-metadata">

**Author:** ![hypermac.6502](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hypermac.6502/32/513319_2.png) [@hypermac.6502](https://forums.developer.nvidia.com/u/hypermac.6502)\
**Post date:** [October 9, 2026, 4:06pm UTC](https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447/5 "2026-10-09T16:06:35Z")

</div>

this!

```auto
# Recipe: zai-org/GLM-5.3
# GLM 5.3 using the B12X-optimized serving stack

recipe_version: "1"
name: zai-org/GLM-5.3
description: vLLM serving zai-org/GLM-5.3

model: zai-org/GLM-5.3
#container: vllm-node-b12x
container: vllm-node-b12x:n128fix2

build_args:
  - --exp-b12x

cluster_only: true

mods:
 - mods/glm53-fix-chat-template

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 8
  pipeline_parallel: 1
  decode_context_parallel: 1
  block_size: 256
  max_model_len: 65536
  max_num_seqs: 4
  max_num_batched_tokens: 4096
  num_speculative_tokens: 3
  max_cudagraph_capture_size: 16
  gpu_memory_utilization: 0.88

env:
  CUTE_DSL_ARCH: "sm_121a"
  VLLM_ENABLE_ROCE_ALLREDUCE: "1"
  VLLM_ENABLE_PCIE_ALLREDUCE: "0"
  VLLM_ROCE_ALLREDUCE_MAX_SIZE: "2MB"
  VLLM_WORKER_MULTIPROC_METHOD: "spawn"
  VLLM_USE_BREAKABLE_CUDAGRAPH: "1"
  B12X_POLICY_MODE: "auto"

command: |
  vllm serve zai-org/GLM-5.3 \
    --host {host} \
    --port {port} \
    --pipeline-parallel-size {pipeline_parallel} \
    --decode-context-parallel-size {decode_context_parallel} \
    --mamba-cache-mode align \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --block-size {block_size} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --tensor-parallel-size {tensor_parallel} \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --async-scheduling \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --load-format b12x \
    --attention-backend B12X \
    --trust-remote-code \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --max-cudagraph-capture-size {max_cudagraph_capture_size} \
    --compilation-config '{{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}}' \
    --override-generation-config '{{"temperature": 1.0, "top_p": 0.95, "top_k": 20, "presence_penalty" : 0.0, "repetition_penalty" : 1.0, "min_p": 0.0}}' \
    --default-chat-template-kwargs '{{"thinking":true,"reasoning_effort":"max"}}'

```

---

<div class="post-metadata">

**Author:** ![hypermac.6502](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hypermac.6502/32/513319_2.png) [@hypermac.6502](https://forums.developer.nvidia.com/u/hypermac.6502)\
**Post date:** [October 9, 2026, 7:05pm UTC](https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447/6 "2026-10-09T19:05:19Z")

</div>

This one is the nvidia/GLM-5.3-NVFP4 weights version.

Model: nvidia/GLM-5.3-NVFP4  
vLLM: Eugr’s spark-vllm-docker + mod  
DGX Spark: x8 (TP=8)  
tool-eval-bench: v2.7.1

```auto
GPU KV cache size: 446,208 tokens,
Maximum concurrency for 262,144 tokens per request: 1.70x

```

 ![Screenshot from 2026-10-08 04-05-43](https://global.discourse-cdn.com/nvidia/original/4X/b/c/0/bc041c3c373157bec7b708264064924ed30cc3c3.png)

 ![Screenshot from 2026-10-08 04-06-01](https://global.discourse-cdn.com/nvidia/original/4X/4/c/4/4c489e4d4009662f8e0457083e106a83bb80fea2.jpeg)

 ![Screenshot from 2026-10-08 04-06-50](https://global.discourse-cdn.com/nvidia/original/4X/3/7/e/37e5652e439aa5480f87865a76edc4c3eadc4375.jpeg)

 ![Screenshot from 2026-10-08 04-07-04](https://global.discourse-cdn.com/nvidia/original/4X/0/d/e/0de81ad94903f20cdd25147248b39336c6b95a06.png)

---

<div class="post-metadata">

**Author:** ![jetspark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jetspark/32/542210_2.png) [@jetspark](https://forums.developer.nvidia.com/u/jetspark)\
**Post date:** [October 9, 2026, 7:25pm UTC](https://forums.developer.nvidia.com/t/zai-org-glm5-3-on-x8-spark-23-tok-s-decode-64k-x-c4-tool-eval-bench-95-using-eugrs-repo/385447/7 "2026-10-09T19:25:48Z")

</div>

I’m waiting for GLM-5.5 to see the TC-68 finally PASS