Any changes that this model run in a Single Spark?
Anyone its in the path of trying this:
GLM 5.1 on Hugging Face... Is this model going to run on a Single Spark? How many will be necessary?
Yes, it should run on Spark, but certainly not on one. With native FP8 quants 8 Sparks are required. The cluster can drop to 4 with 4bits quants. Z.ai has published recipes for the most common serving engines. With vLLM you will need at least 0.19.0 alongside transformers >= 5.4.0.
It’s going to require at least 2 Sparks.
It will be slow because it’s dense 744B of parameters, but only uses 40B of active parameters ; let’s see if the Minimax 2.7 guys sign up.
Ok.. so its seems that the SOTA standard will require at least two sparks in the near future.
Since GGUF is currently being supported experimentally in vLLM, you can consider it impossible unless you have 8 sparks.
For everyone willing to adventure in 2 bits quants (such as UD-IQ2_M), I would recommend running appropriate capability benchmarks, because things won’t all be rosy as in the original model cards. There are times when we can serve models, to a cost.
The Spark can run llama.cpp.
This was to be expected, in fact even 4 might not be enough for many models. If you want to run Kimi2.5 your only option on 4 Sparks is a version with some of the experts gutted and a very low quant. Going forward, true SOTA models you’re going to need 8 Sparks because that’s the next jump up after 4 for tensor parallelism. I think the better play might end up being a couple 512GB M5 Studios once they launch (or four 256GB if they don’t offer the 512GB due to memory shortages, which would also give you more cores to work with anyway).
Spark can run gguf using llama.cpp, but since llama.cpp does not officially support distributed processing, 128GB unified memory is woefully insufficient to run GLM 5.1. That is what I intended to say.
Ah right, forgot about that. I don’t use llama.cpp so I forget what it does and doesn’t support.
I think it could work even on a 2-node Spark cluster if someone quantizes it using the w2a16 method with intel/auto-round.
I expect the model size would be around 200GB if we use the w2a16 method with AutoRound.
Looks like someone built an NVFP4 quant.
Unfortunately it looks like it’s about 490GB in weights alone (rough math) which means it’s going to take 8 Sparks to run that. I did try to make my own AWQ or GPTQ over the past day but even with Claude’s help it was a no-go even trying to use disk caching with 96GB+96GB VRAM+RAM and I wasn’t about to spend 300 bucks on a RunPod with 2TB RAM just to experiment with creating one that would fit on 4 Sparks (Claude and Gemini both said it would come in around 380GB for weights).
With GLM5 (haven’t tried 5.1) - llama.cpp definitely worked fine RPC distributed across two Sparks from a functionality perspective in IQ2-XXS GGUF with a 4 bit quantized KV cache. That is a pretty heavy quant though so expect degradation.
GLM5 (so I am guessing 5.1) works fine on 4 Sparks at at least 64K context in NVFP4 in vLLM once you get the flags right. Might be able to push to 128K, still testing that.
Someone also pushed an AWQ quant. I was going to try to test that one today, was already working on a recipe file. The only NVFP4 quants that I saw looked too big to run on 4 Sparks.
The official NVIDIA GLM5 NVFP4 works fine on 4 sparks! The size you see on the repo includes the 20GB of optional MTP weights - which are off by default anyways. So it’s closer to 460GB!
When loaded I see about ~115GB per node for weights (which tracks, it varies slightly by node), need like 5GB for a float8 KV cache of decent size, the rest of the memory ends up mostly going to other buffers, OS, etc.
So it pushes right up against the limits, but it definitely works! (You’ll need to patch out the sparse indexer stuff, as that breaks on SM120, do the usual Spark optimizations like drop the caches and disable multi user mode, etc)
I have it running right now. I’m hoping official 5.1 NVFP4 weights come soon!
If it helps, this is what I did for GLM5. Unsure if it will work for 5.1 yet (this is with a Microtik QSFP switch):
GLOO_SOCKET_IFNAME=enp1s0f1np1 NCCL_SOCKET_IFNAME=enp1s0f1np1 NCCL_IB_DISABLE=0 NCCL_IB_HCA=rocep1s0f1 NCCL_IGNORE_CPU_AFFINITY=1 NCCL_DEBUG=INFO PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_MOE_FP4=0 VLLM_ENABLE_V1_MULTIPROCESSING=0 VLLM_HOST_IP=192.168.200.4 FLASHINFER_CUDA_ARCH_LIST=12.1a TORCH_CUDA_ARCH_LIST=12.1a TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas HF_HUB_OFFLINE=1 VLLM_CUTLASS_AUTOTUNING_DISABLED=1 vllm serve nvidia/GLM-5-NVFP4 --quantization modelopt_fp4 --trust-remote-code --max-model-len 100000 --gpu-memory-utilization 0.98 --kv-cache-dtype fp8_e4m3 -tp 4 --distributed-executor-backend mp --port 8000 --host 0.0.0.0 --enforce-eager --served-model-name glm5 --max-num-seqs 1 --max-num-batched-tokens 128 --kv-cache-memory-bytes 4600000000 --no-enable-flashinfer-autotune --enable-prefix-caching --nnodes 4 --node-rank 0 --master-addr 192.168.200.4 --master-port 29501
This is my most recent working command. It may be a little unstable as I pushed it to like 100K context, 64K with a smaller KV is probably more stable and I’ve tested more.
You need transformers 5.4 - Change VLLM_HOST_IP and --node-rank per node. Workers add --headless.
The code patches I needed:
Patch 1 — deepseek_v2.py is_v32 gating:
# BEFORE:
self.is_v32 = hasattr(config, “index_topk”)
# AFTER:
self.is_v32 = getattr(config, “index_topk”, None) not in (None, 0)
Patch 2 — deepseek_v2.py indexer weight skip:
# ADD before every `param = params_dict[name` line:
if “indexer” in name and name not in params_dict:
continue
Patch 3 — vllm/v1/worker/utils.py memory validation bypass:
# BEFORE:
if init_snapshot.free_memory < requested_memory:
raise ValueError(...)
# AFTER:
if init_snapshot.free_memory < requested_memory:
import logging as \_lg
\_lg.getLogger(\__name_\_).warning(
"DGX Spark unified mem bypass: free=%s req=%s, continuing",
init_snapshot.free_memory, requested_memory)
Model config.json**:**
“index_topk”: 0,
“num_nextn_predict_layers”: 1
// delete “indexer_rope_interleave” key
I’ve been fighting most of the day trying to get the AWQ version of this model going. Now I’m facing an error that I have no idea what to do about. It’s weird because any digging I’ve done implies this is a DeepSeek error and nothing has been GLM-related.
(EngineCore pid=1146) ERROR 04-12 20:48:23 [core.py:1108] RuntimeError: Sparse Attention Indexer CUDA op requires DeepGEMM to be installed.
That looks like the DeepSeek Sparse Attention indexer I mentioned having to patch out. It’s pretty broken on Spark. The patches above are what I needed to fix that!
I did try getting it working over the course of a couple days, but ultimately every sparse indexer solution I came up with was slower than just disabling it with the patches above.
What kind of performance are you getting with this setup?