The official NVIDIA GLM5 NVFP4 works fine on 4 sparks! The size you see on the repo includes the 20GB of optional MTP weights - which are off by default anyways. So it’s closer to 460GB!
When loaded I see about ~115GB per node for weights (which tracks, it varies slightly by node), need like 5GB for a float8 KV cache of decent size, the rest of the memory ends up mostly going to other buffers, OS, etc.
So it pushes right up against the limits, but it definitely works! (You’ll need to patch out the sparse indexer stuff, as that breaks on SM120, do the usual Spark optimizations like drop the caches and disable multi user mode, etc)
I have it running right now. I’m hoping official 5.1 NVFP4 weights come soon!
If it helps, this is what I did for GLM5. Unsure if it will work for 5.1 yet (this is with a Microtik QSFP switch):
GLOO_SOCKET_IFNAME=enp1s0f1np1 NCCL_SOCKET_IFNAME=enp1s0f1np1 NCCL_IB_DISABLE=0 NCCL_IB_HCA=rocep1s0f1 NCCL_IGNORE_CPU_AFFINITY=1 NCCL_DEBUG=INFO PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_MOE_FP4=0 VLLM_ENABLE_V1_MULTIPROCESSING=0 VLLM_HOST_IP=192.168.200.4 FLASHINFER_CUDA_ARCH_LIST=12.1a TORCH_CUDA_ARCH_LIST=12.1a TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas HF_HUB_OFFLINE=1 VLLM_CUTLASS_AUTOTUNING_DISABLED=1 vllm serve nvidia/GLM-5-NVFP4 --quantization modelopt_fp4 --trust-remote-code --max-model-len 100000 --gpu-memory-utilization 0.98 --kv-cache-dtype fp8_e4m3 -tp 4 --distributed-executor-backend mp --port 8000 --host 0.0.0.0 --enforce-eager --served-model-name glm5 --max-num-seqs 1 --max-num-batched-tokens 128 --kv-cache-memory-bytes 4600000000 --no-enable-flashinfer-autotune --enable-prefix-caching --nnodes 4 --node-rank 0 --master-addr 192.168.200.4 --master-port 29501
This is my most recent working command. It may be a little unstable as I pushed it to like 100K context, 64K with a smaller KV is probably more stable and I’ve tested more.
You need transformers 5.4 - Change VLLM_HOST_IP and --node-rank per node. Workers add --headless.
The code patches I needed:
Patch 1 — deepseek_v2.py is_v32 gating:
# BEFORE:
self.is_v32 = hasattr(config, “index_topk”)
# AFTER:
self.is_v32 = getattr(config, “index_topk”, None) not in (None, 0)
Patch 2 — deepseek_v2.py indexer weight skip:
# ADD before every `param = params_dict[name` line:
if “indexer” in name and name not in params_dict:
continue
Patch 3 — vllm/v1/worker/utils.py memory validation bypass:
# BEFORE:
if init_snapshot.free_memory < requested_memory:
raise ValueError(...)
# AFTER:
if init_snapshot.free_memory < requested_memory:
import logging as \_lg
\_lg.getLogger(\__name_\_).warning(
"DGX Spark unified mem bypass: free=%s req=%s, continuing",
init_snapshot.free_memory, requested_memory)
Model config.json**:**
“index_topk”: 0,
“num_nextn_predict_layers”: 1
// delete “indexer_rope_interleave” key