[weight_utils.py:899] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (83.22 GiB) exceeds 90% of available RAM (29.23 GiB).
Something unrelated to vllm is eating your ram.
I show a “free -h” before and after the vllm startup. The only thing I have running is VLLM itself. Otherwise I have essentially the entire 121GB to work with before VLLM starts up. Then VLLM starts up and instantly eats 95GB. In can start up ds4 and a vllm with Qwen 122B without any issues, which eat almost the same amount.
What’s weird to me is it looks like it’s almost spawning two separate instances of VLLM. Because nvidia-smi shows one copy of it taking up 87GB VRAM, which would then make sense as to why the system thinks it only has 30GB left over for the one whose run I am seeing.
Like I have no RAM being used at all before I try executing your exact command on your huggingface page, one for one.
From my very limited experiences ,
- Do you have enough memory for the context you specified ? Have you tried starting with a smaller size ?
- Have you consider that it might possibly be a issue (compatibility) between the versions of vLLM and the plugin.
- Have you tried asking ds4 and Qwen 122B (since you can start them) to look at the logs ?
-
Again this is a headless DGX Spark with absolutely nothing else running on it. I’m SSHing into the box to execute those commands you see above. This is essentially blank slate. I have also tried starting up with 32k context and encountered the exact same issue. But I took this context directly from the Huggingface page here: rdtand/Laguna-S-2.1-prismaquant-gridbook-6bit-vllm · Hugging Face
If you look at the command in the second pastebin, that is exactly what I am executing after downloading vanilla VLLM into the container. -
That’s fine, but which VLLM should I have pip install, then? The huggingface repo just says to use vanilla vllm and doesn’t specify a version.
-
I only have one box, so I just ask ChatGPT to look at the logs. I did some investigation, but it’s similarly just as mystified as me.
The sequence is:
- Zero ram used going in (again see output of “free -h” in the second pastebin, before I start all of this)
- Start up vllm container (tried both docker eugr built and vanilla)
- OOM’s while trying to load model, after for some reason telling me I only have 30GB free before trying to load model.
nvidia-smi just shows an 87GB instance of VLLM running for some reason.
Edit: Also looking at the VLLM releases page, the current release is from 1 week ago anyway, which about lines up with when one of their stable commits in HF.
Suggest you get claude or chatgpt to debug it for you. I hear and see about a lot of issues, and this doesn’t appear to be a common one that I can give you a solution to without physically having access to your box.
As for recommended versions of vllm, I think everyone on this forum has had great experience with @eugr’s spark-centric vllm distro: GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub. He gates the release to a series of regression tests so its basically guaranteed to run nicely with spark.
I already tried using eugr’s container, as I said.
Status: Downloaded newer image for eugr/spark-vllm:latest
docker.io/eugr/spark-vllm:latest
No host specified, skipping copy.
=========================================
TIMING STATISTICS
=========================================
Prebuilt Pull: 00:04:01
Total Time: 00:04:01
=========================================
Done preparing vllm-node.
docker run -d --name vllm-poolside --gpus all --net=host --ipc=host -v ~/.cache/huggingface:/root/.cache/huggingface vllm-node bash -c '
pip install git+https://github.com/RobTand/gridbook &&
exec vllm serve rdtand/Laguna-S-2.1-prismaquant-gridbook-6bit-vllm \
--served-model-name poolsideq6 \
--port 8080 \
--enforce-eager \
--kv-cache-dtype fp8 \
--max-model-len 262144 \
--gpu-memory-utilization 0.85 \
--max-num-batched-tokens 16384'
1f9ede6096c9477b1302f439e5de6d3c91313463ba71799c2ecb43d79b051ce6
I’m really not sure what’s happening here.
What I noticed is that in Nvidia-smi, I see this VLLM instance taking up 87GB as soon as I start this:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GB10 On | 0000000F:01:00.0 Off | N/A |
| N/A 34C P0 11W / N/A | Not Supported | 1% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2914 G /usr/lib/xorg/Xorg 18MiB |
| 0 N/A N/A 3078 G /usr/bin/gnome-shell 6MiB |
| 0 N/A N/A 2893492 C VLLM::EngineCore 87446MiB |
+-----------------------------------------------------------------------------------------+
But then in “top” it looks like this:
top - 16:06:29 up 2 days, 3:23, 8 users, load average: 1.39, 1.01, 0.54
Tasks: 424 total, 3 running, 421 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.1 us, 2.0 sy, 0.0 ni, 92.9 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 124546.1 total, 1223.4 free, 121676.5 used, 2961.3 buff/cache
MiB Swap: 16384.0 total, 0.1 free, 16383.9 used. 2869.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2893492 root 20 0 220.9g 29.0g 43736 R 102.0 23.8 4:53.06 VLLM::EngineCor
And on startup the VLLM logs say that they only see 30 GB free.
I really wanted to try this out, but I guess it’s just not going to work for me. If my issues are simply that esoteric and rare then so be it. Sorry for wasting your time, have a good one.
kill -9 2893492 or reboot and try again? You may have a zombie process holding onto the ram. Beyond that it’s going to be hard to debug.
I did it from a fresh boot and the same thing happened.
Here’s an excerpt image from my chat with ChatGPT if it helps:
I also verified through nvidia-smi and ps checks (which I pasted into that chat) that the only thing actually running is a single VLLM instance. It’s just that to me it looks like it’s acting like its memory pool is split for some reason. It grabs 87GB in nvidia-smi and then has 30GB left over. This happens consistently on any VLLM I try it on, whether it’s a vanilla one or eugr’s container. I get it if you don’t have time to help debug this. Maybe I’m the only one having this issue. I tried my best with my very limited knowledge, but all I can think of is that it requires an older vllm pin, and I need to move on too. Thanks.
Look into thinkingmachines/Inkling-Small · Hugging Face , with 12B Active parameters, it should have a reasonable speed on Spark.
Following up on @m0l0 post #54: I have approved gated access and verified the repository using both the Hugging Face API and an authenticated hf download --dry-run. The only published objects on the sole main branch are .gitattributes, LICENSE, and README.md. There are currently no checkpoint shards, config.json, tokenizer, model.safetensors.index.json, cb_codebooks.pqcb, or pqserve_format_map.json. I also checked the repository’s complete commit history, tags and revisions.
The README refers to a measured 91.926 GiB artifact and says the full serving recipe was withheld pending upstream changes. I see that Gridbook v0.7/v0.8 now includes the DeepSeek-V4 contract and MXFP8 dense lane. Are the checkpoint files still intentionally withheld while serving validation continues, or are they available under another repository, revision, or storage location?
If the artifact is now runnable with the newer Gridbook releases, could you please provide the currently tested single-Spark command/container and required environment flags? Also, has DSpark speculative decoding been validated end-to-end, or are its weights currently only preserved as passthrough?
I added a bunch of stuff into gridbook to support dsv4 flash. We’ll have a version out some time tomorrow with some neat new features, then @JW2026 and I are going to have a bake-off. I’m building a smaller model now to prove out some new stuff I built today.
As an aside, we’re probably not going to need the mxfp8 path. DeepGEMM supports their own unique format of mxfp8 that is essentially 4x wider. I built that before I realized that vllm .26 supported this natively and I only had .24 install.
Thanks for the update. Looking forward to use it.
If you are refering my build, it was build pre-0.40 using measured but manual assigned bands and not using the GRIDBook allocator. A weekend build. While it should work with GRIDBOOK, and it does on my environment, it was more of a “hey, the idea works” rather than “hey, let’s use this as my daily model” . I’ve put the gate back in place and it’s probably best for you to wait a bit more for @tenari release as I don’t intend to release a seperate plugin and GEMM just for my build which is probably suboptimized.
One of the challenges is that there are soo many versions and build of vllm that just moving one components breaks another, so it serve no purpose for me to add one more to the list.
0.80 ?
I hate to give up so did try to pip install some more vllm versions going back to 24 earlier today. They all do the exact same thing. Also tried updating all of the packages on the spark to see if there was some other dependency.
Unfortunately do not have any more time to just mess around with this, so going to delete the model to free up space. I’m going back to DS4 server and/or Qwen 3.5 122B.
If anyone tries this quant and has the same issues but manages to solve them, do give me a ping. Thanks. I’ve never seen this behavior when loading anything, from DSV4F to Mistral.