Can I concurrently run both oss-20B and oss-120B? I’m currently utilizing TensorRT for simultaneous execution, but I’m experiencing significant performance degradation.
You would have to launch two instances of vllm on different ports. Would that work for you?
You can run two separate instances of vLLM on different ports, like @PrinceHal suggested, just make sure you set --gpu-memory-utilization accordingly, so both take less than 0.9 of total VRAM (e.g. 0.2+0.7).
If you want to access both on a single port, or launch/unload on demand, you can try llama-swap or it’s fork llmsnap which has some vLLM additions.
Not sure if it would be helpful for you, but you could give this a shot for improving performance:
Hey, can you please update with the instruction to run on vllm with spark GB10(in virtual env)?
currently I am hitting this error after when sending the curl request:
(EngineCore pid=419501) out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
(EngineCore pid=419501) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=419501) File “/home/lkesem/.venv/lib/python3.12/site-packages/torch/_ops.py”, line 1269, in _call_
(EngineCore pid=419501) return self._op(*args, **kwargs)
(EngineCore pid=419501) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=419501) torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore pid=419501) Search for
cudaErrorUnsupportedPtxVersion' in  for more information.
I am using those packages:
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
PyTorch version : 2.11.0+cu130
i build it with
TORCH_CUDA_ARCH_LIST=“12.0;12.1” MAX_JOBS=12 pip install -vv --no-build-isolation -e .
i used the recent vllm – which means that the cmake change apply(the patch)
i used python use_existing_torch.py
without pointing to PTX.
and run into the same error building with:
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
At this point, I’d suggest to just use our community Docker: GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub
It is battle tested, has a CI pipeline that generates nightly wheels and runs the resulting image through a series of regression tests. We keep a close eye on the most recent developments and sometimes include PRs that are not
It’s also used by other QOL projects, like sparkrun.
Why stop at 2 or 3 nodes? :)
we have not stopped at 2 nodes, there’s a 3-nodes mesh guide:
Fixed :) ( 3 < # )
4? 5? 6? 7? 8?
Don’t worry. We can keep going ;-)
(sparkrun, community docker, they do it!)
Yes!
Though I would not recommend anything but 1, 2, 4, and 8. Some model architectures just fail with non-power of twos cluster. At least for me running MiniMax-M2.7 (full precision) on 4 node cluster in a 100gbe network. When I attempted to 6 nodes, it would fail using the latest VLLM image from Nvidia (26.04-py3).
Though I would like to be proven wrong.
I now (7 June 2026) get this error (cudaErrorUnsupportedPtxVersion) with the community docker.
I’m on a single DGX Spark.
I have done the following:
$ git pull
$ docker builder prune
$ ./build-and-copy.sh
$ ./run-recipe.sh qwen3.5-122b-int4-autoround --solo
@eugr How can I debug this and find out where the problem is? Are there any commands that I can run to get system information that can pinpoint the problem?
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0
$ which nvcc
/usr/local/cuda/bin/nvcc
$ ls -al /usr/local/cuda
lrwxrwxrwx 1 root root 22 Sep 13 2025 /usr/local/cuda → /etc/alternatives/cuda
$ ls -l /etc/alternatives/cuda
lrwxrwxrwx 1 root root 20 May 31 11:57 /etc/alternatives/cuda → /usr/local/cuda-13.2
LD_LIBRARY_PATH is not set. Should it be? (LD_LIBRARY_PATH=/usr/local/cuda-13.2/compat)
$ apt list --installed | grep cuda-lib
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
cuda-libraries-13-0/unknown,now 13.0.3-1 arm64 [installed,automatic]
cuda-libraries-13-2/unknown,now 13.2.1-1 arm64 [installed,automatic]
cuda-libraries-dev-13-0/unknown,now 13.0.3-1 arm64 [installed,automatic]
cuda-libraries-dev-13-2/unknown,now 13.2.1-1 arm64 [installed,automatic]
$ apt list --installed | grep cuda-compat
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
cuda-compat-13-2/unknown,now 595.71.05-1ubuntu1 arm64 [installed]
Can you post an output of nvidia-smi? I can’t reproduce on my Sparks, but I think I’ve seen that on a machine that had 595.xx driver installed.
$ nvidia-smi
Sun Jun 7 20:16:22 2026
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 |
±----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GB10 On | 0000000F:01:00.0 On | N/A |
| N/A 49C P0 11W / N/A | Not Supported | 0% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 9266 G /usr/lib/xorg/Xorg 416MiB |
| 0 N/A N/A 10427 G /usr/bin/gnome-shell 207MiB |
| 0 N/A N/A 10649 G /usr/bin/kalendarac 6MiB |
| 0 N/A N/A 11546 G …bin/akonadi_archivemail_agent 6MiB |
| 0 N/A N/A 11559 G …/bin/akonadi_mailfilter_agent 452MiB |
| 0 N/A N/A 11567 G /usr/bin/akonadi_sendlater_agent 6MiB |
| 0 N/A N/A 11568 G …/akonadi_unifiedmailbox_agent 6MiB |
| 0 N/A N/A 429777 C+G …writer/86/usr/bin/ghostwriter 272MiB |
| 0 N/A N/A 508702 C VLLM::EngineCore 10151… |
±----------------------------------------------------------------------------------------+
Interesting. Can you try to launch with --no-cache-dirs and see if that happens again?
I get the same error when I do:
./run-recipe.sh qwen3.5-122b-int4-autoround --solo --no-cache-dirs
Are there any other things that I can check?
OK, I was able to reproduce on one of my Sparks, will likely have to downgrade the base image.
For anyone here chasing fast multi-model swapping on the Spark’s unified memory — I ended up writing a small vLLM plugin for it and just open-sourced it: GitHub - dgr237/vllm-snapshot: Fast model suspend/restore for vLLM sleep mode — snapshot weights to disk and swap models on one GPU in seconds, not minutes. · GitHub .
vLLM’s sleep mode gets you most of the way (it keeps the process + CUDA graphs warm), but on GB10 a level-2 wake still has to re-run reload_weights, which was ~82s (safetensors) / ~30 min (instanttensor) for a 26B NVFP4 MoE here — and it’s a processing wall, not disk I/O, so faster loaders don’t help. The plugin snapshots the built weight regions byte-for-byte to disk and restores them with a bulk cudaMemcpy: ~1.6s restore, ~9s full swap, correct and reproducible.
It’s a native plugin (adds /suspend and /restore to vanilla vLLM’s server — no external runtime), with an autosuspend option + a Compose depends_on pattern so N models boot one-at-a-time instead of OOM-ing the box. Research prototype, only validated on GB10/SM121 so far — feedback and swap numbers from other setups very welcome.
Since this thread is where a lot of us landed while fighting vLLM on the Spark, I wrote up the timeline I’ve worked through since December in case it saves anyone the hours:
-
Field notes: the Hard Way — what broke, which images actually work today, the FP4 checkpoint-format rule, unified-memory OOM math, and measured tok/s.
-
Technical report: the Complete Guide — the kernel-level why: sm_121, CUTLASS/FlashInfer/Marlin, NVFP4/MXFP4/FP8 wiring, and how vLLM’s backend oracle picks a path.
Most of this is much less of a problem since ~May '26, but the why still helps for tuning. I’m no expert — if you spot a technical mistake in either, please tell me. Happy to learn from it.