Hello, I’ve been up all night to figure this out. It seems vLLM is perfect for DGX Spark, the only problem is that a 14b models take up to 100gb of ram using vLLM and the performance are not great as advertised, I’ve followed the tutorial from the official website. I’m currently using llama.cpp RP…

A few things: llama.cpp RPC doesn’t support RDMA - you’ll get a lot of performance loss due to latency. llama.cpp doesn’t do tensor parallel, so you will not gain any performance in the cluster, no matter how low latency the link is. You need to use vLLM to gain from the cluster setup. On dense mo…

@eugr - quick random question. Do you see any perceivable difference in coherence between the NVFP4 and AWQ versions of Qwen3 235b? Presumably NVFP4 offers less loss?

NVFP4 if it’s W4A4 (as I believe that quant is - I deleted it as I prefer AWQ for now) has more loss as it quantizes activation weights to 4-bit too, while AWQ keeps activation weights in BF16. Now, NVFP4 W4A16 theoretically would be superior to AWQ quants as FP4 carries more precision than INT4, b…

Thanks – swinging by Microcenter to pick up that second Spark today and thinking about what big model I’m going to load up first. :-)

Distributed Inference - 200gb/s with bottleneck, am I missing something?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10

aniculescu January 21, 2026, 4:22pm 3

Please check out our playbooks on how to use different workloads on the Spark: Spark Playbooks
You can also look over community projects on the forums: DGX Spark / GB10 Projects - NVIDIA Developer Forums
There are also a lot of threads here about how to get better performance if you specifically want to use vLLM: Run VLLM in Spark

Topic		Replies	Views
DGX Spark Multi-Node LLM Inference Report for Qwen3-235B model DGX Spark / GB10 nim , llama	34	2879	May 1, 2026
New bleeding-edge vLLM Docker Image: avarok/vllm-nvfp4-gb10-sm120 DGX Spark / GB10 Projects	32	3401	December 17, 2025
Two-Spark cluster with vLLM using tensor-parallel-size 2 causes one node to drop while the other's GPU goes 100% forever DGX Spark / GB10	36	2128	February 13, 2026
Install and Use vLLM for Inference on two Sparks does not work DGX Spark / GB10	159	5983	December 9, 2025
Help: Running NVFP4 model on 2x DGX Spark with vLLM + Ray (multi-node) DGX Spark / GB10 mistral-large	18	2837	December 25, 2025
Can someone please just help me set the DGX Spark up for optimal LLM use? DGX Spark / GB10 llama	11	1862	June 20, 2026
Who wants to be the hero and help a total newbie! Got a spark and um, yeah DGX Spark / GB10 nemotron	7	717	April 3, 2026
Vllm docker-compose - on DGX Spark from first time user looking for suggestions and question about RAM utilization DGX Spark / GB10 docker	5	2017	December 10, 2025
TensorRT-LLM + nvidia/Llama-3.3-70B-Instruct-NVFP4 = 5 tok/s DGX Spark / GB10 llama	3	758	January 18, 2026
Investigating Performance Issue/Bottleneck DGX Spark / GB10 llama , agentic-ai	9	952	February 1, 2026

Distributed Inference - 200gb/s with bottleneck, am I missing something?

Related topics