Model-Free NIM Container vs. Eugr's vLLM Spark Cluster for Large Models on GB10 Clusters?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

I’m evaluating inference stack options for running larger LLMs across a two-node/ three-node/four-node GB10 / DGX Spark cluster setup and was curious on real-world experiences.

I’m comparing two primary deployment approaches:

  1. NVIDIA Model-Free NIM Container (model-free-nim / vLLM runtime backend /SGLang runtime backend)

  2. Eugr’s spark-vllm-docker setup (eugr/spark-vllm-cluster)

For those running large models in tensor-parallel or multi-node cluster configurations on GB10:

  • Interconnect & Bandwidth: How does the stock Model-Free NIM container perform over dual ConnectX / RoCE links compared to Eugr’s custom NCCL (NCCL_IB_HCA) and InstantTensor optimizations? Any issues with dual-link link aggregation or mesh scaling?

  • Deployment: Have you found Model-Free NIM’s automated HF manifest generation to be plug-and-play for custom quantized weights on clusters, or do you still prefer the fine-grained launcher scripts and patched vLLM/FlashInfer wheels in Eugr’s repo?

  • Throughput / Latency: Are there noticeable differences in token generation speeds or prefill latency between the two types of cluster stacks when running 70B+ / MoE architectures across 2+ nodes? Such as Super120B/ Mistral 3 128B?

Would love to hear what stack you’re currently using in production or benchmarking, and any caveats/trade-offs you’ve run into with either path!