This is invalid. I can run models using vLLM on docker with high-utilization. I suggest you start using llama.cpp over ollama, it’s much more performant.
Sample with vLLM: Running nvidia/Nemotron-Nano-VL-12B-V2-NVFP4-QAD on your spark
Performance numbers, using containers with high-utilization with 1 or 2 sparks:
With properly setup dual Spark cluster you can expect almost 2x performance gain for dense models (slow ones) and less gain for sparse ones. Prompt processing performance scales better than inference.
Here is a compilation of my results - some of these need retesting as I was running with old config, but you can get an idea. That’s using VLLM and two Sparks connected via a single QSFP112 cable.
Model name
Cluster (t/s)
Single (t/s)
Comment
Qwen/Qwen3-VL-32B-Instruct-FP8
12.00
7.00
c…