Hi,
What is the best engine (vllm, sglang or trt-llm) when running several <40B models (eg finetuned GPT-OSS 20B) in parallell on one or two Sparks for an agentic setup?
/magnus
I’d say llama.cpp, as it is the most memory efficient and not overly eager to occupy as much memory as possible like other choices. vLLM will have better concurrency and throughput, but the only way to limit its memory use is through --gpu-memory-utilization flag which takes a percentage of total VRAM available which is not a constant on Spark. I actually run a small vision model and an embedding model on one of my Sparks all the time using llama.cpp, and use vLLM for larger models that I spread across two sparks.
@eugr, I have seen several discussions here on the best container compilation of llama.cpp for Spark, including ones related to MXFP4 . Since llama.cpp pushes out several versions per day, which setup would you suggest to go for? Especially as there are infinite compilation parameters which can be tuned, and one bad setting can ruin the performance.
/magnus
You don’t really need a container to use llama.cpp - it compiles into a single binary (well, several, but you only need llama-server to run models) and doesn’t change anything in the system - why pull additional dependencies?
As for the compilation flags, up until now you just needed -DGGML_CUDA=ON, but now it also makes sense to specify the arch to get blackwell optimizations in:
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real
cmake --build build --config Release -j
Here I also included RPC backend to use with two Sparks, but if you just have one, no need for that.
Most of the system dependencies are pre-installed by default, you may need to install these before building:
sudo apt install clang cmake libcurl4-openssl-dev
@eugr … this is new, maybe this helps also for running parallel models?
- Concurrency for QKV projections: Support for running concurrent CUDA streams to speed up model inference. To use this feature, pass in the –CUDA_GRAPH_OPT=1 flag.
Not really, the way I understand it, it just enables parallel execution of some matrix ops during inference, speeding up some requests within a single llama.cpp instance.