While experimenting with DGX Spark, I found that running models with vLLM is fairly straightforward, but managing multiple local models becomes repetitive very quickly.
Most of my use cases require switching between different models depending on the task. For RAG, I typically need both an inference model and a prompt guard model. For coding, I use a dedicated coding model. Running all of these simultaneously is often not practical on a single Spark due to memory constraints, so I ended up maintaining a collection of Docker command lines and manually starting and stopping models as needed.
This quickly became tedious and error-prone, so I built a small LiteLLM / NVIDIA vLLM Docker orchestrator.
The goal of this project is not to provide another inference engine or orchestration platform, but rather a thin management layer around LiteLLM and vLLM that:
- defines models once in a configuration (
config.yaml+.env) - starts and stops model containers consistently
- reuses existing containers whenever possible
- provides access to logs without relying on tmux or manual Docker commands
- keeps secrets separate from model parameters
- exposes everything through a single OpenAI-compatible LiteLLM endpoint
An additional goal was consistency: the same commands, parameter names, and model aliases regardless of which model is being deployed.
On a single DGX Spark, this has proven useful for experimenting with different models and serving configurations while keeping operational overhead low. The project is essentially a convenience tool for local inference workflows rather than a production orchestration system.
Repository:
I’m posting it mainly to compare approaches with others running multi-model setups on Spark. I’m particularly interested in how people handle:
- model lifecycle management
- port allocation
- container reuse
- switching between models when GPU memory is the limiting resource
- routing requests between local models
I would also be interested to know whether others solved this with custom scripts, tmux/screen sessions, Kubernetes, Docker Compose, llama-swap, or some other approach.
