Is there certain docker images that need to be loaded to run LLMs locally? This is a newbie question. I am new to Linux / Docker / git / uv. I am familiar with BSD / Python / Conda /mlx-lm etc.
On my new GB10, I set up ~/myenv and installed pip, uv, huggingface_hub[cli]. In Docker, set up vllm-node:latest ( eugr/spark-vllm-docker ) plus there is also nvcr.io/nvidia/vllm:26.01-py3. There are also two pytorch images, cuda:13.0.1-devel-…, ubuntu:latest. Tried following…
Running this as ‘Hello World’ to check my system. Error appears to be related to local environment. Any suggestions as to what needs to be done to run any local LLM?
In my experience (I’m about three weeks old here), I started fresh on my system after a few tries with nvidia’s stuff, reimaged everything ala factory reset, update/upgrade my Ubuntu install, then git clone Eugr’s repo, and off I went.
That’s the easiest way to start.
just make sure you’re fully updated:
sudo apt update && sudo apt upgrade
Accept everything update, reboot, and follow’s Eugr instructions in his git repo and you should be gold
I did a factory reset of GB10, easier said than done. Documentation was different my bios settings (but easy enough to guess right steps) and during third reboot step bios did not recognize my usb drive (slightly more rough to sort out).
After updating my system, installed eugr’s spark-vllm-docker. That is the only image in my docker now. ./run-recipe.sh qwen3-coder-next-fp8 --solo installed the llm. Verified it using a single line curl command and it is good to go.