I finally decided to pull the trigger and I ordered 2 x DGX Sparks that I am going to stack. I want to run local models for software development, I am consdering Qwen and DeepSeek for this, but more importantly I want to utilize it for learning more about how models work under the hood, how to train and fine-tune them, how to deploy them, and more. I am just going to take it one step at a time, but it is important for me to slowly dive deeper into the AI rabbit hole and grow my skills in this area.
I would apprecriate any tips the community might have for anything related to setting up my spark properly and using it for local development, which models you have had great success with, and resources I can use to learn more about this new world.
When they arrive and you do the initial install, things may take a while while they reboot and update their factory firmwares. There can be a temptation to think the install might have hung. It hasnt, just be patient go for a long walk.
Before they arrive downliad the recovery image. I say this because you will want to experiment a LOT and you should. But once youve traversed half of the very steep learning curve, you will realise that you took some wrong turns aling the way and it will be best to start clean again.
Oh, and if you dont have 1Gbps+ internet get it, because you will be downloading a couple of terabytes of models from huggingface
Congrats on the purchase! One tip that saved me a headache: do the initial setup over wired Ethernet, since Wi-Fi can drop mid-download during the long OS and firmware updates. Also give them plenty of airflow, since the units run warm under sustained inference loads. Plus one on giles8’s advice to grab the recovery image before you start tinkering.
Welcome! We’re happy to have you here and to tinker with advanced technology alongside us!
Almost all “gotchas” are probably documented here in the forums by now… but if I could pick some that I would highlight, they would be:
sometimes it can only be fixed by shutting down and unplugging it from the wall for a few seconds. The usdc PD stuff can’t recover by itself and it will be down-clocked until the cold restart.
when it OOMs, it’s really dead. You can’t recover it remotely. add an OOM watcher around your heavy stuff and consider something that can remotely power cycle like a smart plug.
The machine will shut itself down if the power draw is too high. If you’re writing kernels or configuring stacks yourself watch out for it.
tailscale / netbird / zerotier / CF tunnels / whatever are cool for accessing your stuff on the go (and from your phone!)
update
the recipes for serving that are available are super cool and really great. You should probably use them. However, tinkering with it yourself and learning why things are done can make you feel like a super genius :D.
Thank you! Excited to be here! Ugh, gotta watch out for the OOM / power draw issue you mentioned if I am on the go, I was planning on leaving it at home when I am traveling and using tailscale to access it, and if it shuts down while I am away for 2 weeks that would suck, so that’s good to know, thanks!
While the PD issue must be real, some of us have never experienced it (me for example).
OOM tends to happen during model launch most of the time, any model that loads and uses around 117GB on your pair will generally be stable. Having an OOM wwatcher can help against a freeze up.
Some people guard against temperature issues by underclocking the GPU (surprisingly with no overall performance hit, because dethrottle - throttle up seems to be wasteful and to be avoided).
I was considering having a small fan pointed at them at all times, but I am not sure if that would even help anything, but good call with the ethernet cable during setup
Using Hermes and getting one of its profile connected to a cloud LLM via API has been very helpful in setting up the sparks. It can do all the configuration for you and at least in the early days it will be very helpful to get you up and running quickly.
Run it headless. Set up SSH access. You’re leaving memory on the table by loading in the desktop environment. Ideally after you set up SSH access, don’t connect any displays at all, just login remotely.
Set up public/private passwordless key access for the above, it’ll save you time in the future.
I would run any variant of GLM 5.3F at this point over anything else for basically any purpose now. DSV4F used to win out on speed, but now you can get Tensorfold variants that even do great on that.
Get an external fan.
Look into downclocking them (or at least the head unit). Once you link 2 together, the heat ramps up more than you would expect.
Get used to keeping up with various forum threads for your best recipes lol.
yeah I was planning on only using ssh + adding keys for my local models to litellm so I can use them natively from my other devices. I already have a small fan ready for 'em, am considering getting a very small server rack with fans for them as well (not just for cooling, but wouldn’t want them to fall down or something to fall onto them).
Thank you very much for all of the links, I just got an email that they have been shipped and I should receive them in 1-2 days, I will make sure to look over some of this before they arrive
Welcome onboard! Getting a smart plug to power cycle may not be enough to resurrect a dead node remotely. You may need to “setup BIOS to boot up upon power on”. I had to do that for my GB10 nodes, but forgot if that is needed for my DGX Spark (Nvidia) or GX10 (Asus), or both.