Hope all is well. I’m currently working on a personal project with my security cameras, and getting tired of using my laptop to run LLM inference. Highly considering a DGX spark!
However, I live in a remote area with pretty terrible internet access, so fully offline setups is a huge win! Looking at the latest release notes from April/July, it seems air-gapped setups are on the table, but struggling to find instructions to setup this way before purchasing, and would love to make sure this will work.
Would I need to boot the latest recovery image for the first time rather than running through the out of box setup? Apologies if this is a dumb/newbie question. Thanks in advance for the help!
To setup spark first time you would need some internet access. It will download latest packages (drivers and system updates from ubuntu and nvidia repositories). Also you’ll need to download llm engine and models (i’d go with latest llama.cpp sources from github to compile it locally and some GGUF models, this is easier to manage without reliable internet access).
My GX10 came with quite full setup of packages, probably could compile llama.cpp without even downloading anything except llama.cpp itself. But not sure if setup wizard will allow you to run fully offline.
Copy that! Though I’ve dug a bit deeper into the DGX OS images and they look to be locked behind the enterprise support portal which I will not have access to. Is there no other way to commission one of these fully offline as a retail consumer?
I suppose you could forgo DGX os altogether and boot fedora or an equivalent for ARM, but assuming there would be hardware compatibility issues immediately for gpu support…
I’d do starlink. Another thing I thought about too was hooking a VLLM API into a loRA wireless network for offline long range connection to the LLM running on the spark.
that’s not a problem, though, I have no idea what you mean by lora wireless network, LoRA as Low Rank Adaptation is a post-training technique done right at the Spark itself, no internet needed, but maybe you mean something else
LoRa(WAN) does not support natively TCP. But there are a number of gateway technologies can could do the talking with an HTTP API.
That being said, the idea to expose an LLM via LoRa is wild. If you manage to do the “network plumbing” and can live with the bandwidth please share your solution. The final user experience would be remindful of the dial-up BBS in ideal conditions with an additional spark of “intelligence.”
All you need to do is a proxy layer that translates UDP or whatever packets are being fired to TPC/IP and translate to the model. Can run directly on spark
Well, you can download and install ubuntu 26.04 (arm version) on the spark. But will need to download and install some more packages from nvidia repository (595 drivers, cuda toolkit 13.3, etc). It does work though seem some incompatibility with this version of ubuntu, my GX10 consumes 15W more total power than with DGX OS, even in idle mode (25W idle in dgx os → 40W idle in 26.04, 180W full load → 195W full load). llama.cpp works on ubuntu 26.04 quite fine.
May be 24.04 is better with this but will need some time to test.
The latest Spark image no longer requires an internet connection to finish setup. You can fully setup and use a DGX Spark in an air-gapped environment.