Attention: This might void your warranty, but I had this idea (have not done any of those) some time ago, saw this, like the idea, do not really know, if it works and if it make sense, but I want to share the setup link below, my thoughts on this and I wonder, what do you think about this.
I like the innovation in this forum and maybe the following could benefit the community to have a DGX Spark Mini Station (DGX Spark + dGPU, DGX Station links below) or NVIDIA IGX (Thor + dGPU links below).
I came across this, I have not done this and it would be interesting to get your thoughts on this.
Accelerator (text, image and video generation) most interesting for inference speed increase: Possible today in Llama.cpp, but maybe also available in VLLM
Before, my HP ZGX, I mainly used a gaming laptop (AMD GPU (7945hx with AVX512) + 4090 (cuda)) for learning AI, etc.
I wonder, how NVIDIA will do this with NVIDIA DGX Station (system + discrete GPU) or NVIDIA IGX Thor Developer kit (system + discrete GPU) or what is the thinking of system + discrete GPU?
System VRAM increase to run a bit larger models. Maybe, better to have two or more DGX Sparks connected.
I assume this might work with NVIDA GPUs, but I also wonder, if this might also work with cheaper AMD or Intel (Pro B70 with 32GB VRAM for about 1000 USD) GPUS. Maybe, can run via VLLM. Of course, the question is are there any ARM linux drivers and will it work?
This approach seems viable (moving the SSD to USB and using the PCIe x4 5.0 slot for an external GPU).
However, you’d lose one of Spark’s key advantages: the ability to toss it in a backpack with a laptop and have a «supercomputer always with you».
But if you really want to, it’s doable.
I’m not sure if VLLM supports hybrid execution (with router weights on the GPU and experts in Spark’s fast memory). llama.cpp definitely can, but it’s slower overall.
Whether this hybrid mode would be faster than a two-Spark cluster depends on the specific models.
In short, this is a very niche and limited use case. But if you know exactly why you’re doing it, it might actually work.
How many of us will toss the Spark into a backpack… but then how many of us will install a dGPU on a spark.. OK, I was thinking of it in recent weeks, and like magic, I just saw that being done on a Mac (mini or studio) on youtube yesterday.
My solution to bring the Spark with me without bringing it with me was to buy a $50 picoKVM (with tailscale for emergency) and install tailscale on the DGX Spark AND stole/create/copy/modified a dashboard to spin up the models based on my needs. I build in the rules to make sure it would not be OOM. My client apps will point to the tailscale IP for APIs, so it’s always accessible.
I took the one from here. GitHub - thx0701/dgx-spark-status: DGX Spark real-time system monitoring dashboard (SvelteKit + SSE) · GitHub (I’m sure it’s creted by somebody here in the community), replaced the default DGX SPARK and run with it. Was initially getting lots of OOM without knowing it, so get it to refuse to start a model if it calculates that it will OOM. Still very much WIP. llama.cpp working great but vLLM still a bit of pain and I’ve NOT got SGLang working YET.
Haha, I’m curious about the performance of your Minimax M2.7 ud-iq3_s, especially in terms of its programming ability and prefill speed. From my understanding, the highest-quality model that can run on a DGX Spark might be Step 3.5 iq4xs. This ensures 4-bit quantization that can rival q4km in terms of lightweight quantization size, but it has a drawback: due to the unique nature of IQ quantization, it requires more computation, making it slower. After running it, I found that it still uses about 120 GB of VRAM, and its prefill speed is only around 300–500. Honestly, the lowest acceptable quantization size for me is ud_q3_k_xl, which is undoubtedly worse than q4. Sometimes, when generating responses, it exhibits logical leaps, and it even makes syntax errors when writing code. So, theoretically, I don’t think using Q3 quantization is a good choice.
I can only compare Minimax M2.5 (2 bits) vs Minimax M2.7 (3bits).
Both of them works fine on my Asus GX10 , using llama.cpp, I was getting 37 t/s on M2.5 (2bits) vs 35 t/s M2.7 (3 bits), file size on M2.7 was smaller for the same quantization. Logic wise, it was fine and told me that I obviously need to drive to the car wash even though it was only 50m away, otherwise I won’t be able to wash my car. Enabling TurboQuant brings that down another 2-3 t/s . Using Open WebUI, I run a couple of “tell me a story of 2000 words about xyz” and the t/s gets progressively lower with each iterations. Dropping 2-3 t/s each time.
I tried using it with Claude Code CLI (Terminal) running on my laptop, it feels really slow and unresponsive. I tried on Anything LLM with Tools use, “web search” and “create powerpoint”. The first time, it was fine, the 2nd time, it told me it’s just a language model and cannot search web or create files. Probably the harness but I’m open to suggestions for “better” models that will fit onto a single 128G DGX. And better harness that will code, work with documents, etc. Essentially looking for a replacement for Claude Desktop.
Unsloth studio (unfortunately cannot take API) run fine locally on the DGX but when I load the models over the Web UI (Chat) from my laptop, it OOM . I rebooted just to make sure it wasn’t anything else I was running locally, don’t even need 4 bits.
