My sparks (2) have begun to shut off randomly. I cannot find any out-of-memory or thermal logs indicating that was the issue. They were running for 55 days straight, and then all of a sudden they keep powering off. Has anybody had an issue like this recently? And is there not a way for them to automatically reboot themselves?
That is indeed a tricky issue. If i would have such problem with my cluster probably i would build some small power relay driven by an esp32 or rpi which pings the sparks on local network periodically and if nothing there i would automate a hard power cycle - in bios the sparks by deafult turn back on upon full power cycle - Auto Boot. But the fact that both have the same issue is too suspicius. I would put there a UPS because if you have any power fluctuation on the power grid that could also cause this.
Same happened to mine, there were no signs of overheating in the logs, but they were definitely overheating (because it all stopped when I repasted, and later removed the outer shell as that dropped temps even more). I suspect thermal sensor exposed in the OS might not actually capture the hot-spot, but instead is an average temp of all cores - so one core going 105c, while the rest are lower might induce the shutdown.
I suspect the reason for my sparks operating ok for a while before hitting this issue was because thermal compound was fresh out of the factory (albeit not of great quality), then over few months of constant use it dried up and lost effectiveness. When I repasted, thermal compound was dry as a rock on both units (one unit went few weeks before the other).
I now run mine without outer shells with some USB fans pointing at the front and my temps idle around 27c and sit around 65-73c during load. They might not look as pretty as they did in their cases, but I am only interested in logevity / stability, so not really bothered about it.
Prior to repasting / case removal they were hitting 95c on CPU temp sensor reading regularly, though I suspect one of the CPU corners might have been way hotter than the rest, as when I removed my heatsink it had a bit of a gap in coverage / air bubble.
As a bonus, since repasting / removing outer cases, I have not had a single occurrence of that issue where GPU becomes power limited until you unplug the PSU, thats 6 weeks of solid running without experiencing this issue which used to happen once every week or 2, so I am pretty sure that problem is related to temps as well.
Wow that sounds crazy! - if this is is due to thermal paste degradation or insufficent contact area caused by bad manufacturing thats is definatly can be the cause. I will keep an eye on this too.
I have a pi sitting next to them, is there anyway to boot them up over Ethernet for instance? Would a ups eliminate the straight shut down? And if it overheats and shuts down will it startup again?
Oh man I hope I don’t have to pull this apart to do this, would they offer a warranty service for this?
I’ve seen similar once. It was my PDUs fault. I noticed one of my sparks unable to exceed around 35w before tripping and shutting down or reboot.
0-logs.
Fix was to unplug PDU from wall and spark for ~30s.
Hope yours is similar and not an overheating issue !
I had mine shut off yesterday too. I had started running some InspectAI evals against the new Nemotron Puzzle model and after a few hours noticed the box was unresponsive. I pressed the power button briefly once, and shorly after it had come back to life.
I haven’t had this before, and I don’t know if it was related to running this model in any way (I have had issues with some of the Nemotron models crashing in the past).
That was one of my disapointment when i realized the spark doesnt support wake on lan - the only way to wake them as far as i understand is by using Auto Boot ( on by deafult ) and preform hard power cycle. The way you could make this work using a PI is to write a little python service for the PI to ping the mDNS of each node in every ie. 5 min or less then if there is a few consecutive ping fail then simply using the pi GPIO ports we can connect to a Relay which could remove power and give back on affter 30 second or so - this would cause a hard power cycle and the auto boot would bring the sparks back up online. - This is also good accidental OOM cases because usually sparks become unresponsible but ssh fail is a better indicator ping can still work. You can get on amazon easly some fully built IoT Relay which you just need to hook up with the rpi gpio GND+GPIO and you can toggle the power easly the rest is only a python orchestration Amazon.com : iot relay - For the other question UPS would only eliminate shutdown if the origin of the shutdown is the layline power instability or voltage / freqfrency drop - but given the facts this is an expensive rig a protective UPS allways a good practice to avoid HW damage from the grid (under and over voltage protection). - And to your other question if overheats and frozen or become non responsive then yes this method would bring back them up by the hard power cycle.
They continue to shut down,both now, is this worth opening a warranty for? Has anyone had any experience with this?
I wrote a governor that will scale based on temp and hopefully not allow to run too hot or draw too much, it’s definitely been hotter lately environmentally ,so if it’s spiking and shutting off I hope to solve this. Will keep you posted and work on better cooling solutions. I separated them as they are stacked also.
Thank you for reporting this. Unexpected shutdowns under load are not expected behavior on DGX Spark.
A few things would help narrow this down:
- Are you using the original NVIDIA-supplied power adapter for each unit?
- What is the approximate ambient temperature around the systems, and are the units still stacked or now separated? The documented ideal operating temperature is 5°C to 30°C.
- Does the system power off only under sustained workload, or also when idle?
- If possible, please run DGX Spark Field Diagnostics and share the logs. If the units still boot into the OS, that is the recommended next step for thermal/performance-related issues.
In the meantime, please keep the systems unstacked, ensure clear airflow, and confirm each system is on stable wall power with the supplied adapter.
We have logged an internal ticket to track this issue and will update the thread once we have more information.
Hi Neill, thanks for the ticket. Here are the answers and diagnostic results:
1. Power adapters — Yes, original NVIDIA-supplied adapters on both units.
2. Ambient temp / stacking — Units are now separated (were stacked during the shutdown cluster). Ambient is approximately 22–26°C indoors, though it has been warmer recently. Both are now in open air with clear space around them.
3. Load vs idle — Shutdowns occurred exclusively under sustained GPU workload (running local LLM inference via vLLM). Never observed at idle.
4. DGX Spark Field Diagnostics results:
DCGM v3.3.9 installed and run on both nodes. Summary:
Overall Health: HEALTHY on both (dcgmi health passed)
spark-ff13 — GPU-f2c73f99 | 46°C | 11W idle | SM clock 2411 MHz | No ECC errors
spark-4f0c — GPU-4ab10510 | 42°C | 11W idle | SM clock 2398 MHz | No ECC errors
No thermal throttle events, no HW power brake events, no kernel MCE or GPU fault messages in any log since last boot
dcgmi diag -r 1 returns “Detected unsupported Cuda version” — DCGM 3.3.9 does not support CUDA 13.0 (shipped with driver 580.142 on GB10). 3.3.9 is the latest available in the SBSA repo. The diagnostic module cannot run on its own hardware.
SysMon module: Failed to load on both nodes (every run)
Current status: Both nodes have been stable for 3d 20h+ (since Jul 9) after physically separating the units and implementing a thermal governor that scales CPU frequency and LiteLLM throughput based on GPU/ACPI temperature readings. No shutdowns since.
Happy to provide full nvidia-smi -q output, kernel logs, or anything else from either node. The DCGM diag version gap and SysMon failure may be worth flagging internally — we couldn’t complete the full diagnostic suite due to those issues.