DGX Spark (GB10) reproducibly hard powers-off under GPU load — fully updated, zero crash capture

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, , ,
1

My DGX Spark does a full hard power-off (not a reboot — the machine powers completely off and must be switched back on) within ~60 seconds of putting the GPU under load. I can reproduce it on demand with a vLLM stress test. The unit is on the latest OTA (“May 2026”) with all firmware/driver updates applied, yet nothing is captured at crash time (no kernel panic, no kdump vmcore, empty /sys/fs/pstore). ~8 unclean shutdowns in 11 days. Looking for known issues / a firmware fix, or guidance toward RMA.

System

  • Product / S/N: NVIDIA DGX Spark / 1984125000037
  • System version: A.7 (baseboard P4242 rev A04)
  • SBIOS: AMI 5.36_0ACUM018 (2025-08-06)
  • DGX OS: 7.5.0 — latest OTA “May 2026” / OTA2604.Computex applied
  • OS / kernel: Ubuntu 24.04.4, 6.17.0-1021-nvidia (aarch64)
  • GPU: GB10 — driver 580.159.03, vBIOS 9A.0B.25.00.00, CUDA 13.0
  • Memory: 119 GiB unified

Reproducible trigger + evidence

I run a lightweight 1-minute recorder that logs memory / GPU / container state to disk. Here is the GPU column leading right up to a crash during a vLLM stress test:

22:56  GPU 48C  util  4%  11.6 W   (idle)

22:57 GPU 58C util 3% 13.9 W

22:58 GPU 62C util 3% 15.0 W

22:59 GPU 74C util 96% 79.4 W <- stress test begins, GPU goes to full load

23:00 GPU 79C util 96% 82.4 W <- last sample before death

--- machine HARD POWERS OFF (full power loss; had to power it back on) ---

At the moment of death: ~33% system memory free (no OOM), swap barely touched, earlyoom never fired, no NVRM Xid, no thermal trip logged. After the manual power-on, /sys/fs/pstore is empty and /var/crash has no vmcore even though kdump is armed (crashkernel=2G, “ready to kdump”). The systemd journal simply truncates mid-line.

The signature — full power-off (not reboot) the instant GPU power ramps to ~82 W — makes me suspect a power-delivery / power-protection cutoff under load, not a software hang (a software hang would reboot or freeze, not cut power).

Ruled out (with evidence)

  • RAM OOM — 33% free at death, earlyoom never triggered, no oom-kill in logs.
  • Software kernel panic/oops — no vmcore despite kdump armed; no panic/Call-trace in the journal.
  • GPU Xid / fatal GPU fault — none logged.
  • Overheating (as a logged thermal trip) — none; GPU was 79 C at the last sample.
  • Stale firmware — already on the latest OTA; apt reports 0 upgradable; fwupdmgr get-updates reports no firmware updates; nvidia-spark-ota-check is-ota-available → false.

Crash history

~8 unclean shutdowns between 2026-06-03 and 2026-06-13, including one boot that survived only ~2 minutes. The system is rock-stable while the GPU is idle; every crash correlates with GPU load.

Questions

  1. Is this a known issue on early GB10 units, and is there a fix in a newer SBIOS / EC / power firmware not yet in the public OTA?
  2. Is there an out-of-band power/EC/PMIC event log I can pull? /dev/ipmi0 is absent on this unit, so ipmitool can’t read a SEL.
  3. Given it reproduces on demand and the unit is fully updated, what is the path to RMA evaluation?

I can attach the full nvidia-bug-report.sh output, the recorder logs, and journalctl --list-boots. Thanks in advance.

2

I’ve never experienced this for the months that I’ve had it, but I get it repeatedly now the last few days, maybe there is a regression in one of the latest updates. I’ve tried it across a lot of models but it just keeps happening.

3

Its overheating, few posts around here about it, but I got both of mine to completely stop powering off under load (and coincidentally, never since had the power limiting bug re-appear either) after I repasted the GPU / CPU + removed the outer shell case - which severely limits airflow on my Asus GX10.

4

It doesn’t even barely get to be under load most of the time, I decided to update the gpu drivers just for fun didn’t help. I also have one of the variations of 3d prints with two fans constantly pushing air in. And I never got this before, this is recent.

I’m about to factory reset it and cross my fingers.

5

mine started like that, one of the sparks started shutting down on longer runs, then it became more frequent and I could get it to shut off sometimes within 5 mins of inferencing. I repasted that unit and shutdowns stopped, then my 2nd unit started doing it (although much rarer than the first unit), repasting it ensured it never re-occured.
As mentioned, since then I discovered that simply removing the case (I dont use WIFI on my units and they are both set in bios to power up when power is turned on, so no need for power button either) drops the temps by another 15c, so I took mine off and stuck a couple of 80mm usb fans pointing at the front. Since then both my sparks are rock solid, no crashes, no weird power limit bugs, nothing.

I stuck them both into a perspex box with the rear end cut off, it’s not as nice looking as it was, but I am far more interested in stability / longevity, so can live with it

PXL_20260613_171012293
PXL_20260613_1710122931920×1446 448 KB

PXL_20260613_171028163
PXL_20260613_1710281631920×1446 610 KB

Ive flipped top unit upside down as it seems to get better air hit from front fans, but it works just fine with both the right way up.

6

What are your idle and load temps?

7

Now? idle around 30-35c, load goes between 55-70c.

8

Thanks @arctic.gus and @nbsparkz, this is really helpful.

Temps on my unit (same 1-minute recorder): idle sits at ~47-48 °C even after idling overnight, and right before the crash the GPU went 74 °C → 79 °C and was still climbing as load hit 96% / 82 W — then hard power-off. So my idle alone is ~13 °C hotter than your 35 °C, @arctic.gus — looks like my thermal baseline is just too high.

An EC/firmware thermal-protection cutoff actually fits everything I see: instant power-off (not reboot), no kernel thermal-trip, and empty pstore/vmcore — the kernel never gets a chance to log it. So I’m now treating overheating as the leading suspect.

Like @nbsparkz though, mine is fully updated and the onset is recent, and one crash hit ~2 min after boot — so a thin thermal margin and/or a recent regression both seem in play.

Plan before I open the case (warranty): test the non-invasive fixes and report back — (1) cap the GPU clock with sudo nvidia-smi -lgc 200,2150 (a few threads here say this stops the “shutdown at full GPU speed”), and (2) more external airflow + lower ambient. In parallel I’m pursuing RMA, since it reproduces on demand on a fully-patched unit.

@arctic.gus when you repasted — was the stock TIM visibly dry/pumped out, and did removing the shell alone do most of the temp drop, or was the repaste the bigger factor?

9

I now see @arctic.gus is probably right but I still feel like it has to be a regression somewhere, I know things go bad after some time but I’ve run a lot of hardware to the ground and I’ve babied this spark from the beginning and I just did some updates a few days ago. Allegedly the AI found some things online that it’s actually the SoC/cpu temp and not the GPU temp that triggers the thermal event so nvidia-smi isn’t reporting the number causing the issue. The GPU may be at 79C but the SoC/cpu may be at 100C.

I put a more intense fan in it and it got up to 100W with a max temp of 80C on the GPU and kept going, I may just try to under clock it to keep the temp down, I’m also still running the 595 drivers so I’m not sure if that has any play in it. Here is a graph of the data smi has been reporting over the last 10 minutes. The GPU usage is basically unimportant for understanding the temp and it’s all correlated to the wattage which makes sense.

image
image851×383 64 KB

10

When I replaced TIM, there seemed to be a patch of CPU / GPU that had very little, if any TIM coverage, but TIM itself was very dry and brittle, so I suspect shutdowns happened when one corner of the chip overheated, even though the rest of the chip had some contact with the heatsink… or the compound itself lost effectiveness by getting boiled too many times.
AFAIK, replacing TIM does not void warranty, unless you break something in the process - I have logged many warranty requests for repasted gaming laptops over the years and never had any issues. One word of caution though - the CX7 chip heatsink screw pots are fragile and will break if overtightened, one of mine broke when I used an electric screwdriver, luckily the other 3 hold the heatsink just fine and CX7 chip itself doesnt generate as much heat as the CPU / GPU anyway.
Main issue imho is poor airflow design of the case itself (at least on ASUS GX10 model), there seems to be very few inlet vents and the fans themselves are puny and not agressive enough, hence me opting for a DYI solution of sticking a couple USB ones.
I tried using a fan with the case on, but because case design it really did not amount to much.

11

Yep, its the CPU on my sparks that was overheating. It would regularly hit 96c during inference (even though only 1 or 2 cores were loaded), followed by a sudden power off.

12

i’m not 100% sure it is just gpu + cpu temp, i think issue overall pcb/case trapping heat. For me I got shutdowns where gpu was 80C and cpu 70C (yet in stress test CPU goes 92C for 30 minutes and totally fine).
I have 2 asus gx10s stack on top of each other right in front of inflow fan, and right behind them top extraction fans, and rack shelf has ventilation holes for bottom unit. Top one was shutting down unpredictably, sometimes on very first inference, sometimes after hours of work.

When I pulled top GX10 in front of bottom (with added overhang support), so top’s bottom intake completely uncovered it stopped shutting down.

I think it was some performance improvements in last updates, connect x7, cpu/gpu, etc, which pushed silicon/power delivery over thermal threshold, while main sensors reports all good, keep pushing, something already too hot/has too big resistance and cause thermal shutdown from controller/firmware level. I have node exported collecting all thermal sensor exposed to OS and all of them looks normal, also gpu frequencies stays 2385-2411mhz (boost clocks).

13

My read of it is:

-I am pretty certain now that something overheats to trigger the shutdown / crash and I am 99% sure its the CPU temp just based on the fact that it provides the hottest reading from all the sensors before the sudden shutdown occurs. GPU is usually at least 10-15c cooler than CPU during inference, but the GB10 CPU / GPU chiplets are right next to each other so maybe there’s a chance that some transference of CPU heat causes GPU to shut down.

-CPU temp sensor itself might not be on the exact spot that is overheating (i think CPU sensor is a compound, possible average of all CPU cores), and unlike traditional GPUs / CPUs we dont really have a hotspot temp metric exposed on our sparks, so it might not capture the exact culprit.

-this is likely a compound issue, so TIM quality / effectiveness, airflow, fan curves, case heat-buildup are all likely factors that contribute (to varying degrees) and addressing any one one of them could be enough to keep the temp below that critical crash threshold.

14

If you go this path try removing the case first so we can get another perspective on case v paste importance.

I started by repasting mine which live in an enclosed rack, but that wasn’t enough. Still had a few shutdowns. Now, repasted, no cases, and a 120mm fan, I have reliably been running a loop for many days with repeated bursts of concurrency 20 and not a single shutdown (doc ingestion for OCR/RAG/knowledge graph). The downside is I can’t close the rack door yet. Next step is setting up a shelf with three 60mm fans per node.