Don’t write it off — we had this identical symptom on our 2× DGX Sparks (hard death under sustained load, fine at idle, model-agnostic, no warning, sometimes needing a physical power-cycle) and it turned out to be fixable in software + firmware. The tell is in your own report: it’s model-agnostic and only dies once they’re doing real work, never just loading — and it got worse with two models loaded at once (your 35b+27b). That’s the fingerprint of the GB10’s 128GB unified-memory cliff, not a bad model.
Here’s what actually fixed it for us, in order:
1. Give the unified memory real headroom (this was the big one). On the GB10, weights + KV cache + CUDA workspace all share one 128GB pool. Under sustained jobs the KV cache grows, and if total allocation creeps past the pool you do not get a clean OOM — you get a UVM page-migration livelock that hard-locks the whole box with no warning and no log, and the OOM-killer never fires. Loading two big models (35b+27b) or the 122B leaves almost no headroom, so a busy run tips it over. Fix: cap how much the server grabs — in vLLM that’s the flag --gpu-memory-utilization set to 0.85–0.92 (not 0.94+), and don’t co-load multiple large models. Leave ~10–15GB free. This alone stopped ours from dying.
2. Update the platform firmware — BIOS/BMC, not just the OS. “Everything’s updated” almost always means OS packages + driver. The Spark’s system firmware is a separate update and it materially improved our stability under sustained load. Do that update specifically.
3. Cap the GPU’s max power draw. Enable persistence mode and pull the power limit a touch under max so sustained inference can’t spike past the envelope — enable it with sudo nvidia-smi -pm 1, then set the cap with sudo nvidia-smi -pl followed by a wattage a few watts under the reported max.
4. Lock the clocks. Pinning the clocks instead of letting them boost and drop kills the power transients that compound everything above — do it with sudo nvidia-smi -lgc followed by your min,max clock values.
Do #1 and #2 first — between capping memory utilization and updating the platform firmware, ours went from dying every heavy run to rock-solid. #3 and #4 are the belt-and-suspenders for the power/thermal side.
And keep the physical basics honest too: dedicated wall circuit (ideally a UPS), real airflow clearance — a browning circuit or heat-soak will still trip it no matter how clean the config. But start with the memory cap; that’s the one that gets described as “dies under load for no reason” the most.
