Spark `hangs` - requires a hard-reset (physically unplugging)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

We’re running custom PyTorch RL training code on the DGX Spark, and we’ve hit a reliability issue around out-of-memory events. When the model OOMs, the entire system becomes unresponsive—SSH freezes and the box effectively hangs—so recovery requires a hard power cycle (we’ve literally had to unplug the power cable).

On our H100 systems we OOM frequently as well, but it’s typically a process-level failure and the machine stays reachable. On the DGX Spark, because memory is unified, an OOM appears to exhaust system memory and takes down the whole host, leaving us with no clean way to recover remotely.

This is becoming super-annoying, as we run everything remotely…

2

Some answers in the forum include disabling your swap so the process crashes rather than the system, or to tune your process to match memory constraints (keeping cpu and gpu constraints in mind).

I bought a remote KVM to see if I could alt+ctnl+del when it happens, but I haven’t tried it yet. I selected this one: Comet Pro (GL-RM10) Remote KVM over Wi-Fi — GL.iNet EU

I also researched ‘smart plugs’ for power cycling but didn’t go that route.

Fingers crossed that the next DGXos release gives us some relief.

3

Sorry to hear you are running into this problem. This is a known issue which we are actively working to fix. The next major Spark OS release should have better handling of OOM and other issues.

4

Hi @aniculescu, is there a timeline when when a patch/fix/solution might be released. I have two Sparks and this keeps happening to me. Thank you!

5

The behavior difference versus H100 comes down to memory architecture. On H100, GPU memory is discrete — an OOM is contained to the CUDA context and the host stays up. On GB10, CPU and GPU share the same physical memory pool, so an OOM event can starve the kernel itself before the OOM killer gets a chance to act.

While waiting for the OS-level fix, the earlyoom approach documented here is a practical mitigation: https://forums.developer.nvidia.com/t/mitigating-oom-system-freezes-on-uma-based-single-board-computers/362769

6

Do we have a timeline for this, since this makes the dgx unusable when we only have the ssh access, even if I can physically reach the device, I have to hard reset when OOM.
For LLama.cpp, it’s fine that if we already know what settings will not cause OOM and quite stable.
But for embedding model like using transformers library, flag embedding( already tried different versions of the library, CUDA and pytorch), I don’t know why, it cause OOM and seems the VRAM leak unusually, for the same script on a X86 machine with CUDA(GPU), it never happen and only use like 20% of VRAM compared to dgx spark also no memory leak, but in dgx spark, the starting VRAM is already much large compared to on a X86 machine, and on X86 it doesn’t leak but on spark it leaks.
So for now, I can only run llama.cpp and some OCR on the spark, was planned to run embedding model on it, but it is not possible and require hard reset when it hangs.