During training on a DGX Spark, whenever the model encounters a sequence that exceeds available GPU memory, instead of raising a normal CUDA out of memory error and crashing the process, the entire machine becomes unresponsive. SSH hangs, the node stops reacting, but external monitors (e.g., W&B) s…

Thanks! it might be the same issue. I turned swap off, and verified it is off. but now a few hours later i managed to kill the device (this time with lots of CPU tasks - ffmpeg, not GPU - no hardware acceleration even)

I checked both machines that crashed, after a restart, no /var/crash/ logs (only an empty lock file)

Sounds like more a case for the NVIDIA SWAT team then. They will ask for the output of their bug reporting tool. see *** If you have a problem PLEASE read this first ***

Could you share the steps for reproduction.

Attached the nvidia debug log file. nvidia-bug-report.log.gz (601.9 KB) Full reproduction (this crashed the machine twice and i am afraid to run it again) # Clone the repository git clone https://github.com/sign/word-sense-disambiguation.git cd word-sense-disambiguation # Install depende…

Hi @amit59 , I am not able to access the repo link which you posted here for the word-sense-disambiguation project. It seems to be a private repo. Could you please provide me read access for this repo? I am working on reproduction for this issue. My github username: knirmal-nvidia

Thanks @knirmal . I updated the instructions, to a public repo. Please try again.

Hi @amit59 , I am able to access the repo now, Thank you. However, the training data is missing in the repo (generated.tar.xz). Can you please share that too?

My apologies, I was on a branch 🤦🏻♂️ The file and all the other files are now in main word-sense-disambiguation/training/data/generated.tar.xz at main · sign/word-sense-disambiguation · GitHub

DGX Spark becomes unresponsive (“zombie”) instead of throwing CUDA OOM

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10

cosinus December 7, 2025, 8:42am 3

There is a another post discussing this problem System crashes when memory is full - #15 by RazielAU

Try to disable swap (via sudo swapoff -a for example) and see if the problem persists.

Topic		Replies	Views
DGX Spark Shutdown around 95°C during nanoChat Pretraining (20-30 min) DGX Spark / GB10	21	1432	March 23, 2026
System crashes when memory is full DGX Spark / GB10	28	1769	December 22, 2025
Spark `hangs` - requires a hard-reset (physically unplugging) DGX Spark / GB10	5	361	April 10, 2026
NVML Support for DGX Spark Grace Blackwell Unified Memory - Community Solution DGX Spark / GB10 Projects cuda , kernel	7	618	April 4, 2026
My DGX Spark Hangs ... is this normal? DGX Spark / GB10 Projects llm , dgx	4	247	April 13, 2026
DGX Spark OS crash on llama4 launch DGX Spark / GB10	7	256	March 14, 2026
My DGX System is getting shut itself down while running my LLM Fine tuning project . RAM Reaches to 100 percent along with GPU reaches 100 percent DGX Spark / GB10	11	746	March 31, 2026
Sparkview — GPU monitor tool with GB10-aware unified memory handling DGX Spark / GB10	38	1042	April 25, 2026
Memory Creep on DGX Spark: Where Your 128 GB Actually Goes (And How to Stop It) DGX Spark / GB10 jetson , nemotron	2	565	March 30, 2026
Anyone have hard crashes on The DGX? DGX Spark / GB10 cuda , llama , dgx	8	391	January 16, 2026

DGX Spark becomes unresponsive (“zombie”) instead of throwing CUDA OOM

Related topics