During training on a DGX Spark, whenever the model encounters a sequence that exceeds available GPU memory, instead of raising a normal CUDA out of memory error and crashing the process, the entire machine becomes unresponsive. SSH hangs, the node stops reacting, but external monitors (e.g., W&B) s…

There is a another post discussing this problem System crashes when memory is full - #15 by RazielAU Try to disable swap (via sudo swapoff -a for example) and see if the problem persists.

Thanks! it might be the same issue. I turned swap off, and verified it is off. but now a few hours later i managed to kill the device (this time with lots of CPU tasks - ffmpeg, not GPU - no hardware acceleration even)

I checked both machines that crashed, after a restart, no /var/crash/ logs (only an empty lock file)

Could you share the steps for reproduction.

Attached the nvidia debug log file. nvidia-bug-report.log.gz (601.9 KB) Full reproduction (this crashed the machine twice and i am afraid to run it again) # Clone the repository git clone https://github.com/sign/word-sense-disambiguation.git cd word-sense-disambiguation # Install depende…

Hi @amit59 , I am not able to access the repo link which you posted here for the word-sense-disambiguation project. It seems to be a private repo. Could you please provide me read access for this repo? I am working on reproduction for this issue. My github username: knirmal-nvidia

Thanks @knirmal . I updated the instructions, to a public repo. Please try again.

Hi @amit59 , I am able to access the repo now, Thank you. However, the training data is missing in the repo (generated.tar.xz). Can you please share that too?

My apologies, I was on a branch 🤦🏻♂️ The file and all the other files are now in main word-sense-disambiguation/training/data/generated.tar.xz at main · sign/word-sense-disambiguation · GitHub

DGX Spark becomes unresponsive (“zombie”) instead of throwing CUDA OOM

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10

cosinus December 8, 2025, 4:55pm 6

Sounds like more a case for the NVIDIA SWAT team then. They will ask for the output of their bug reporting tool.

see *** If you have a problem PLEASE read this first ***

Topic		Replies	Views
System crashes when memory is full DGX Spark / GB10 dgx-spark-issue	35	2539	June 15, 2026
DGX Spark Shutdown around 95°C during nanoChat Pretraining (20-30 min) DGX Spark / GB10	21	1784	March 23, 2026
DGX Spark stability / out of RAM / overheating DGX Spark / GB10 llama , dgx-spark-issue	33	1947	July 1, 2026
Spark `hangs` - requires a hard-reset (physically unplugging) DGX Spark / GB10	5	495	April 10, 2026
NVML Support for DGX Spark Grace Blackwell Unified Memory - Community Solution DGX Spark / GB10 Projects cuda , kernel	7	766	April 4, 2026
My DGX Spark Hangs ... is this normal? DGX Spark / GB10 Projects llm , dgx	4	479	April 13, 2026
DGX Spark OS crash on llama4 launch DGX Spark / GB10	6	352	March 14, 2026
My DGX System is getting shut itself down while running my LLM Fine tuning project . RAM Reaches to 100 percent along with GPU reaches 100 percent DGX Spark / GB10	10	865	March 31, 2026
Sparkview — GPU monitor tool with GB10-aware unified memory handling DGX Spark / GB10	38	1640	April 25, 2026
【Urgent】DGX Spark (GB10 UMA) NIM Container Memory Runaway, All Limitation Parameters Invalid, Memory Occupies 120GB+ NVIDIA AI Workbench nim	0	81	May 24, 2026

DGX Spark becomes unresponsive (“zombie”) instead of throwing CUDA OOM

Related topics