System crashes when memory is full

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
24

Yep, the easiest way to reproduce is to run vllm with ~0.85 memory utilization and then start llama.cpp with a sizeable model - gpt-oss-120b or something like that. Instant freeze, once llama.cpp starts loading the model. If you don’t kill the process within a few seconds, you are out of luck.

Looks like it’s not just the swap itself, as I had a situation where my swap was ~10GB and the system was working just fine. Maybe swapping out mmap’ed files, given that mmap is not working great on Sparks currently?

25

Is it just my imagination or has this issue been improved? I tried after the last update, and deliberately tried to lock it up and couldn’t seem to lock it up completely. It still slowed down to an absolute crawl, but with three different attempts I was always able to terminate the offending process with CTRL+C without needing to physically reboot the box. Furthermore, it went back to being responsive once I killed the process, whereas normally, I found I’d have to reboot after this happened as the machine would just chug afterwards, which doesn’t seem to be a problem now. I might leave my swap file on for the time being to see how it goes.

26

Nope, happened to me yesterday. I didn’t kill the runaway process on time, and when it consumed all RAM and all swap, the system became unresponsive. I was able to kill the process previously, but it only worked if you do it within the first couple of minutes.

27

Yeah, you’re right, I just ran another experiment, and straight away I just locked it up. So I guess I got lucky before with the other tries. sigh I really hope they address this as it’s probably the biggest issue the Spark has at the moment.

28

Any update on this issue? Even if I tried to turn off swap space and it still crashez on me. Would be nice to know if there is any other workaround while we wait on the fix.

29

Hi, this case remains open. Once memory diagnostic log can be correlated to internal repro consistently, engineering expects to proceed toward a targeted fix.

30

I’ve found this to be the easiest repro:

  1. Open jupyterlab in Firefox from the DGX Dashboard

  2. Run this:

import torch

x = torch.randn((200000, 200000), device=‘cuda’, dtype=torch.float16)
while True:
    torch.matmul(x, x)

With sudo swapon -a (the default configuration) the machine will hard lock and you have to hard reset the box with the power button.

With sudo swapoff -a semi-bad things happen. Best case, processes are killed. Worst case the screen goes black and you have to enter your password once things return. But no need to hard reset the box.

Change the 200000 to 170000 and everything works fine with or without swap and memory caps out around 115GB.

This is dead simple and 100% reliable for me.

31

@hoesing your code is designed to trash the system. That’s a huge tensor with 40 billion elements (200000 x 200000) and each element is using 2 bytes. That requires about 80GB of contiguous GPU memory, emphasis on contiguous. But you also need memory for tensor multiplications, and whatever else is running. So you just found out that a tenor of that size is not suitable for the Spark!

32

It’s a simple repro case, not intended to be practical code :)

33

Closing out this thread, the latest driver update has fixed this issue, ensure you are on driver 580.159.03 or later

34

Hi, I’ve been having this same problem. However, updating to the newest drivers (I have 580.159.03), I’m still having the problem.

35

I am still seeing this issues even with 580.159.03. Wrote up a small guide for steps that are working for me, hope it helps: Hardening Your DGX Spark for AI Workloads - Geeta

36

Interesting, can you send repro steps if you are still experiencing a system crash due to OOM?

37

@aniculescu If you load one of the large model Gemma4-35B or Qwen3.6-27 models using HF without the earlyOOM tunings system goes into slow creep memory mode and crashes.

Any model that exercises memory to the limit puts the system in this state.

MODEL = "google/gemma-4-31B-it" (or Qwen3.6-27B or similar sized model)
model = AutoModelForImageTextToText.from_pretrained(
MODEL, dtype=torch.bfloat16, low_cpu_mem_usage=True).to("cuda")

or with AutoModelForCausalLM – all models 27B and larger show same behavior.

Logs show errors like:

Jun 07 17:23:26 kernel: oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,
task=python,pid=170642,uid=1000
Jun 07 17:23:26 kernel: Out of memory: Killed process 170642 (python)
total-vm:454941032kB, anon-rss:1236kB, file-rss:25724kB,
shmem-rss:852kB, pgtables:6444kB oom_score_adj:0

Jun 07 17:34:36 kernel: oom-kill:...task_memcg=/user.slice/user-1000.slice/session-3.scope,
task=python,pid=202567
Jun 07 17:34:36 kernel: Out of memory: Killed process 202567 (python)
total-vm:454941036kB, anon-rss:0kB, ... oom_score_adj:0

Jun 06 12:51:46 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
Jun 06 12:52:05 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
Jun 06 12:52:05 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:345
Jun 06 12:52:06 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
Jun 06 12:53:22 sparky kernel: tokio-rt-worker invoked oom-killer: gfp_mask=0x402cc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM|__GFP_NOWARN), order=0, oom_score_adj=-1000
Jun 06 12:53:22 sparky kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/session.slice/wireplumber.service,task=wireplumber,pid=34090,uid=1000
Jun 06 12:53:22 sparky kernel: Out of memory: Killed process 34090 (wireplumber) total-vm:421940kB, anon-rss:0kB, file-rss:3000kB, shmem-rss:0kB, UID:1000 pgtables:168kB oom_score_adj:200
Jun 06 12:53:22 sparky systemd[34071]: wireplumber.service: Failed with result 'oom-kill'.
Jun 06 12:53:22 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
Jun 06 12:53:22 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:345
Jun 06 12:53:24 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
Jun 06 12:53:51 sparky kernel: tokio-rt-worker invoked oom-killer: gfp_mask=0x402cc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM|__GFP_NOWARN), order=0, oom_score_adj=-1000
Jun 06 12:53:51 sparky kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/session.slice/wireplumber.service,task=wireplumber,pid=35847,uid=1000
Jun 06 12:53:51 sparky kernel: Out of memory: Killed process 35847 (wireplumber) total-vm:421964kB, anon-rss:0kB, file-rss:3240kB, shmem-rss:0kB, UID:1000 pgtables:164kB oom_score_adj:200
Jun 06 12:53:52 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
Jun 06 12:53:52 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:345
Jun 06 12:53:53 sparky kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
Jun 06 12:57:33 sparky kernel: dbus-daemon invoked oom-killer: gfp_mask=0xdc0(GFP_KERNEL|__GFP_ZERO), order=0, oom_score_adj=-900

Once system goes into slow creep mode, only option to exit is hard reboot.

I finally figured out to have earlyOOM to catch at 2% memory that solves things.
Additional settings that work:

sudo tee /etc/default/earlyoom << 'CONF'
EARLYOOM_ARGS="-m 2 -s 100 --prefer vllm|python3|python|llama-server|triton --avoid sshd|systemd|journald|NetworkManager|dropbear|dockerd|containerd"
CONF

For large model inference / ft / benchmarking regular PyTorch methods of cleaning CUDA memory also don’t work fully. After an OOM occurs that even earlyOOM catches, one need to reset paged memory to get memory fully freedup to reattempt loading the model. For this I am now using this to clean before every large model experiment:

docker run --rm --privileged alpine sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

Please see the article link for all the settings I had to change to get DGX Spark in stable state and stop crashing for large model fine tuning and inference testing. Hardening Your DGX Spark for AI Workloads - Geeta

System details:

Platform NVIDIA DGX Spark, GB10
GPU NVIDIA GB10, Bus-Id 0000000F:01:00.0
Driver 580.159.03 (NVIDIA UNIX Open Kernel Module, aarch64), Release Build, built 2026-04-24
CUDA 13.0
Kernel 6.17.0-1018-nvidia, aarch64, PREEMPT_DYNAMIC (#18-Ubuntu, 2026-05-05)
OS Ubuntu 24.04.4 LTS
NVIDIA modules nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset (driver 580.159.03)
GPU module build gcc 13.3.0
38

What do you mean by “slow creep memory mode”? The system is designed to kill a process that uses too much memory so that the system does not crash. From your log it looks like it worked

sparky kernel: Out of memory: Killed process 35847 (wireplumber) total-vm:421964kB, anon-rss:0kB, file-rss:3240kB, shmem-rss:0kB, UID:1000 pgtables:164kB oom_score_adj:200

39

slow creep memory mode – where system takes 10min or longer to respond to any simple command like ls or if trying to ssh or anything. most of the shells hangs and if any survive by any chance it doesn’t respond. So only option is to hard boot the DGX spark. Makes it very difficult to do when working on headless remote SSH mode.