First, I would run just after a reboot (as soon as the system is back up), sudo nvidia-bug-report.sh. This will generate an nvidia-bug-report.log.gz file in your working directory.
Another thing would be trying to run the code with sudo swapoff -a a training run that would otherwise freeze the system will instead cause the process to be killed by the OOM killer, allowing the OS to survive
I would also check the following threads first:
DGX Spark Device is getting shutdown by itself while i am running my LLM Fine Tunning project.
i am running Ubuntu 24 Version on it. once it get back on started again i have checked logs in it and i have found something which i am sharing
root@spark-1eaa:/home/administrator# echo “----- Slot Power -----”; \ sudo lspci -vv -s 01:00.0 | grep -i “SlotPowerLimit”; \ sudo lspci -vv -s 01:00.1 | grep -i “SlotPowerLimit”; \ sudo lspci -vv -s 0002:01:00.0 | grep -i “SlotPowerLimit”; \ sudo lspci -vv -…
Hi NVIDIA Team,
I would like to provide feedback regarding a crash issue on the DGX Spark.
Currently, I am using the system for LLM RL fine-tuning (TRL+GRPO+vLLM). I have noticed that if the process consumes all available memory, the system does not kill the process but instead crashes the entire OS.
ENV:
docker: nvcr.io/nvidia/vllm:25.09-py3
When this happens:
SSH becomes inaccessible.
The HDMI monitor goes black.
Mouse and keyboard lights go out.
I am forced to physically res…
During training on a DGX Spark, whenever the model encounters a sequence that exceeds available GPU memory, instead of raising a normal CUDA out of memory error and crashing the process, the entire machine becomes unresponsive.
SSH hangs, the node stops reacting, but external monitors (e.g., W&B) still show the process “alive” with no progress. Only a physical reboot recovers it.
Expected behavior
GPU OOM → process throws RuntimeError: CUDA out of memory → training crashes cleanly → machine s…