DGX spark keeps rebooting every 20-30 minutes

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I installed the updates on the DGX dashboard.

I was able to run a few demo code. However, every 20-30 mininutes, the DGX Spark device just decided to reboot on its own despite i have no workload running on it.

Any way that we can fix this? This does not seem to be normal.

Can anyone confirm if any of the following AI suggested ideas would work?
Because your system is likely in a bad state from the failed update, the official solution is to reflash the system using a recovery image.

  1. Get the Recovery Media: You will need to download the official DGX Spark recovery image from NVIDIA. This is enterprise hardware, so you should find this on your NVIDIA Enterprise Support portal or on the DGX Spark product support page.

  2. Follow NVIDIA’s Instructions: The NVIDIA Developer Forums (specifically the “DGX Spark / GB10” section) have threads on this. The official guidance is to reflash the system to restore it to a clean, working state.

  3. Contact NVIDIA Support: This is an expensive, enterprise-grade machine. Your fastest and safest path to a solution is to contact NVIDIA Enterprise Support directly. They are aware of this specific “boot loop” issue and will have the exact instructions and files you need.

2

I have had a lot of crashes. The DGX Spark crashes In a lot of different ways when you run out of memory. Often I will get a hard lock, where the mouse cursor just isn’t responsive. Other times, I will come back to a setup that has rebooted completely to the original user login.

If you run DGX Dashboard, do you see your memory running up? Even if your setup is idle, if you have out of control docker containers running with just health checks, you can run into issues if they aren’t constrained.

If you don’t have that issue, reinstalling from scratch is good.

The DGX Spark is too new and beyond the training cut off date for all LLMs and the boot loop was an issue in the first two days of release where you couldn’t boot into the system at all, not the rebooting scenario you describe.

3

@kylezheng04 Just a thought… If this is something happening at a deep system level you may want to have a look at log files in /var/log. Some of these files might shed some light on what’s causing the system to crash/reboot.

4
  1. Easy way: Use the recovery media to reflash it. I had to do few times. Each time, make sure to check DGX dashboard to update whatever available then do
sudo apt update
sudo apt dist-upgrade
sudo fwupdmgr refresh
sudo fwupdmgr upgrade
  1. Hard way: if you want to troubleshoot and don’t want it to restart automatically, reboot, go to BIOS/UEFI by pressing DEL key repeteadly. Advanced → Advanced → Watchdog. This automatically restart DGX when it detects somethings wrong. I think Somethings wrong is “subjective”. But still Turn it off at your own risk!
5

Nothing really works with the initial image.

The thing works is to start a new recovery image on a clean ubuntu 24.04 desktop PC. and then put that recovery system back to the DGX Spark internal SSD. This fixed the problem.

I was using 1016 linux kernel, has the 30mins rebooting issue. Then i switch to 1014, have the same issues. So i conclude that this would not work.

```Verify the running kernel with uname -r. If the system is on a kernel version known to cause instability (e.g., 1016), revert to a previous stable kernel or update to the latest DGX OS kernel via the GRUB menu or by reinstalling the latest DGX OS image.```

I got message from the nvidia customer service.

So i follow the instruction System Recovery — DGX Spark User Guide and rebuild the image.

Everything works fine after that.

I check the linux version on the one that works, it is 1013 version.

6

Hi!

I have similar experiences. After the first initialization and the first kernel updates arrived, the host starts a reboot in every 20 min.

The root cause: The watchdog kernel module was not loaded with the system.

  • linux-generic meta package was missing in the stock Spark [Ubuntu] OS, so the kernel headers were not updated
  • After the kernel update, aside from the previous kernel, it has stuck a blacklist file:
    • grep -r “blacklist.*sbsa” /etc/modprobe.d/ /lib/modprobe.d/
    • /lib/modprobe.d/blacklist_linux_6.8.0-106-generic.conf:blacklist sbsa_gwdt
  • So you need to correct this:
    • sed -i ‘s/^blacklist sbsa_gwdt/#blacklist sbsa_gwdt/’ /lib/modprobe.d/blacklist_linux_6.8.0-106-generic.conf
    • apt install linux-generic # if you dont have
    • modprobe sbsa_gwdt # manual kernel module load
    • update-initramfs -u
    • reboot
    • Test:
      • lsmod | grep sbsa_gwdt
      • Expected result: sbsa_gwdt 20480 1

Regards,
Z

7

@zocek what Spark are you using? On the FE the latest kernel is 6.17.0-1008-nvidia and the sbsa module is definitely not blacklisted.

elsaco@spark1:~$ cat /lib/modprobe.d/blacklist_linux-nvidia-6.17_6.17.0-1008-nvidia.conf
# Kernel supplied blacklist for linux-nvidia-6.17 6.17.0-1008-nvidia arm64
# modprobe.d/common.conf
# LP:1434842 -- disable OSS drivers by default to allow pulseaudio to emulate
blacklist snd-mixer-oss
blacklist snd-pcm-oss
blacklist coresight_etm4x
# Autogenerated watchdog blacklist
blacklist airoha_wdt
blacklist alim7101_wdt
blacklist apple_wdt
blacklist arm_smc_wdt
blacklist armada_37xx_wdt
blacklist bd9576_wdt
blacklist bd96801_wdt
blacklist cadence_wdt
blacklist cros_ec_wdt
blacklist da9052_wdt
blacklist da9055_wdt
blacklist da9062_wdt
blacklist da9063_wdt
blacklist dw_wdt
blacklist gpio_wdt
blacklist hpwdt
blacklist i6300esb
blacklist imx2_wdt
blacklist imx7ulp_wdt
blacklist imx_sc_wdt
blacklist keembay_wdt
blacklist kempld_wdt
blacklist marvell_gti_wdt
blacklist max63xx_wdt
blacklist max77620_wdt
blacklist mena21_wdt
blacklist menf21bmc_wdt
blacklist menz69_wdt
blacklist meson_gxbb_wdt
blacklist meson_wdt
blacklist mlx_wdt
blacklist mtk_wdt
blacklist nic7018_wdt
blacklist of_xilinx_wdt
blacklist pcwd_pci
blacklist pcwd_usb
blacklist pm8916_wdt
blacklist pretimeout_panic
blacklist qcom-wdt
blacklist rave-sp-wdt
blacklist renesas_wdt
blacklist retu_wdt
blacklist rn5t618_wdt
blacklist rti_wdt
blacklist rza_wdt
blacklist rzg2l_wdt
blacklist rzn1_wdt
blacklist rzv2h_wdt
blacklist s32g_wdt
blacklist sl28cpld_wdt
blacklist softdog
blacklist sp805_wdt
blacklist sprd_wdt
blacklist stm32_iwdg
blacklist stpmic1_wdt
blacklist sunxi_wdt
blacklist tegra_wdt
blacklist twl4030_wdt
blacklist visconti_wdt
blacklist wdat_wdt
blacklist wdt_pci
blacklist wm831x_wdt
blacklist wm8350_wdt
blacklist xen_wdt
blacklist xilinx_wwdt
blacklist ziirave_wdt

Also, the metapackage linux-generic is not part of DGX OS. It’s easy to mess up you system when adding extra packages.

From linux-generic package description: This package will always depend on the latest complete generic Linux kernel and headers and you might not want that on your Sparks!

DGX OS is using the linux-image-6.17.0-1008-nvidia currently.