NVIDIA DGX Spark continuously freezing, hanging, and or rebooting

Redirect here from customer care for consumer products.

Day 1 of issue: Powered on NVIDIA DGX Spark, connected a Logitech Unifying USB-A adapter to Anker USB-C hub to use a new Logitech Mouse. Within ~15 minutes, OS froze and I had to reboot the Spark. Rebooting the Spark froze on the login screen when attempting to log into local account.

Disconnected the Logitech Unifying USB-A adapter from the Anker USB-C hub and using a Logitech keyboard (using an older Logitech Unifying USB-A apadter) allowed me to log into the spark, but it immediately became difficult navigating the UI using hotkeys, tabs, etc. Plugged the new Logitech Unifying adapter to use the new Logitech Mouse back into the USB hub worked for a while to run OS updates, but after installing the latest updates for Spark’s OS. The host kept restarting.

Removing the Logitech Unifying adapter from hub did not resolve the issue, so I removed the Anker USB-C hub. Found a Dell keyboard (USB-A), connected it to an Apple USB-A to USB-C dongle allowed me to enter UEFI boot menu and GRUB, but the device kept on freezing and rebooting. I noticed two events where the boot-up sequence threw a kernal panic, but it rebooted to quickly for me to capture the error message.

The reboots triggered verbose outputs on the top left corner, and then “outputted a verbose audit log”. I noticed that the xHCI/USB runtime-PM kept appearing, so I tried to boot the Spark without any USB hubs or cables connected. While that worked, The OS booted as far as the user login screen, but without the field for me to enter my account password. I plugged in the Dell keyboard connected to an Apple USB-A to USB-C dongle, and restarted the Spark, only to end up seeing that verbose audit log.

I followed the system recovery steps (https://docs.nvidia.com/dgx/dgx-spark/system-recovery.html) and attempted to restore the Spark back to out-of-the-box (OOTB) setup, but the Spark kept on rebooting after copying just over 32GB of install files. Having spent more than 10 hours; a few different external USB drives to store the system to factory defaults, and the limited number of usable keyboards and mice; I only managed to get the Spark to complete the OS recovery installation twice. And each of those successful installation ended up failing because the Spark continued to reboot itself because of the USB issue.

I tried to suspend or delay the USB-runtime via GRUB (usbcore.autosuspend=-1), but was unable to reboot the Spark (i.e. ctrl+x or F10). And the only GRUB command that allowed me to reboot the Spark was entering ‘normal’. However, ‘normal’ reverts the change that I was trying to apply to suspend the USB-runtimes.

Day 2 of issue: Customer Care Representative recommended that I run the Field Diagnostics, but currently not possible.

Unable to boot past the UEFI menu. I had to create a boot path because it was not added, created the boot path to boot into the USB recovery image. And running the recovery process on first attempt successfully installed the OS and brought me through the OOTB setup–for Day 2’s first attempt. However, typing any keys from the keyboard froze and rebooted the Spark.

Now, I cannot get past either UEFI or grub menu without the device rebooting itself.

Caveat, there were a number of other troubleshooting steps taken, but not significant to include in this post. Other attempts included adjusting UEFI settings or editing configurations via GRUB resulted in the same issue (e.g. freeze, kernel panic, reboots, etc).

Any recommendations to resolve this issue? Or is this issue “fatal” and warrants an RMA?

Plug out all peripherals and boot the machine. When you get to the login screen, can you remotely login via SSH from another machine? If so, please generate and share an nvidia-bug-report by running nvidia-bug-report

TL;DR, That attempt was unsuccessful.

Context; I had attempted that on Day 1, before I opted to erase and reinstall the OS. The Spark just kept on rebooting at a few points;

  1. NVIDIA logo (normal size) with buffering circle,
  2. NVIDIA logo (stretch or large size) with buffering circle,
  3. or freezing at logon screen, requiring a manual and hard-power cycle

Delayed entry.

On Day 3, I noticed a few behaviors;

From cold-start the Spark can boot the Recovery software from the external USB drive–without any issues; recovery software can run its process (from start, to completion) without being interrupted by a triggered reboot (observed on Day 1).

However, once presented with the initial setup screen, the Spark freezes and reboots endlessly. And after reboot X number of times, the boot process throws a kernel panic. This behavior occurred post-OS recovery and regardless if

  1. I did or did not have a USB-C hub or adapter connected–after OS recovery.
  2. And whether or not I pressed any keys on the keyboard for the Spark to detect a connected keyboard.

Other observations.

When the Spark is powered on for more than ~15 minutes, the undercarriage at and around the USB-C power port feels hot to the touch.

Apple USB-A to USB-C adapters; (using three (3) in total) these adapters–the USB-C section–only, feels hot to the touch. The temperature of the OEM USB-C power connector feels room temp.

Other notes;

Prior to experiencing any issues; I had an Anker USB-C hub connected to the USB-C port closest to the HDMI port. And only two devices were wirelessly connected to that hub;

  1. Logitech Unifying Adapter (older model) that wirelessly connects a Logitech keyboard
  2. Logitech Unifying Adapter (newer model) that wirelessly connects a Logitech mouse.

Spark is plugged into a Belkin six (6) outlet surge-protected power strip. I have tested the outlet for nominal performance using multimeter and ground tester.

Current state of the Spark

Having ran the recovery software (from the USB external drive) multiple times, I can only access UEFI boot menu or GRUB.

Part A. Cannot create boot path for the Spark to locally boot from its onboard drive due to either issues;

  1. Recovery software fails to complete the recovery process (reboots mid-way through recovery process, then completes the recovery process upon reboot).
    1. Presumably, this is not intended behavior, therefore the recovery process failed the first time, and the second time was a false positive (?).
  2. Recovery software completes the recovery process, but UEFI cannot locate the NVMe EFI System Partition or the firmware just cannot see it.
    1. Another observation; when this occurs, UEFI can read the recovery software boot paths from the USB drive, but not the *boot paths that should have been available locally on NVMe.

Part B. Conversely, when the recovery process does complete in one operation, UEFI can read the boot paths on NVMe. However, when booting up from powering on the Spark or explicitly from UEFI or GRUB menu (selecting DGX OS) the DGX OS does not load.

Visually, I see the boot up logo in one of two sizes; NVIDIA logo and the buffering circle, scaled to normal size OR stretch/expanded size. Thereafter the Spark goes into a boot-loop.

*Boot paths:

  • \EFI\ubuntu\grubaa64.efi
  • \EFI\ubuntu\shimaa64.efi

Notables. I created and installed the recovery software on two (2) different drives;

  1. SansDisk thumb drive (USB-A connected to Apple USB-A to USB-C adapter)
  2. Samsung T5 external drive (USB-C to USB-C)

Observations. Recovery process had better stability using SansDisk thumb drive from a cold-start. If the Spark was powered on for more than ~15 minutes, onboard temperatures affected stability or integrity of the recovery process.

Stability and integrity of the recovery process using an external drive (USB-C to USB-C) had lesser stability–higher occurrances of reboots early or mid-way (~32GB of copied data) through recovery process. Software integrity was mostly unreliable, as noted in Parts A & B.

Thank you for the extra info. If you access the UEFI menu and go to Advanced/NVMe Configuration, Can you perform an extended self-test of the NVMe drive?

Done.

Short Device Selftest Result [ Pass ]
Extended Device Selftest Result [ Pass ]

Case was escalated and unit will be RMA’ed.