Recovery from kernel panic after dashboard update — unable to mount root fs on unknown-block(0,0)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
1

My Spark went unbootable after a routine “Update Now” through the DGX Dashboard. Symptoms and fix below.

Symptoms

  • Initiated update through DGX Dashboard

  • System rebooted automatically when the update “finished”

  • Boot landed at: Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

  • No GRUB menu appeared at any point — system goes straight from POST to the panic

  • LAN ping and Tailscale both unreachable; the box was fully off the network

  • Hardware was fine — POSTed normally, UEFI worked, NVMe detected in boot order

What actually broke

The kernel update from 6.17.0-1008-nvidia to 6.17.0-1014-nvidia got far enough to install the kernel image but the post-install script bailed out before generating the initramfs. The kernel package reported a CRITICAL error, but the dashboard triggered the reboot anyway. The result on disk:

/boot/vmlinuz-6.17.0-1014-nvidia       ✅
/boot/config-6.17.0-1014-nvidia        ✅
/boot/System.map-6.17.0-1014-nvidia    ✅
/boot/initrd.img-6.17.0-1014-nvidia    ❌ missing

GRUB’s symlink pointed at the missing initrd, kernel had no way to find root, panic.

The dpkg state of the kernel package was iF (installed but failed to configure), and the failure was DKMS exiting non-zero because the NVIDIA modules were already present at the target version and DKMS refused to rebuild without --force.

Why you can’t recover this remotely

/etc/default/grub ships with GRUB_TIMEOUT=0 on systems set up in headless / network-appliance mode. With zero timeout there’s no menu, no opportunity to fall back to an older kernel from the boot loader. Holding Shift or Esc doesn’t help — there’s no menu to interrupt. GRUB_RECORDFAIL_TIMEOUT did not engage for me after multiple failed boots either.

Result: even though older working kernels are sitting right there in /boot, you can’t reach them without physical access.

Recovery (what worked for me)

You’ll need:

  • An ARM64 Linux live USB. The Spark is aarch64; standard amd64 ISOs will not boot. I used Ubuntu Server 26.04 LTS arm64 (ubuntu-26.04-live-server-arm64.iso).

  • A USB-C flash drive, or a USB-A stick plus a USB-C adapter — the Spark has no USB-A ports.

  • A keyboard and monitor.

Boot from the USB:

UEFI won’t enumerate a USB device for boot ordering unless it’s plugged in when you enter UEFI. Plug the USB stick in before powering on, then mash Esc/Del to enter UEFI, set USB Hard Drive as first boot, save and exit. This boots to the Subiquity installer…

IMPORTANT: DO NOT RUN THE INSTALLER! YOU WILL WIPE YOUR PARTITION IF YOU DO.

Instead, get to a shell: Ctrl+Alt+F2. Login as ubuntu with no password.

Mount the broken installation and chroot in:

bash

sudo -i
mount /dev/nvme0n1p2 /mnt/spark
mount --bind /dev /mnt/spark/dev
mount --bind /dev/pts /mnt/spark/dev/pts
mount --bind /proc /mnt/spark/proc
mount --bind /sys /mnt/spark/sys
mount --bind /run /mnt/spark/run
mount /dev/nvme0n1p1 /mnt/spark/boot/efi
chroot /mnt/spark /bin/bash

Generate the missing initramfs:

bash

update-initramfs -c -k 6.17.0-1014-nvidia

(Replace the version with whatever’s missing in your case — compare ls /boot/vmlinuz-* with ls /boot/initrd.img-* to identify which kernel has no initrd.)

Clear the dpkg iF state. This is what tripped me up — the package’s postinst keeps failing because DKMS sees its modules already installed and exits non-zero. Force DKMS past the check:

bash

dkms autoinstall --force -k 6.17.0-1014-nvidia
dpkg --configure -a

Update GRUB:

bash

update-grub

Fix the GRUB timeout while you’re in here, or this can happen to you again:

bash

nano /etc/default/grub

Set:

GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=5

Then:

bash

update-grub

Exit and reboot:

bash

exit
umount -l /mnt/spark/sys
umount /mnt/spark/proc
umount /mnt/spark/dev/pts
umount /mnt/spark/dev
umount /mnt/spark/boot/efi
umount /mnt/spark
reboot

Pull the USB stick out as it powers down so you don’t boot the installer again.

Hold the kernel afterward

To avoid having the dashboard re-trigger the same broken transition before NVIDIA addresses it:

bash

sudo apt-mark hold linux-image-6.17.0-1014-nvidia
sudo apt-mark hold linux-image-nvidia-hwe-24.04

Verify with apt-mark showhold. Unhold (apt-mark unhold) when you’re ready to attempt the update again, and consider running it via terminal apt upgrade rather than the dashboard so you can see error output and defer the reboot if anything looks wrong.

Two things that will save your time if this hits you

  1. Have an ARM64 Linux live USB on hand before you need it. Building one when your dev box is already a brick is annoying. Make one now and stash it.

  2. Edit /etc/default/grub to set a non-zero GRUB_TIMEOUT on a working system, today. If you’ve never touched it, it’s almost certainly 0 and you have no remote recovery path if a future kernel update goes sideways. Set it to 5, run sudo update-grub, done.

Hope this helps someone.

2

Thank you for the detailed instructions. Helped a lot after today’s apt update/upgrade. By some reason standard GRUB menu still not visible but at least system boot up and working.

Windows users can burn standard ubuntu arm64 live image with well-known Rufus tool.

If NVIDIA team monitors topics, it would be nice to have “Terminal” menu option in the Rescue Disk image. BTW, it fails to create on Windows 11 OS at the USB disk format step.

3

Having a rescue mode would indeed be a great addition…

Tru

4

This is a very interesting failure and something we have not seen before. The latest update upgrades the kernel to 6.17.0-1018-nvidia so you should not need to hold the older image anymore.

5

The problem roots may be different. In my case it was also failed compilation of some driver part or something like that, so upgrade was incomplete. Next apt autoremove seems broke the links so system become unbootable.

6

I just did an update now and I’m having this same issue. I had to reflash when I first got it too so I wonder if there is just a bad batch out there? I’m starting to be afraid to hit the update button when I see it.

7

I got trapped between 1014 and 1018, I followed your instructions and got back into the UI, but my drivers weren’t loading, I followed claude down a path to resign them and I’m back up.

The only thing I had to do that you didn’t have down was the mkdir /mnt/spark

Thanks for putting this up! I would have had to reimage again.

8

I have 2x Asus Ascent GX10, this just happened to both on the same update.

Painful.

9

Thank you for this excellent post.

10

Thank you for this recovery guide, this kernel panic after upgrade just happen to me also on the FE !

11

Holding the Shift button AND the Insert button during boot will bring up the GRUB menu. From there, you can select “Advanced options for DGX OS GNU/Linux” → and select the previous installed Kernel to boot and then you can do system recovery running command such as “sudo apt full-upgrade” or “sudo apt dist-upgrade”

12

This is still happening. Just updated now from the dashboard and got the kernel panic message. I am on 1029 version. Seems like NVIDIA has not yet fixed this issue.

13

Happened to me too just this week.

14

Thank you! This saved me today on 8/23/26

16

Follow-up from the OP: root cause found, and a way to stop it recurring

I’ve been holding 6.17.0-1014 since May. Today I dug through the apt terminal log from the original failure (/var/log/apt/term.log.*.gz) and found the actual mechanism, then did a clean upgrade to 7.0.0-1019-nvidia without a live USB. Posting in case it saves someone else the chroot dance.

What actually happens

It is not the initramfs step failing. The kernel package runs its hooks with run-parts --exit-on-error, and /etc/kernel/postinst.d/dkms sorts before /etc/kernel/postinst.d/initramfs-tools. The dkms hook fails, run-parts stops, and the initrd is never generated. GRUB then gets a menu entry with no initrd line.

The dkms hook fails because of an architecture naming mismatch that only exists on ARM:

  • The nvidia-dkms-595-open postinst builds the module with ARCH=$(dpkg --print-architecture), which is arm64. Its postinst even prints WARNING: unsupported arch: arm64 and continues.
  • The kernel hook runs dkms autoinstall, which uses uname -m, which is aarch64.

dkms keys builds by kernel+arch, finds no aarch64 build, compiles again, then refuses to install over the arm64-labelled modules that are already sitting in /lib/modules/<ver>/updates/dkms/:

Autoinstall of module nvidia/595.84 for kernel 7.0.0-1019-nvidia (aarch64)
Building module(s)................... done.
Module /lib/modules/7.0.0-1019-nvidia/updates/dkms/nvidia.ko.zst already installed at version 595.84, override by specifying --force
...
Error! Installation aborted.
run-parts: /etc/kernel/postinst.d/dkms exited with return code 1
dpkg: error processing package linux-image-7.0.0-1019-nvidia (--configure):

That is why dkms autoinstall --force from a chroot “fixes” it, why it comes back on every kernel, and why holding the image package doesn’t help. On x86 both names normalize to x86_64, so nobody upstream sees it. The postinst in 595.84 still has the bug.

How to verify this yourself

The chain is visible in two scripts. Line numbers are from dkms 1:3.4.3-1ubuntu1 (/usr/sbin/dkms) and the nvidia-dkms-595-open 595.84 postinst.

  1. The nvidia postinst has a case on dpkg --print-architecture that maps amd64 to x86_64 and i386 to i686. There is no arm64 branch; it falls to the default, prints WARNING: unsupported arch: arm64, and passes arm64 through to /usr/lib/dkms/common.postinst, which calls dkms build/dkms install with -a arm64.
  2. When no -a is given, dkms defaults the arch to uname -m on Debian/Ubuntu (line 418). The kernel hook /etc/kernel/postinst.d/dkms runs dkms autoinstall --kernelver <ver> with no -a, so it operates as aarch64.
  3. dkms keys its tree and its “already installed?” check on arch: builds live under /var/lib/dkms/<module>/<version>/<kernel>/<arch> (lines 1461, 1775), and autoinstall calls _is_module_installed with that arch (around line 3025). Looking for aarch64, it finds nothing under arm64/, concludes the module needs installing, and builds it again.
  4. The install step then finds the arm64-built file already in /lib/modules/<ver>/updates/dkms/, compares versions, gets “equal”, and without --force prints already installed at version ... override by specifying --force and returns 1 (line 1117). That non-zero exit is what run-parts --exit-on-error stops on.

Physical corroboration: on my machine, after the May chroot --force recovery, /var/lib/dkms/nvidia/595.58.03/6.17.0-1014-nvidia/ contained both an arm64/ and an aarch64/ directory, and dkms status listed the kernel twice, once per label. Two differently-labelled builds of the same module for the same kernel is exactly what the mismatch produces.

On x86 step 1 maps to x86_64, which is also what uname -m returns, so the lookup in step 3 succeeds and autoinstall skips the module. That’s why this only bites ARM.

The fix: stop using dkms for the nvidia modules

Canonical publishes prebuilt, signed nvidia modules for the -nvidia kernel flavour, and the meta package satisfies the driver’s dkms dependency. With dkms out of the picture the hook has nothing to do and the initrd gets built normally.

sudo apt-mark unhold linux-image-nvidia-hwe-24.04 linux-image-<your-held-kernel>
sudo apt-mark manual nvidia-driver-595-open linux-nvidia-hwe-24.04
sudo apt install linux-nvidia-hwe-24.04 linux-image-nvidia-hwe-24.04 linux-headers-nvidia-hwe-24.04 \
    linux-modules-nvidia-595-open-nvidia-hwe-24.04 \
    nvidia-dkms-595-open- nvidia-dkms-590-open- nvidia-driver-590-open-

(Trailing - means remove. Drop the 590 entries if you don’t have that series installed. Adjust 595 if you’re on a different driver series; check apt-cache policy linux-modules-nvidia-<series>-open-nvidia-hwe-24.04 first.)

If a kernel is already stuck half-configured (dpkg -l | grep '^iF'), the same command recovers it in place. Removing nvidia-dkms re-arms the kernel trigger, which then runs clean.

Before you reboot, verify, every time:

ls -l /boot/initrd.img-$(ls /boot | grep -oP 'vmlinuz-\K.*nvidia' | sort -V | tail -1)
sudo grep -n "initrd" /boot/grub/grub.cfg | grep <new-kernel-version>
dpkg -l | grep -E '^(iF|iU)'      # must print nothing

Other notes

  • Do the update from a terminal, not the dashboard. The dashboard runs the same apt upgrade but hides the error that tells you the initrd wasn’t generated.
  • The hold did not stop the new kernel arriving. linux-modules-nvidia-fs-nvidia-hwe-24.04 and linux-tools-nvidia-hwe-24.04 pulled linux-image-7.0.0-1019-nvidia in as a dependency anyway. Only removing dkms makes it safe.
  • If you previously recovered via chroot, dkms status will show a duplicate aarch64 entry for that kernel. It’s harmless and disappears when nvidia-dkms is removed.
  • After the switch, modinfo nvidia shows signer: Canonical Ltd. Kernel Module Signing, so Secure Boot is fine without the MOK key.

Result here: 6.17.0-1014 to 7.0.0-1019-nvidia, driver 595.84, GPU and docker --gpus all working, no live USB. The real fix belongs in the nvidia-graphics-drivers-595 postinst (map arm64 to aarch64 the way amd64 is mapped to x86_64); I’ll file that on Launchpad.