DGX Spark kernel panic after OOBE/update – missing initrd for 7.0.0-1019-nvidia and DKMS arm64/aarch64 conflict

Hi,

I’m reporting an issue I encountered on a brand-new NVIDIA DGX Spark during the initial setup / OOBE process.

The system completed the initial setup and package update, then rebooted. On the next boot it failed with a kernel panic:

VFS: Unable to mount root fs on unknown-block(0,0)

The machine also spent a long time attempting PXE boot before eventually reaching the failed local boot path.

I disabled PXE in UEFI and booted an Ubuntu ARM64 live USB to inspect the internal installation.

The installed root filesystem was intact, but I found that the newly installed kernel:

7.0.0-1019-nvidia

had:

/boot/vmlinuz-7.0.0-1019-nvidia

but was missing:

/boot/initrd.img-7.0.0-1019-nvidia

The previous kernel:

6.17.0-1014-nvidia

still had both kernel and initrd and could boot successfully.

After chrooting into the installed system, dpkg showed the new kernel package in a partially configured state.

The relevant package was:

linux-image-7.0.0-1019-nvidia

and configuration was failing in the DKMS post-install path.

The NVIDIA driver stack installed by the update was:

nvidia-driver-580-open 580.178.04
nvidia-dkms-580-open 580.178.04

What looked suspicious is that the NVIDIA DKMS package postinst and the kernel DKMS hook were using different architecture strings.

The NVIDIA package postinst used Debian architecture:

arm64

while the kernel postinst DKMS path was operating with:

aarch64

On this system, those refer to the same architecture family, but DKMS appeared to treat them as distinct bookkeeping entries.

During repair I saw the kernel postinst fail with a message equivalent to:

Module nvidia/580.178.04 already installed on kernel 7.0.0-1019-nvidia (aarch64)
Use --force to override

while the NVIDIA DKMS package had previously built/registered the module under arm64.

This caused the kernel package configuration to fail and, in the failed OOBE state, the initramfs for the new kernel had not been created.

The immediate recovery steps were:

sudo update-initramfs -c -k 7.0.0-1019-nvidia
sudo update-grub

and then forcing DKMS for the kernel:

sudo dkms autoinstall --force -k 7.0.0-1019-nvidia

After that:

sudo dpkg --configure linux-image-7.0.0-1019-nvidia

completed successfully.

The system could then boot kernel:

7.0.0-1019-nvidia

However, to avoid the same DKMS issue recurring, I later migrated away from DKMS and installed the precompiled kernel modules instead:

linux-modules-nvidia-580-open-7.0.0-1019-nvidia
linux-modules-nvidia-580-open-nvidia-hwe-24.04

and removed:

nvidia-dkms-580-open

I also removed stale DKMS modules from:

/lib/modules/7.0.0-1019-nvidia/updates/dkms

so that modinfo now resolves to the precompiled Canonical-signed NVIDIA module.

Current working state:

Kernel: 7.0.0-1019-nvidia
NVIDIA driver: 580.178.04
CUDA: 13.0
GPU: NVIDIA GB10
DKMS status: empty
dpkg --audit: clean

The system is now booting normally.

The reason I’m posting is that this happened during the initial OOBE/update flow on a new DGX Spark, without any manual kernel modification before the failure.

The sequence strongly suggests that the failed kernel post-install / DKMS architecture bookkeeping conflict left the new kernel without a valid initrd, which then caused the root filesystem mount panic on reboot.

Has anyone else seen this on DGX Spark with the 7.0.0-1019-nvidia kernel and NVIDIA 580.178.04?

In particular, I’d be interested to know whether the arm64 vs aarch64 DKMS handling is a known issue in the current Spark OTA stack, and whether NVIDIA recommends using the precompiled linux-modules-nvidia-* packages instead of nvidia-dkms-* on DGX Spark.

I can provide the full dpkg, DKMS, initramfs, and boot logs if useful.

Thanks.

Thank you for reporting, we are currently looking into this issue

Updates are rolling out to mitigate this issue and can take up to several hours to appear in some regions

Title:

Confirmed: DKMS arm64/aarch64 conflict still breaks boot on 7.0.0-1019 — a subsequent OTA update did not fix it (GB10, driver 580.178.04)

Body:


Prior history, and why I am posting this: this unit was bought less than a year ago and has already been out of service several times, each time following an update. On top of that, and predating all of the September issues, it has repeatedly run hot and shut itself down without warning. I do not know whether the thermal shutdowns are connected to the kernel and driver problems described below, or whether they are a separate hardware or firmware issue — but between the two, this machine has spent a significant part of its short life unusable.

I am documenting the full diagnosis and fix below because it took a full working day to recover, the information needed was scattered across several threads, and one part of it was not documented anywhere. I would also like to know whether anything in the recent updates is expected to address the spontaneous shutdowns, and I hope they do not recur.

System: DGX Spark, 4 TB (SAMSUNG MZALC4TOHBL1-OOB07), Secure Boot enabled, DGX OS with nvidia-driver-580-open / nvidia-dkms-580-open 580.178.04-0ubuntu0.24.04.1.


Timeline

After a DGX Dashboard update the machine first lost the GPU driver only. The Dashboard showed GPU Telemetry Error: failed to run nvidia-smi command: exit status 9, nvidia-smi failed, and lsmod | grep nvidia returned nothing. The system was otherwise usable.

dkms status showed duplicate entries for the two newest kernels, one per architecture label:

nvidia/580.178.04, 6.17.0-1031-nvidia, arm64: installed
nvidia/580.178.04, 6.17.0-1032-nvidia, aarch64: built
nvidia/580.178.04, 6.17.0-1032-nvidia, arm64: installed
nvidia/580.178.04, 7.0.0-1019-nvidia, aarch64: built
nvidia/580.178.04, 7.0.0-1019-nvidia, arm64: installed

Older kernels had a single arm64 entry each and were unaffected. dpkg had linux-image-6.17.0-1032-nvidia and linux-image-7.0.0-1019-nvidia in iF state, both failing in the DKMS post-install hook with already installed at version 580.178.04, override by specifying --force.

On the next reboot the machine stopped at:

KERNEL PANIC! VFS: Unable to mount root fs on unknown-block(0,0)

Cause confirmed: /boot/vmlinuz-7.0.0-1019-nvidia existed, /boot/initrd.img-7.0.0-1019-nvidia did not. All eight older kernels had both files.


The part I have not seen reported

I then ran another OTA update from the Dashboard, hoping it would correct the state. It did not. After that update, initrd.img-7.0.0-1019-nvidia was still missing and the machine still panicked on reboot.

The broken dpkg state persists across updates, because any apt operation tries to configure the pending kernel packages first and hits the same DKMS failure. Users already stuck cannot update their way out of this.


Recovery without a chroot

Ctrl+Alt+F2 from the Ubuntu Server live USB did not give me a shell on this machine. What worked instead was the GRUB command line, which needs no chroot and no live filesystem.

Boot an ARM64 live USB, press c at its GRUB menu, then:

ls
ls (hd0,gpt2)/boot/
linux (hd0,gpt2)/boot/vmlinuz-6.17.0-1032-nvidia root=/dev/nvme0n1p2 ro
initrd (hd0,gpt2)/boot/initrd.img-6.17.0-1032-nvidia
boot

That booted the installed system on the last good kernel, with the desktop and all data intact.

Note: Force boot to Grub menu = Enabled in UEFI did not produce a GRUB menu on this machine. /etc/default/grub.d/no-grubmenu.cfg is present and appears to override it, which is why the local GRUB menu was never reachable and a USB was required at all.


Repair, from the running system

sudo update-initramfs -c -k 7.0.0-1019-nvidia
sudo dkms autoinstall --force -k 7.0.0-1019-nvidia
sudo dkms autoinstall --force -k 6.17.0-1032-nvidia
sudo dpkg --configure -a
sudo update-grub

The --force is needed once per affected kernel. After this the machine booted on its own and dpkg -l | grep -E '^(iF|iU)' was empty.


The GPU was still dead at this point

modprobe: ERROR: could not insert 'nvidia': Key was rejected by service

Secure Boot was rejecting the DKMS-built modules, signed with the local MOK key at /var/lib/shim-signed/mok/MOK.priv, which is not enrolled in firmware.


Permanent fix: drop DKMS for the prebuilt signed modules

sudo apt install linux-modules-nvidia-580-open-nvidia-hwe-24.04 nvidia-dkms-580-open-

Result:

modinfo nvidia | grep signer
signer: Canonical Ltd. Kernel Module Signing

nvidia-smi
NVIDIA GB10 | Driver 580.178.04 | CUDA 13.0

Both problems are gone. With DKMS removed, the failing kernel hook no longer exists, so the initrd should be generated normally on future kernel updates.


Questions for NVIDIA

  1. Is the arm64 vs aarch64 DKMS bookkeeping mismatch a known issue in the current Spark OTA stack? It was analysed in detail on this forum on 11 September and was still breaking machines on 16 and 17 September.

  2. Does NVIDIA recommend linux-modules-nvidia-* over nvidia-dkms-* on DGX Spark? A DGX Spark reference package list posted here previously by NVIDIA staff contains the prebuilt modules and no DKMS package, which suggests the prebuilt modules are the intended configuration.

  3. Is having nvidia-dkms-580-open installed the factor that determines who breaks? One user reported updating two identical units with only one failing. If DKMS presence is the discriminator, users could check for it before updating, and that check should be published.

  4. Given that a later OTA did not repair the broken state, is a repair path planned for machines already stuck?

  5. Are the spontaneous thermal shutdowns reported on GB10 a known issue? Is anything in the recent kernel or firmware updates expected to address them? For a unit under a year old that has been repeatedly unusable after official updates, I would appreciate a clear answer on this.

Thank you

Isabel Fidalgo Carvalho

Those that are in a bad state due to kernel panic can try these steps to recover:

sudo apt update
  sudo apt-get install \
    linux-modules-nvidia-580-open-nvidia-hwe-24.04 \
    nvidia-dkms-580-open-the trailing 

Requests removal of nvidia-dkms-580-open in the same transaction. this is preferable to purging it separately because the LRM is installed as its replacement provider atomically.

and then to double-check:

  sudo dpkg --audit
  sudo apt-get check
  test -s /boot/initrd.img-7.0.0-1019-nvidia &&
      echo "7.0 initrd is present"

and reboot if all looks good

Thank you — that is the command that fixed this machine.

One addition: those steps need a shell, and a machine stopped at VFS: Unable to mount root fs cannot reach one. It first has to be booted on an older kernel that still has its initrd. Ctrl+Alt+F2 from the live USB gave no shell here; the GRUB command line did, with no chroot:

ls (hd0,gpt2)/boot/
linux (hd0,gpt2)/boot/vmlinuz-6.17.0-1032-nvidia root=/dev/nvme0n1p2 ro
initrd (hd0,gpt2)/boot/initrd.img-6.17.0-1032-nvidia
boot

Adjust partition and version to your own machine. Your commands then run normally.

Still open: are the spontaneous thermal shutdowns on GB10 a known issue, and is anything in the recent updates expected to address them? They predate the September kernel problems on this unit.

Thank you.

I’m am dealing with the kernel panic issue right now. I used a “live” Ubuntu Arm64 USB to boot. Exited to a terminal. From there I followed these instructions:

sudo su
mount /dev/nvme0n1p2 /mnt
mount /dev/nvme0n1p1 /mnt/boot/efi
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
mount --bind /run /mnt/run

chroot /mnt

“ls” shows the same situation as your post. 7.0.0-1019-nividia appears, but no matching initrd.img-7.0.0-1019-nvidia

I’m about to run:

update-initramfs -c -k 7.0.0-1019-nvidia

to build the missing initrd.img

followed by:

dpkg --configure -a
apt-get install -f
update-grub

to clean up.

Does this look like the correct? Am I overlooking anything?