Latest update pulls in mis-matched kernal/driver

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I updated both my Nvidia DGX Spark and Dell Promax GB10 this morning. They both now display an error message on the DGX Dashboard page where the GPU/Memory indicators usually sit.

Error getting GPU telemetry: failed to run
nvidia-smi command: exit status 9. Reboot and try again.

I applied the recommended fix using Claude and the DGX dashboard is working correctly again now as a visual indicator that the system is working again.

Claudes diagnosis was:

I found the cause on the machine itself, and none of the forum threads describes this exact combination yet. I have not changed anything; the fix below is verified with an apt dry run only.

The cause

Secure Boot is rejecting the NVIDIA kernel module, so nothing can talk to the GPU. Running modprobe nvidia gives:

modprobe: ERROR: could not insert ‘nvidia’: Key was rejected by service

This morning’s Dashboard update moved the driver from 580.173.02 to 580.178.04 and pulled in kernel 7.0.0-1019. The pre-built, Canonical-signed module packages NVIDIA and Ubuntu publish are all still built for 580.173.02, so apt removed them and installed the DKMS package instead. DKMS then compiled its own modules and signed them with a brand new local key that is not enrolled in your firmware, which Secure Boot refuses. As a side effect the 7.0.0-1019 kernel package was left half-configured, which is why you are still booting 6.17.0-1029. Rebooting cannot fix any of this.

Recommended fix

Put the driver userspace back to 580.173.02, which is what you were running this morning, reinstall the signed module packages for both kernels, drop DKMS, and hold the driver until the matching 580.178.04 module packages appear.

V=580.173.02-0ubuntu0.24.04.1
P=“nvidia-driver-580-open nvidia-kernel-common-580 nvidia-kernel-source-580-open nvidia-utils-580 nvidia-compute-utils-580 xserver-xorg-video-nvidia-580 libnvidia-cfg1-580 libnvidia-common-580 libnvidia-compute-580 libnvidia-decode-580 libnvidia-encode-580 libnvidia-extra-580 libnvidia-fbc1-580 libnvidia-gl-580”
sudo apt-get install --allow-downgrades $(for p in $P; do echo “$p=$V”; done)
linux-modules-nvidia-580-open-6.17.0-1029-nvidia
linux-modules-nvidia-580-open-nvidia-hwe-24.04
nvidia-dkms-580-open-
sudo dpkg --configure -a
sudo apt-mark hold $P
sudo reboot

If the removal of the DKMS package errors, run sudo dkms remove nvidia/580.178.04 --all and repeat the first command. After the reboot, nvidia-smi should work and the Dashboard telemetry will return.

Two things to know about that reboot:

  • GRUB will boot 7.0.0-1019 by default. NVIDIA’s advisory only concerns multi-node NCCL over RoCE, and your RoCE ports are all down, so it does not affect you. If you would rather stay on 6.17.0-1029 until the advisory closes, pick it from “Advanced options” in the GRUB menu, or add the kho=off kernel parameter the advisory suggests.
  • Un-hold later. Check occasionally with the command below. When the Provides line shows 580.178.04, run sudo apt-mark unhold $P and let the Dashboard update again.

apt-cache show linux-modules-nvidia-580-open-nvidia-hwe-24.04 | grep Provides

Alternatives I would avoid

  • Enrol the DKMS key with mokutil --import and keep 580.178.04. It needs a monitor and keyboard at boot for the MOK screen, and NVIDIA staff on the forum say to stick with the modules the update process provides rather than DKMS on Spark.
  • Disable Secure Boot in UEFI. NVIDIA staff confirm it works but you lose the protection.

Sources:

  • DGX Spark Update Advisory
  • DGX Spark regression: kernel 7.0.0-1019 NCCL/RoCE, 6.17.0-1032 works
  • Apt upgrade has broken nvidia driver
  • DGX Spark – Unable to Load NVIDIA Drivers / Dashboard Updates Stuck
  • Reinstalling the NVIDIA driver on DGX Spark
  • Secure Boot Requirements
  • DGX Spark modprobe error
  • DGX Spark July 2026 software update issue
  • Trouble with DGX Dashboard Update (note.com)
2

I had similar problems after updating. I can’t run sparkrun, I always got the error: Failed to init nvsandboxutils and “failed to generate CDI spec: failed to create device CDI specs: failed to construct device spec generators: failed to initialize NVML: Driver Not Loaded”. Hope I will get it running again.

4

Yes, it is a know issue:

5

I hat the same and I don’t know why they can’t stop the update request. I assume that several 1000 DGX will have a problem.

6

Thanks, I read that, but to be honest, the language is somewhat indecipherable (‘affecting multi-node NCCL/RoCE workloads’ ) and implies it’s only affecting multi-machine configurations. Maybe I’m just being dumb, though.

Of course, I only read that after I did the updates and went looking to see if there was anything in the forum. Anyway, no real harm done; they are both working ok right now for me without the update.

7

The whole situation is very frustrating, I’ve seen NVIDIA staff recommend “kho=off” on the bootline and proceed with the 7.0.0 kernel.

However, I’ve then seen real-world experiences of others saying it doesn’t work. and indications that “kho=off cma=128M” may work with running 7.0.0.

I am still in the state of my original suggestion, stick with 6.17 kernel and avoid any updates for now.

It isn’t clear, no one is replying to NVIDIA staff to say “Yeah, it is working now” are they?

(I am 2 x GB10s with NCCL cable).

8

Probably the best solution for the moment is to revert the kernel as described in the link.

I is a of course a little bit cumbersome and requires some Linux knowledge.

9

Hey everyone,

I see nvidia is working on a fix.

I think I found the root cause and ultimately my ‘fix’ is the same as the sticky one in the "DGX Spark Update Advisory".

I was getting intermittent NCCL RDMA failures (around 75% failed) so perhaps on the heavier side of intermittent :p

As all 3 of my nodes were running at around 100GB allocation at the time I thought it was memory pressure, but even when shutting down the processes the tests were showing the same symptoms. I’m pretty sure they were fine before the kernel update.

/proc/meminfo was reportint a CMAtotal of 0 and CmaFree of between 3 and 9 GB depending on the node. Obviously Free should never report more than Total. It looks like “Kexec HandOver (KHO)” is enabled by default with the new kernel 7.0.0-1019. This releases pages back to the kernel as cmo page blocks. I dont think the cx7 nic can pin those pages so whenever NCCL tried to allocate there it got a no mem error.

I just added the below to grub and rebooted and now it works.

“GRUB_CMDLINE_LINUX=”$GRUB_CMDLINE_LINUX kho=off""

I see the sticky posts one looks like this:

“GRUB_CMDLINE_LINUX_DEFAULT=“console=tty0 console=ttyS0,921600 kho=off””

But only noticed this after my one. Either way it’s now working in every test.

Hope this helps.

10

Yeah, that link is my post, I wouldnt recommend with going ahead with any updates while you have that in there…

Maybe kho=off will work. Dunno - I am running GLM5 3 EXL3 latest and have to run with gmu 0.88 for 850k context on tp=2. Get the feeling going to kernel 7 is going to taper my context down for it to work.

For what? A minor 580 driver update…??

11

I hope there well be a notification somewhere when they have the next working update. So far I’m waiting.