Upgrading the GPU Driver from 580 to 590 on DGX Spark using CLI

The official update guide recommends using the DGX Dashboard for all system updates. This has never worked for me, maybe because we modified the Spark into a multi-user workstation. In any case, I needed a CLI approach to upgrade the driver. Information on how to do this is absent from the official documentation and standard NVIDIA driver-install procedures appear not appropriate for the Spark. The following solution cleanly upgrades all 580 driver packages to 590. The following is curated Claude output. I verified the upgrade worked as described below. (You can verify what apt will install and remove by adding --dry-run as an option.)


Documented CLI Method — and its limitation

The user guide’s documented manual CLI procedure is:

sudo apt update
sudo apt dist-upgrade
sudo fwupdmgr refresh
sudo fwupdmgr upgrade
sudo reboot

However, this does not upgrade the GPU driver. The dist-upgrade step will upgrade general system packages but will not transition from nvidia-driver-580-open to nvidia-driver-590-open.


Why standard NVIDIA driver upgrade methods don’t apply

NVIDIA’s driver installation guide and most forum advice assumes the driver comes from the NVIDIA CUDA repository. On the DGX Spark this is not the case.

The Spark uses a tightly coupled kernel+driver architecture where the GPU kernel modules are pre-built into Ubuntu’s linux-nvidia kernel flavour and delivered via noble-updates/restricted — not via DKMS or NVIDIA’s CUDA repo. In fact, the DGX factory configuration explicitly pins all driver packages from the CUDA repo to priority -1 (never install) via /etc/apt/preferences.d/cuda-compute-repo-lowpri*, precisely to prevent the CUDA repo driver from being installed.

This means:

  • sudo apt install nvidia-open pulls from the CUDA repo — wrong on DGX Spark
  • DKMS-based approaches don’t apply — kernel modules are pre-built

Correct CLI upgrade from 580 to 590
(Note: I have manually verified that this works as intended on an NVIDIA DGX Spark).

First run general system updates:

sudo apt full-upgrade
sudo reboot

Then upgrade the GPU driver:

sudo apt install \
  nvidia-driver-590-open \
  linux-modules-nvidia-590-open-nvidia-hwe-24.04 \
  nvidia-prime-
sudo apt autoremove
sudo reboot

The nvidia-prime- trailing minus excludes nvidia-prime, which is not required and not factory-installed either.

After reboot, verify with:

nvidia-smi  # should report 590.48.01

To revert to 580 if needed:
(Note: since the upgrade was successful for me, I have not manually verified that this fully reverts to the original state).

sudo apt install \
  nvidia-driver-580-open \
  linux-modules-nvidia-580-open-nvidia-hwe-24.04 \
  nvidia-prime-
sudo apt autoremove
sudo reboot

Hope this helps others needing to do a CLI driver update.

Just be aware that driver 590 and hwe kernels are not yet supported on the Spark

Hi @aniculescu ,
Thanks for your advice! I wonder, when (due date) can we or the DGX Spark will automatically be updated to the driver version >=595.45.04, because this driver is needed for CUDA version 13.2?

Many thanks!

Hi!

I made the mistake of updating the drivers to 590 and my spark is basically frozen from the moment I login. Is there any way of fixing this without a full system re-install?

Which package did you install? The meta-package nvidia-driver-590-open?

Then

apt purge nvidia-driver-590-open

apt install nvidia-driver-580-open

reboot

should do the job. Otherwise the name of the package you choose to install.

Try

dpkg -l|grep nvidia-driver

when in doubt.

EDIT: When you are unable to login via console, login via ssh.

Any showstoppers ?

I might be diving into upgrading for funsies, I just prepared a full backup so…why not?

I think CUDA13.2 works just fine with the 580 drivers. Not seen much about the 595 drivers outside of this thread, and I certainly haven’t seen any WOW posts about performance increases from people running 595 drivers. For that, I’m just going to stick with the stable branch for now :-)

Unfortunately, this upgrade process is now broken :-(

it appear it’s trying to installed some 610 packages.

this previously worked to install 595.

$ sudo apt install   linux-modules-nvidia-590-open-nvidia-hwe-24.04   nvidia-prime-
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package 'nvidia-prime' is not installed, so not removed
The following package was automatically installed and is no longer required:
  nvidia-firmware-580-580.159.03
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  libnvidia-cfg1 linux-modules-nvidia-595-open-6.17.0-1021-nvidia linux-modules-nvidia-595-open-nvidia-hwe-24.04 nvidia-firmware-595-595.71.05 nvidia-kernel-common-595 nvidia-xconfig
Recommended packages:
  libglx-nvidia0
The following packages will be REMOVED:
  libnvidia-compute-580 linux-modules-nvidia-580-open-6.17.0-1021-nvidia linux-modules-nvidia-580-open-nvidia-hwe-24.04 nvidia-kernel-common-580 nvidia-utils-580
The following NEW packages will be installed:
  libnvidia-cfg1 linux-modules-nvidia-590-open-nvidia-hwe-24.04 linux-modules-nvidia-595-open-6.17.0-1021-nvidia linux-modules-nvidia-595-open-nvidia-hwe-24.04 nvidia-firmware-595-595.71.05
  nvidia-kernel-common-595 nvidia-xconfig
0 upgraded, 7 newly installed, 5 to remove and 0 not upgraded.
Need to get 83.3 MB of archives.
After this operation, 233 MB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports noble-updates/multiverse arm64 nvidia-firmware-595-595.71.05 arm64 595.71.05-0ubuntu0.24.04.1 [74.9 MB]
Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa  libnvidia-cfg1 610.43.02-1ubuntu1 [137 kB]
Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa  nvidia-xconfig 610.43.02-1ubuntu1 [80.3 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports noble-updates/multiverse arm64 nvidia-kernel-common-595 arm64 595.71.05-0ubuntu0.24.04.1 [35.6 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports noble-updates/restricted arm64 linux-modules-nvidia-595-open-6.17.0-1021-nvidia arm64 6.17.0-1021.21 [8,035 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports noble-updates/restricted arm64 linux-modules-nvidia-595-open-nvidia-hwe-24.04 arm64 6.17.0-1021.21 [15.5 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports noble-updates/restricted arm64 linux-modules-nvidia-590-open-nvidia-hwe-24.04 arm64 6.17.0-1021.21 [5,306 B]
Fetched 83.3 MB in 1s (92.9 MB/s)
Selecting previously unselected package libnvidia-cfg1:arm64.
(Reading database ... 310366 files and directories currently installed.)
Preparing to unpack .../libnvidia-cfg1_610.43.02-1ubuntu1_arm64.deb ...
Unpacking libnvidia-cfg1:arm64 (610.43.02-1ubuntu1) ...
Preparing to unpack .../nvidia-xconfig_610.43.02-1ubuntu1_arm64.deb ...
Unpacking nvidia-xconfig (610.43.02-1ubuntu1) ...
dpkg: error processing archive /var/cache/apt/archives/nvidia-xconfig_610.43.02-1ubuntu1_arm64.deb (--unpack):
 trying to overwrite '/usr/bin/nvidia-xconfig', which is also in package nvidia-utils-580 580.159.03-0ubuntu0.24.04.1
Errors were encountered while processing:
 /var/cache/apt/archives/nvidia-xconfig_610.43.02-1ubuntu1_arm64.deb
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

I just upgraded one of my Gigabyte AI Atop Top. Nvidia is now 580.159.03 Cuda 13.0, the other one is 580.142 Cuda 13.0

Claimed 2GB back, now both 122G

the point here is to upgrade to 13.2 which apear to have significant performance boost on DGX spark.

Not saying I’m doing it right. but I’ve used apt-mark hold nvidia-firmware-595-595, and used the firmware picker in the UI to switch to 595-open… My Acer GB10’s are running, but I’m new enough to not know what errors to be looking for.

FWIW, I’m running nemotron-3-super-120b-a12b-nvfp4 with a local vllm-node container build to get 0.22.rc1 and the sparkrun project.

Hey Guys,
have not been here for a while.
Any news about the official support of 590 for the Spark?

oh have been away for to long I guess we are at CUDA 13.4 R615 now.

So whats the experience - what driver runs stable and fast?

can someone from the nvidia spark team clarify how we are supposed to maintain the host driver on dgx spark? many projects now use CUDA 13.3+ are now expecting driver 610+. yet no automatic driver updates officially till now. do we have an expected driver release cycle? or we are expected to upgrade on our own like on the pc? if so, what are the officaly instructions and what branch to use?

This follow-up describes how to upgrade both the kernel and driver (Ex.: kernel 6.17 → 7.0, driver 595 → 615), and how to avoid upgrade issues.

First, choose your usage

You should assess if you really want to do this because it is not supported as described in the comments. I see two choices.

  • (a) Supported: Use the Spark as intended, as an AI appliance, and stick to using the DGX Dashboard for updates.
  • (b) Unsupported: Use the Spark more like a workstation and/or experiment with the GB10 chip using more recent software stacks. This post describes how to perform a remote upgrade using the CLI to get the latest drivers.

Before choosing (b), read this thread and make sure you are comfortable doing what it describes. The steps below are designed to avoid that recovery, but I suggest only proceeding with (b) if you could do that recovery if you had to.

To answer some questions, I have one Spark following (a), and another Spark following (b). They both seem to work fine and I have not seen any showstoppers either way. I personally would not recommend (b) unless there was a good reason.


Prerequisite: no DKMS

On the Spark, the GPU modules come prebuilt from Ubuntu as linux-modules-nvidia-<branch>-open-<kernel>. The panic in thread 368939 came from DKMS: its kernel hook fails on arm64, so the initramfs hook never runs, the new kernel gets no initrd, and the boot fails with VFS: Unable to mount root fs.

  1. Block DKMS. Create /etc/apt/preferences.d/nvidia-dkms:

    Package: nvidia-dkms*
    Pin: release *
    Pin-Priority: -1
    

    If apt can’t find a prebuilt module, it now refuses the install instead of building one.

  2. Check that nothing DKMS-related is installed (both should print nothing):

    dpkg -l | grep -i dkms
    ls /etc/kernel/postinst.d/ | grep -i dkms
    

    If DKMS packages are present, first switch to the prebuilt modules for your current kernel (post #16 in 368939).

  3. Optional: keep nvidia-prime out. The driver recommends it, but it’s for hybrid-GPU laptops, so it’s just clutter on a Spark and not a cause of failure. Also, the supported upgrade path, using the Dashboard, would not install this. Create /etc/apt/preferences.d/local-exclude:

    Package: nvidia-prime
    Pin: release *
    Pin-Priority: -1
    

Plan: where you are, where you can go

Rule: a driver branch only works on a kernel that has a linux-modules-nvidia-<branch>-open-<kernel> package.
Strategy: kernel first (Part B), then driver (Part C), one big change per reboot. New branches usually need the newest kernel, while your current branch is usually still built for it. Values marked Ex. are from my upgrade.

  1. Where you are:

    uname -r                                               # Ex.: 6.17.0-1018-nvidia
    nvidia-smi --query-gpu=driver_version --format=csv,noheader   # Ex.: 595.71.05
    apt-cache search --names-only "^linux-modules-nvidia-[0-9]+-open-$(uname -r)$"
    

    The search lists the driver branches built for your kernel. Ex.: 535, 580, 595. The highest is the newest branch you can run without a new kernel.

  2. Which kernel apt offers:

    apt-cache policy linux-image-nvidia-hwe-24.04
    

    Ex.: Candidate: 7.0.0-1019.19~24.04.2, which means kernel 7.0.0-1019-nvidia. If Candidate = Installed, there’s no newer kernel.

  3. Which driver branches the new kernel supports:

    apt-cache search --names-only "^linux-modules-nvidia-[0-9]+-open-<kernel from step 2>$"
    

    Ex.: kernel 7.0.0-1019-nvidia gives 580, 595, 610, 615. Your target branch must be listed. Your current branch should be listed too, so the GPU keeps working after Part B.

  4. Optional: which kernels a branch supports, from the other direction:

    apt-cache search --names-only "^linux-modules-nvidia-615-open-[0-9]"
    apt-cache depends linux-modules-nvidia-615-open-nvidia-hwe-24.04
    

    Replace 615 with your target branch. This is how I found 615 existed only for 7.0.0-1019 (ignore the -64k, azure and old 6.8 results). If a metapackage depends on another branch’s metapackage, it’s an empty transitional package: 590 now just pulls in 595.

  • If the target branch is already built for your current kernel, skip Part B.
  • If your current branch isn’t built for the new kernel, expect no GPU on it until Part C.

Substitute your own values below:

  • OLD = current branch (Ex. 595)
  • NEW = target branch (Ex. 615)
  • OLDK = current kernel (Ex. 6.17.0-1018-nvidia)
  • NEWK = target kernel (Ex. 7.0.0-1019-nvidia)

A. Remote-safe boot setup

A stock Spark hides the GRUB menu, always boots the newest kernel (GRUB_DEFAULT=0), and doesn’t reboot after a panic (kernel.panic=0). A bad kernel leaves the machine stuck, and someone has to go to fix it. Instead, modify grub with a fallback kernel:

  • GRUB_DEFAULT=saved: boot the kernel you chose (saved_entry).
  • grub-reboot: try another kernel once (next_entry, cleared as soon as it’s used).
  • panic=30: a panic reboots after 30 seconds, back to saved_entry, and you can SSH in again.
  1. List the exact menu titles and note them:

    sudo grep -E "menuentry |submenu " /boot/grub/grub.cfg | cut -d"'" -f2
    

    Copy titles exactly into the commands below. If a title doesn’t match, GRUB silently boots entry 0 (the newest kernel), which defeats the fallback.

  2. Create the override (zz- sorts last, so it wins):

    printf '%s\n' 'GRUB_DEFAULT=saved' 'GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT panic=30"' | sudo tee /etc/default/grub.d/zz-remote-safe.cfg
    
  3. Regenerate GRUB and pin the current kernel:

    sudo update-grub
    sudo grub-set-default "Advanced options for Ubuntu>Ubuntu, with Linux OLDK"
    
  4. Protect the fallback kernel from autoremove:

    sudo apt-mark manual linux-image-OLDK linux-modules-OLDK
    
  5. Check, then do a test reboot on the same kernel:

    sudo grep -c 'panic=30' /boot/grub/grub.cfg   # > 0
    sudo grub-editenv list                        # saved_entry=...OLDK
    sudo reboot
    
  6. Afterwards:

    cat /proc/sys/kernel/panic    # 30
    sudo grub-editenv list        # saved_entry=...OLDK
    

B. New kernel (keep the current driver branch)

  1. Preview:

    sudo apt update
    sudo apt full-upgrade --dry-run
    

    Check that:

    • The new kernel comes with linux-modules-nvidia-OLD-open-NEWK. If not, stop.
    • Nothing mentions dkms.
    • No packages are held back (“kept back”). None are expected; if any are, find out why before continuing.
    • The module for OLDK being removed is expected. The fallback will boot with SSH but no GPU.
  2. Upgrade:

    sudo apt full-upgrade
    
  3. Before rebooting, check the initrd (the 368939 failure). If any check fails, don’t reboot:

    ls -l /boot/initrd.img-NEWK                        # exists
    sudo grep -c 'initrd.*NEWK' /boot/grub/grub.cfg    # > 0
    sudo dpkg --audit                                  # prints nothing
    sudo grub-editenv list                             # saved_entry still OLDK
    
  4. Boot the new kernel once:

    sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux NEWK"
    sudo reboot
    
  5. Verify:

    uname -r       # NEWK
    nvidia-smi     # still the OLD branch
    

    If uname -r shows OLDK, the new kernel failed and the fallback worked. Check journalctl -b -1. An empty previous boot points to a missing initrd.


C. New driver branch (on the proven kernel)

  1. Install the new driver. Apt won’t swap branches by itself: the OLD metapackages conflict with NEW, so name them for removal with a trailing -. List the installed ones:

    dpkg -l | grep -E '^ii +(nvidia-driver-[0-9]+-open|linux-modules-nvidia-[0-9]+-open-nvidia-hwe)' | awk '{print $2}'
    

    Install nvidia-driver-NEW-open and linux-modules-nvidia-NEW-open-nvidia-hwe-24.04, and remove each listed package. Ex. command, one line (590 is the transitional package left over from the earlier upgrade):

    sudo apt install nvidia-driver-615-open linux-modules-nvidia-615-open-nvidia-hwe-24.04 nvidia-driver-595-open- linux-modules-nvidia-595-open-nvidia-hwe-24.04- linux-modules-nvidia-590-open-nvidia-hwe-24.04-
    

    Naming the linux-modules-…-hwe-24.04 package satisfies the driver’s nvidia-dkms-NEW-open dependency with the prebuilt module. In the prompt, check that:

    • the only removals are OLD (or older) NVIDIA packages
    • no new kernel is installed
    • nothing mentions dkms
  2. Before rebooting, run the checks from B.3, plus:

    modinfo -k NEWK nvidia | grep '^version'    # NEW version
    
  3. Boot once:

    sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux NEWK"
    sudo reboot
    
  4. Verify:

    nvidia-smi    # NEW driver (Ex.: 615.71.09)
    

D. Finish

  1. Make the new kernel the default:

    sudo grub-set-default "Advanced options for Ubuntu>Ubuntu, with Linux NEWK"
    

    I kept zz-remote-safe.cfg permanently. A future kernel won’t boot until I choose it with the same grub-reboot pattern. After every future kernel update, repeat B.3–B.5 and grub-set-default to the new kernel. Otherwise the saved kernel keeps booting, and once a driver update removes its NVIDIA module, it comes up with no GPU.

  2. Clean up:

    sudo apt autoremove --dry-run
    sudo apt autoremove
    

Note that I didn’t have the pleasure of testing the fallback kernel in grub since everything worked. On a previous upgrade, I did have the pleasure of a kernel panic (the dkms pinning described here should avoid it), and can verify that documentation for fixing that works well. Consider using a coding LLM to help with the upgrade, especially for checking the --dry-run output.

Is there any actual performance boost to be seen with LLM recipes upgrading the GPU drivers and CUDA? Or are people doing this just for fun?

I updated drivers from 580 to 615 because of this issue, working fine for me rn

https://github.com/NVIDIA/spark-local-inference/issues/137

You dont need to, this is not windows where updating your drivers like its a video game becomes OCD… Any LLM engine will use the drivers for that kit and it expects your drivers to be the ones supported for the sparks. So if you update to unsupported drivers thinking it’s like video games cards and it will be “better”, you’re asking for trouble…