Request: DGX Spark OS Recovery Without Wiping User Data

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Today, I encountered a critical issue during a software update on my DGX Spark that resulted in a complete system failure. The only recovery option available was a full restore, which wiped all user data, even though I had good backups and a well-managed environment in place. I have now spent over 14 hours just getting back to a working baseline after this failure. For a modern AI appliance, that kind of recovery story is extremely painful and, in many cases, avoidable.

Given that this is 2026 and NVIDIA is a leader in AI infrastructure, it would be extremely valuable to move beyond recovery mechanisms that resemble legacy BIOS-era approaches. Modern systems—across both Linux distributions and platforms like macOS—have long supported partitioning strategies or recovery environments that allow the OS to be restored without impacting user data.

For DGX Spark, a more robust baseline could include:

  • Separate system and user data partitions by default
  • A reliable OS recovery mechanism that preserves user workloads and data
  • Snapshot or rollback capabilities for safe updates

These features are not just “nice to have” quality-of-life improvements; they are essential for maintaining developer productivity and protecting critical work, especially in environments where DGX Spark is used as a primary development and inference platform.

I strongly encourage NVIDIA to prioritize improvements in this area. Many developers depend on DGX systems for serious workloads, and the current recovery model introduces unnecessary risk during routine updates. A first-class, data-preserving recovery path on DGX Spark would save a tremendous amount of time and frustration for users who are heavily invested in this platform.

The more I build on my Spark the more important this becomes …

Please Please devote some engineering resources to this. It is so traumatic to go through this.

The solution is simple if you control the OS and restore. Please address this as a priority.
I encourage everyone with a spark to post a reply like a petition that hopefully get NVIDIA to address.

Mark

2

I know it is a bit of a wild idea but if you use something for production it is probably a good practice to make regular backups.

3

Ummm.. how you work is up to you, if you are keeping user data on your system drive AND expecting resilience where it doesnt exist then that is a problem for you to solve, not NVIDIAs. The DGX GB10 is not on its own a resilient redundant system, that comes down to infrastructure design which the end user is responsible for.

4

I think some of the thinking around this topic is about 20 years out of date.
We already have exactly this kind of functionality on macOS and have for a very long time: a separate recovery partition, the ability to reinstall or repair the OS without touching user data, and simple, robust paths to recover from a bad update. This is not a novel idea and it is not technically hard in 2026.
I’m not asking NVIDIA to replace good backup hygiene. I already take this seriously: I back up via git, iCloud, Borg to local HDD, to my Mac, and to Wasabi, precisely because I know infrastructure can fail. I accept that part as my responsibility.
What I’m asking NVIDIA to do is to meet us halfway: ship DGX Spark with a proper recovery partition and tooling so we can restore or reimage the OS without wiping user data by default, similar to how macOS handles OS reinstalls. This is a very basic expectation for a premium, “appliance-like” AI workstation in 2026.
A data-preserving OS restore path should be the baseline, not an optional extra or something users have to engineer around themselves.

5

It is all possible, you can shrink your main partition, dd the reocvery ISO to the free space in a new partition, add a grub option to boot from the ISO partition. Pretty much done. What you arr saying is you didnt do that and expect NVIDIA to have done it for you I guess.

If your data was important then you also could have transplanted the SSD to another system, mounted it read only, backed up your data and then moved it back to the DGX to re-image.

These are all very standard Linux Sysadmin tasks. Your expectations are going to be different as you come from MacOS world, unfortunetely for you Linux is different.

6

Probably you can get some useful information from this topic.

As you can see some standard Ubuntu arm64 live iso + hands can make you happy without wiping.

7

That would be nice for many users, so there is nothing wrong with asking for it. And it would be nice if NVIDIA made that an option for that user segment; however, I do need to point out that I think the premise is wrong…

Yeah… that’s not what the DGX Spark is… I mean I own 5x and I like NVIDIA & CUDA, but let’s call it what it is… an experimental dev kit. It’s not even a polished dev kit.

8

Also, unfortunately for many of us, we have a couple of terabytes hanging around in $HOME/.cache/huggingface. Theres a bit of a problem finding a place yo put that when you do a full re-image 😂😂

9

Strawman proposal (AI‑assisted, needs expert review): Modern DGX Spark recovery with data‑preserving options

First, thank you to everyone who has shared detailed recovery steps, especially the kernel‑panic writeup with the missing initramfs and GRUB timeout issues. That kind of deep, hands‑on documentation is incredibly valuable, and it clearly comes from people with serious Linux and DGX experience.

At the same time, I think these threads also highlight a systemic problem with the DGX Spark recovery story as it exists today.

Right now, the practical options look like this:

  • The official USB recovery image that boots through a very BIOS‑era flow and then completely wipes and reimages the internal SSD, destroying all user data by design.
  • Very low‑level manual procedures that involve building an ARM64 live USB, going through UEFI, chrooting into the internal NVMe, running update-initramfs, forcing DKMS, dpkg --configure -a, and editing /etc/default/grub plus update-grub to recover from a kernel panic or bad update.

When you contact NVIDIA support, the only officially offered path for an unbootable Spark is effectively: “download the recovery image and wipe the system.” There is no officially documented or supported intermediate path that even tries to repair the OS or bootloader without erasing the entire device. For many of us with multi‑TB datasets, models, and carefully built environments, that “nuke from orbit” option is extremely expensive.

The community writeups show that in many cases the failure is clearly at the kernel/initramfs/bootloader layer and can be repaired without touching user data. But the gap between what’s technically possible and what’s officially supported is huge. And for newer users, especially people coming from macOS who bought a DGX Spark primarily for local AI compute, the current recovery UX is frankly overwhelming.

By contrast, macOS and several Linux distros (e.g., Pop!_OS) have, for years:

  • A dedicated recovery partition on the internal disk
  • The ability to boot into that recovery environment without juggling USB sticks every time something goes wrong
  • A guided UI that can reinstall or repair the OS while preserving user data by default

That’s the baseline I think DGX Spark should be aiming for.

What I’m asking for from NVIDIA

Concretely, I’m asking NVIDIA to move toward a modern, appliance‑grade recovery baseline for DGX Spark:

  • A documented, supported partition scheme that cleanly separates:

    • EFI system partition
    • A dedicated recovery partition
    • The main OS/root partition
    • One or more user/data partitions

    Recovery operations should target only the OS/root partition by default, leaving user/data partitions untouched unless the user explicitly chooses a full wipe.

  • An on‑device recovery environment on that recovery partition, bootable via a simple menu entry (“DGX Spark Recovery Environment”), so we are not forced to rely on 30–40‑year‑old BIOS/USB patterns as the primary recovery story in 2026.

  • A data‑preserving repair/reinstall path that encapsulates what the community is already doing by hand:

    • Detect and rebuild missing initramfs for selected kernels
    • Fix DKMS/dpkg “installed but failed to configure” states
    • Regenerate GRUB configs and set sane, non‑zero GRUB timeouts so older kernels remain reachable when new ones break
    • Allow reinstall of the OS/root partition from a known‑good image while preserving user/data partitions

  • Safer bootloader defaults out of the box (non‑zero GRUB timeout, accessible menu, reasonable failure handling) so that falling back to a working kernel does not require physical access and manual grub surgery every time an update goes sideways.

This would let support offer a “repair / reinstall OS, preserve data” option as a first‑line response in many cases, reserving “wipe and reimage” for genuinely unrecoverable failures.

AI‑assisted strawman: Community “Spark Recovery” project

To get the conversation started, I’ve used AI tools to draft a strawman design for a community‑driven “Spark Recovery” project. This is explicitly AI‑assisted and should be treated as a starting point for review, not a final design.

The idea is similar in spirit to the community Docker tooling around DGX/Spark (sparkearna, sharkrun, etc.), but focused on safe OS recovery with preserved user data.

High‑level goals

  • Provide a data‑preserving recovery path that doesn’t default to wiping the SSD.
  • Make recovery accessible to users who aren’t long‑time Linux admins.
  • Keep everything scriptable and auditable so power users can see and control exactly what happens.

Draft architecture (for discussion only)

  1. Standard partition scheme

    On the primary NVMe:

    • ESP – EFI system partition (existing)
    • RECOVERY – dedicated recovery partition (e.g., 8–16 GiB)
    • OS_ROOT – main DGX OS / root filesystem
    • USER_DATA – user projects, models, datasets, home dirs

    The key invariant: recovery flows operate on OS_ROOT by default and treat USER_DATA as read‑only unless a full wipe is explicitly requested.

  2. On‑device recovery environment

    • A minimal, self‑contained Linux environment installed on RECOVERY.
    • Exposed via a bootloader entry like “DGX Spark Recovery Environment”.
    • Ships with tools and scripts to:
      • Inspect kernels and initramfs
      • Rebuild missing initramfs for selected kernels
      • Resolve DKMS and dpkg --configure -a failures
      • Regenerate GRUB configs and set non‑zero timeouts

  3. Guarded workflows

    A small number of clearly labeled actions in the recovery environment, for example:

    • “Repair current OS (preserve user data)”

      • Rebuilds initramfs for the chosen kernel
      • Fixes DKMS/dpkg state
      • Regenerates GRUB and enables a menu with timeout

    • “Reinstall OS (preserve user data partition)”

      • Formats and reinstalls OS_ROOT from a known‑good image
      • Leaves USER_DATA intact (mounted read‑only during the operation)

    • “Full factory reset (wipe everything)”

      • Clearly marked as destructive and used only as a last resort

    Each workflow would show exactly which partitions will be modified and which will not.

  4. Modern UI on top of scripts

    • Underneath: shell scripts / automation that advanced users can inspect and run manually.
    • On top: a simple text‑based or web‑style UI to guide less experienced users through the options and confirm destructive actions.

Call for help from the Linux / DGX community

This strawman is not meant to be “the answer”; it’s an AI‑assisted sketch that almost certainly has gaps and rough edges. What I’m hoping for is:

  • Feedback from people who have strong opinions about safe partitioning schemes for appliance‑like systems.
  • Input from folks who know GRUB/systemd‑boot on aarch64 inside out.
  • Suggestions from anyone who has designed or operated safe “repair vs reinstall vs wipe” flows on Linux for production systems.

If we can converge on a community‑vetted design for:

  • A recommended partition layout
  • A standard recovery environment layout and boot entry
  • A guarded, data‑preserving “repair / reinstall OS” workflow

…then we have something concrete to:

  • Share among current users as a safer pattern than “USB and wipe,” and
  • Present to NVIDIA as a serious, well‑thought‑out proposal for future DGX OS / DGX Spark releases.

For many of us who came from macOS and bought a Spark because we needed more local AI power, having a modern, partitioned recovery path with a simple UI would save enormous amounts of time and frustration. I’d love to collaborate with anyone willing to help turn this AI‑assisted strawman into a practical, community‑maintained “Spark Recovery” project.

10

@griffith.mark it might surprise you, but when issues arrive the Spark can be booted with a live USB distro, Ubuntu preferably, mount the NVMe and recover your data before wiping it with FastOS (the recovery method supplied by NVIDIA)

11

These are certainly constructive points where improvements could be made. Im not however sure how much of the community would be interested, certainly the quadrant which you come from (MacOS) might be, but the Linux side will already have been dealing with these kind of issues on other systems and already have the processes they need to execute in mind and ready.

There is a real constraint here on how NVIDIA would make decisions e.g. how much space to designate to user and how much to designate to system, different users different requirements. No one size fits all here

12

Thank you for mentioning the live USB approach — it’s definitely useful for anyone who doesn’t have backups yet, and I’m glad you highlighted it.

In my case though, I already had solid backups in place: local HDD, my Mac, Wasabi, and iCloud. The problem here isn’t “no backups”; it’s what happens even when you do have 100% perfect backups.

Restoring terabytes of user data, images, and artifacts is not trivial:

  • I have around 750 GB of HF cache alone, which takes a long time to rebuild.
  • RSYNC’ing TB of small files can easily take 6+ hours end‑to‑end.

So the pain point isn’t just data loss — it’s the sheer amount of time and friction involved in getting back to a usable state after a forced full wipe. That’s why I’m pushing for a proper recovery partition and data‑preserving OS repair/reinstall flow: even with perfect backups, “wipe and start over” is extremely costly on a DGX‑class box.

13

@giles8

Thank you for your comments. Could you elaborate on your comment

“but the Linux side will already have been dealing with these kind of issues on other systems and already have the processes they need to execute in mind and ready.”

It would be really useful to work out what we could recommend as a Spark Recovery Solution that in proven track for Linux and make it user and newbie friendy.

14

I spent the good part of yesterday and today watching our Dev Ops team recover a hypervisor with a dozen failed guests. They knew what to do as a team, boot into single user mode, change boot options, handle remote access via iLO. Recovery completed. Skills I dont actually understand myself in detail. I just know this is normal in Linux world.

Their alternative option was to deploy a new hypervisor and rebuild all the guests from puppet, then resync the DBs.

Point is they had the skills to recover. NVIDIA could do that for you, but their professional services invoice would be unacceptable to you in terms of cost.

GB10 is a niche developer platform, I dont put anything on there that I cannot gleefully throw away. Agentic code produced goes to a private forgejo server. Support systems are by and large seperate.

DGX OS is also based on Ubuntu, many changes may be needed to be addressed upstream at Ubuntu, not by DGX. The further NVIDIA moves from Ubuntu, the further we move from being in-step, and the less likely that we will be able to support DGX once 24.04 goes out of service. So careful what you wish for there.. Some are already clamouring that we get DGX aligned to Ubuntu 26.04…

15

Suggestion from the Dev Ops team:

Now you have it rebuilt, you might want to have /home mounted as an NFS export on your NAS rather than local to the spark.

Then you only have to remount it next time you do a restore?

And you know your data is resiliently storred with RAID1 / 5 / 6 / 10 whatever, rather than rely on the single SSD that could fail at any time.

16

Thanks for the thoughtful reply and for relaying your DevOps team’s perspective — I do hear you on the NAS/NFS angle and the “treat it as disposable” mindset.

On the storage side: I’m not against NAS or NFS at all, but for my specific workloads (LLM inference, HF cache, lots of tiny files and random IO), I don’t think I can get the same I/O characteristics off-box as I can from the internal NVMe.

  • Internal NVMe on something like the Spark is PCIe-attached and easily in the multi‑GB/s range with very low latency and high IOPS.[1][2]
  • Even very fast external SSDs over USB 3.2 / USB 4 / Thunderbolt tend to top out around 1–3.5 GB/s in ideal cases, and they still carry higher latency than direct NVMe.[3][4][5]
  • Typical NFS NAS setups (especially over 1–2.5 GbE) can bottleneck badly under random I/O or lots of small files; many tests show tens of MB/s and much higher latency compared to local NVMe unless you invest in serious 10GbE+ and a very capable NAS.[6][7][8]

So for some users, mounting /home over NFS to a RAIDed NAS is a great answer. For heavy on-box AI workloads with TBs of models and HF cache, that trade-off in latency and throughput is non-trivial. I’m deliberately keeping a lot of that hot data local because I need the internal NVMe’s performance envelope.

On your broader point about DevOps skills: I actually agree that what your team did (single-user mode, iLO, boot options, etc.) is “normal Linux world.” And I agree that NVIDIA sending in a pro services team every time would be wildly uneconomic.

But for me, that reality reinforces the core argument:

  • The fact that we need specialist DevOps skills to recover from a routine “click Update” failure on a customer-facing appliance like Spark is exactly why I’m asking for a better, customer-focused recovery story.
  • It’s not that those skills shouldn’t exist — they absolutely should — but that the default out-of-the-box behavior for a product positioned as an “AI box for developers” shouldn’t require them every time an update misfires.

I’m not asking NVIDIA to diverge wildly from Ubuntu or to become my personal DevOps team. What I’m asking for is:

  • A recovery partition and on-device environment that can repair or reinstall the OS while preserving user data,
  • A simple, modern UI on top of that,
  • And defaults that make it harder to strand users in a “kernel panic, no GRUB fallback, wipe everything” corner.

Your strategic view on keeping DGX close to Ubuntu is interesting and I agree there’s a balance to strike. My hope is that we can get a more user-friendly recovery experience within that ecosystem, rather than accepting that the only safe path is “treat everything on Spark as disposable and rely on NAS + DevOps heroics” forever.

17

I am sure that there are people from NVIDIA who have seen this thread and thought “sure, that would be nice”. However, I think a lot of them will think, if Ubuntu developed that feature the we would get it too.

And the truth is Ubuntu doesnt have this feature, while it permits users to go from 24.04 to 26.04 or from 22.04 to 24.04 then to 26.04 the upgrade tool caveats the risk to the user explicitly.

I know you want a really well polished dev environment that would have got you out of the fuzz you got into. But I dont think that GB10 DGX platform is about that. This is an expermental platform, it is going to have rough edges.

You spoke about NFS over a non performant network (<10Gbps). Unfortunately I come from a time where 10Mbps at home, and 56kbps to the Internet was more the norm.

Consumer internet ratios have changed with Internet access now equalling or exceeding the local access speeds that individual systems have. I cant do anything about that, you have a system with 10Gbps network, 200Gbps RDMA fabric, if your NAS is slow then that needs to be looked at.

18

What Ubuntu has is a better installer.Subiquity for the server and Ubuntu Desktop Bootstrap with Subiquity as the backend for the desktop.

On DGX Spark the recover image just creates a disk-wide partition and dumps the OS. Wipe everything and reboot!

19

Im yet to work in a Linux environment that runs Ubuntu as a production OS to be fair, sounds like you know mire than me 😄