Community “Spark Recovery” Project (Data-Preserving OS Recovery for DGX Spark)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

@eugr @raphael.amorim @DBSIDBSI

Others that I am not able address directly.

image
image1812×310 48.6 KB

Related thread: Request: DGX Spark OS Recovery Without Wiping User Data

You’re some of the biggest contributors and builders around DGX Spark — tools, recovery writeups, deep Grace/SM121 knowledge, escalation paths into NVIDIA, and all the low‑level debugging that the rest of us rely on.

This isn’t the most glamorous problem, but I think we really need to tackle it together: we need a community‑designed “Spark Recovery” path that’s better than the current “download FastOS and wipe the entire SSD” story.

Why this matters

Right now, even with good backups (local HDDs, NAS, cloud, etc.), the official answer to an unbootable Spark is effectively “nuke it and reimage.” For anyone running multi‑TB workloads, HF cache, models, and a ton of small files, that means:

  • Hours of rsync and cache rebuilds just to get back to a usable baseline.
  • Manual live‑USB rescues and chroot sessions for those who do know Linux internals.
  • A very steep cliff for newer Spark owners coming from macOS/Windows who bought the box for local AI compute, not to become GRUB/initramfs surgeons.

The current recovery image flow (download the dgx-spark-recovery-image-*.tar.gz, turn it into a FastOS USB, boot through UEFI, then START RECOVERY) works — but it completely erases the internal SSD by design. That’s the opposite of “appliance‑grade” if you’re actually running real workloads on the device.

I’m not saying NVIDIA shouldn’t stay close to Ubuntu, or that we don’t need backups or DevOps skills. I am saying that for a customer‑facing appliance‑class product like Spark, the default recovery experience shouldn’t be:

  • “Hope you know how to boot a live USB and hand‑fix initramfs/GRUB,” or
  • “Click this and we’ll erase everything on the internal SSD.”

AI‑assisted strawman: “Spark Recovery” design

I’ve used AI tools to help sketch a strawman design for a better recovery baseline. This is not authoritative; it’s explicitly AI‑assisted and needs real‑world correction. But it gives us something concrete to discuss.

1. Standard partition scheme

On the internal NVMe, a recommended layout like:

  • ESP – EFI system partition (as today).
  • RECOVERY – dedicated recovery partition (e.g., 16–32 GiB).
  • OS_ROOT – main DGX OS / root filesystem.
  • USER_DATA – user projects, models, HF cache, home directories.

Key invariants:

  • Recovery operations target OS_ROOT by default.
  • USER_DATA is preserved unless the user explicitly chooses a full wipe.
  • Hot data stays on local NVMe for performance; this isn’t “move everything to NAS.”

2. On‑device recovery environment

A minimal, self-contained Linux environment installed on RECOVERY, with:

  • A bootloader entry like “DGX Spark Recovery Environment”.
  • Tools and scripts to:
    • Inspect kernels and initramfs.
    • Rebuild missing/broken initramfs for selected kernels.
    • Resolve DKMS and dpkg --configure -a issues.
    • Regenerate GRUB configs and set non‑zero timeouts so you can actually fall back to a known‑good kernel.

3. Guarded workflows

Inside the recovery environment, a small set of clearly labeled flows, for example:

  • “Repair current OS (preserve everything)”

    • Rebuilds initramfs for the selected kernel.
    • Fixes DKMS/dpkg state.
    • Regenerates GRUB and enables a timeout/menu.

  • “Reinstall OS (preserve USER_DATA partition)”

    • Formats and reinstalls OS_ROOT from a known‑good image.
    • Leaves USER_DATA mounted read‑only and untouched by default.

  • “Full factory reset (wipe everything)”

    • Explicitly destructive, documented as last resort only.

Each flow shows exactly which partitions will be touched and which will not.

4. Modern UI on top of scripts

Under the hood:

  • Shell scripts / automation that advanced users can inspect, run, and modify.

On top:

  • A simple text UI or web UI that guides less‑experienced users through the options and confirms destructive actions. Think: “appliance‑grade” UX over a Linux‑grade toolbox.

Quick poke at the strawman (open challenges)

My own first pass at “poking holes” in this, all meant as constructive constraints:

  1. Existing installs
    Most current DGX Spark owners have the factory layout (EFI + one giant root partition). Shrinking that root and creating RECOVERY + USER_DATA partitions is doable but scary without a bulletproof migration script. We’d need a safe, idempotent post‑install script (run from a live USB or from a running system) that does the resize + rsync + fstab update. If it fails gracefully, people can still fall back to backup + full wipe.

  2. aarch64 + NVIDIA stack
    Everything in RECOVERY has to be Arm64 native. FastOS is already a minimal aarch64 Linux; the recovery environment should reuse as much of that as possible (or a minimal Ubuntu Server aarch64 + DGX repos). Chroot‑based repair flows need kernel headers, DKMS, nvidia‑driver packages, etc., pre‑staged or fetchable.

  3. Reinstall OS_ROOT flow
    Today, the official recovery scripts assume they own the whole drive and reflash the entire SSD. We’d need to extract/parameterize that logic so it only formats/mounts OS_ROOT while leaving USER_DATA untouched (and mounting it read‑only during the process).

  4. Secure Boot / UEFI / self‑encrypting NVMe
    The recovery env needs to play nicely with existing EFI variables and any SED management. GRUB in the ESP must gain a “DGX Spark Recovery Environment” entry safely.

  5. Btrfs/LVM alternative
    Not a requirement, but btrfs subvolumes + snapshots on OS_ROOT (and optionally USER_DATA) could enable non‑destructive rollbacks without repartitioning. Many people already experiment with this; it could be an optional “advanced mode” alongside the simpler partition scheme.

None of these feel like show‑stoppers — they’re just the details that the real experts here can help get right.

Concrete partition layout (ready for a README)

For a 4 TB NVMe, GPT layout via sgdisk, keeping the existing ESP:

# Size Type FS Label Mountpoint Purpose
1 512 MiB EF00 FAT32 ESP /boot/efi EFI System Partition (existing)
2 16–32 GiB 8300 ext4 RECOVERY /recovery On‑device recovery environment
3 128–256 GiB 8300 ext4/btrfs OS_ROOT / DGX OS root (snapshots optional if btrfs)
4 Remainder (~3.7+ TB) 8300 ext4/btrfs USER_DATA /data User projects, models, HF cache, /home, etc.

Notes:

  • Use labels or UUIDs in fstab (never raw /dev/nvme0n1pX names).
  • Mount USER_DATA at /data (and symlink /home → /data/home) so it’s obvious and easy to preserve.
  • New systems: ship a custom sgdisk.txt with the recovery media that creates this layout out of the box.
  • Existing systems: provide a community migration script (resize + migrate). Always back up first.

Recovery environment details (sketch)

  • Minimal aarch64 Linux (reuse FastOS components if possible).
  • GRUB menu entry: “DGX Spark Recovery Environment”.
  • Pre‑installed tools: update-initramfs, dkms, apt/debootstrap, grub-install, chroot helpers, dialog/whiptail for TUI.
  • Optional: tiny local web UI (Python/Flask or static page) for friendlier UX in the future.

Guarded flows:

  • Repair current OS (preserve everything)
    Chroot into OS_ROOT → fix initramfs/DKMS/dpkg → regenerate GRUB with a visible timeout/menu.

  • Reinstall OS (preserve USER_DATA)
    Format OS_ROOT → install base DGX OS + NVIDIA stack → ensure fstab mounts USER_DATA correctly.

  • Full factory reset
    Only after explicit double‑confirm, and clearly communicated as “this is equivalent to today’s FastOS full wipe”.

Start with shell + whiptail for transparency and auditability. If this matures, someone can build a small web dashboard on top.

Next step if there’s interest

If there’s interest from this group, I’ll:

  • Create a GitHub repo (spark-recovery) with this strawman as a README.
  • Add a sample sgdisk.txt, a migration script stub, and skeleton scripts for the three main workflows.
  • Iterate in public (forum + GitHub) so others can PR real improvements.

If you’re interested, please reply with a simple “+1” or “interested”, or drop specific feedback. Even a short “this is feasible / this is insane because X” would be really helpful.

We’re all going to get bitten by bad updates or kernel transitions sooner or later. I’d rather we collaborate now on a community‑approved recovery route than keep watching people rediscover the same pain one by one.

2

I’m not against this in principle but there are valid reasons it does not come from the factory set up like suggested. While there are other factors, the crux is:

  • Nvidia has no idea how many devices you may want to cluster
  • Running ~1T class models across 4-8 nodes requires the model to be downloaded in full on all of them
  • 1TB units are being sold where the OS plus a minimal container and a single above model basically exhaust the formatted space
  • CUDA/cuDNN and required tooling in the core OS have a variable footprint, which could make a hard partition problematic in future

The above calculations are off. For a 4TB device, you start with 3.7T after formatting and boot. For those of us with 4TB SSDs, it feels reasonable to dedicate space as suggested. However, a 1TB base device deployed in a cluster of 4 probably can’t hold more than a single model - before such a framework. If you started with this framework, once you experiment with a few 20GB vLLM containers you couldn’t even make full use of a cluster of 4. The model would not fit.

Personally, I usually set up my devices with a strict OS and user partition then map in outside storage. No recovery, actual key code is already safe.

All that said, I agree it would be nice to have the option to partition off at least the OS from /home in setup.

3

Specificaly:

You also have the option to install the NVIDIA DGX Software Stack on top of a vanilla Ubuntu distribution while still benefiting from the advanced DGX features. This installation method supports more flexibility, such as custom partition schemes.

I don’t think we will see too much in the way of changes to factory defaults. Although the auto-recovery Grub option would probably be pretty trivial to implement and would only use 9GB of partition (less if you make it a network install - i.e. minimal boot, download image from Internet).