Can DGX Spark work with non-DGX OS?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Anyone has similar request?

Can NVIDIA officially confirm whether DGX Spark supports customer-installed operating systems (e.g., Red Hat Enterprise Linux/OpenShift) with only the NVIDIA GPU driver installed instead of DGX OS? If so, what is the supported installation method, and would this configuration have any impact on the hardware warranty or NVIDIA software support entitlement?

2

Spark DGX OS is the officially supported Operating System, as it’s the environment and stack that we believe is optimzied and validated. User may install any other OS, but understand Support will be very limited. If there’s a need to triage hardware/warranty issues, we may ask you to reimage it back to factory image, as the diagnostics suites were tuned for DGX OS.

3

Then, if NVIDIA still provides GPU drivers for DGX Spark if customer is using other OS? I tried to look for those GPU drivers but in vain. So, if customer insists to install GPU drivers, then what we as a NVIDIA partner should tell them?

4

TLDR: yes, but with some efforts.

There is a long thread talking about options in this forum: Has anyone tried an alternative Linux distro?

You can also find some semi-official (preview) support, like this one: Building a custom Red Hat Enterprise Linux kernel for NVIDIA DGX Spark. Of course, since DGX OS is based on Ubuntu, you can use Ubuntu as well.

Theoretically, you can choose whatever Linux distro you’d like (or even Windows, if Nvidia doesn’t gatekeep the WoA for RTX Spark). It comes with several caveats, though:

  1. No matter which distro you choose, it’s better to use the Nvidia-maintained kernels, meaning that you need to compile kernels yourself (or rely on others to do it for you).
  2. Since you will be using a custom kernel, when trying to install the Nvidia driver, you are highly unlikely to be able to depend on your distro’s package maintainer.
  3. Some distros ship newer toolchains as their default, which may make the CUDA stack unhappy. This usually is not a big problem if your distro provides older toolchains for compatibility, which most do.

I’m currently running Fedora 44 on both of my GX10:

$ fastfetch
             .',;::::;,'.                 hiroshiya@GX10-00
         .';:cccccccccccc:;,.             -----------------
      .;cccccccccccccccccccccc;.          OS: Fedora Linux 44 (Forty Four) aarch64
    .:cccccccccccccccccccccccccc:.        Host: GX10 (5.36_GX10DGX)
  .;ccccccccccccc;.:dddl:.;ccccccc;.      Kernel: Linux 7.0.12-nv-1016.16
 .:ccccccccccccc;OWMKOOXMWd;ccccccc:.     Uptime: 2 days, 1 hour, 45 mins
.:ccccccccccccc;KMMc;cc;xMMc;ccccccc:.    Packages: 1315 (rpm)
,cccccccccccccc;MMM.;cc;;WW:;cccccccc,    Shell: bash 5.3.9
:cccccccccccccc;MMM.;cccccccccccccccc:    Display: 2560x1440 in 27", 144 Hz [External]
:ccccccc;oxOOOo;MMM000k.;cccccccccccc:    Terminal: /dev/pts/0 10.2p1
cccccc;0MMKxdd:;MMMkddc.;cccccccccccc;    CPU: Cortex-A725+Cortex-X925+Cortex-A725+Cortex-X925 (10+10) @ 3.90 GHz
ccccc;XMO';cccc;MMM.;cccccccccccccccc'    GPU: NVIDIA GB10
ccccc;MMo;ccccc;MMW.;ccccccccccccccc;     Memory: 3.90 GiB / 121.63 GiB (3%)
ccccc;0MNc.ccc.xMMd;ccccccccccccccc;      Swap: 0 B / 24.00 GiB (0%)
cccccc;dNMWXXXWM0:;cccccccccccccc:,       Disk (/): 85.88 GiB / 930.51 GiB (9%) - btrfs
cccccccc;.:odl:.;cccccccccccccc:,.        Locale: en_US.UTF-8
ccccccccccccccccccccccccccccc:'.          
:ccccccccccccccccccccccc:;,..
 ':cccccccccccccccc::;,.

$ nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.84                 Driver Version: 595.84         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0  On |                  N/A |
| N/A   43C    P8              3W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

It actually took me quite a few days to get the system working properly, because I have never compiled a Linux kernel before. Besides, I had to use the Runfiles to install both the driver and the CUDA stack. Nonetheless, I learned a lot during the process, and I was able to get NCCL working properly - even submitted a small patch for nccl-tests to make it seamlessly compile on Fedora.

Speaking of the driver, you can just download it from the official website. GB10 is not listed there, but it doesn’t really matter - the driver is basically universal. Just choose a same-generation product (e.g. “RTX Pro 6000”), and make sure to select “Linux aarch64” as the architecture.

If you want to be 100% sure about compatibility, you can also click “View” to go to the detail page of the driver, click the “Additional Information” tab, find the “README”, and find the “Supported NVIDIA GPU Products” section to confirm the compatibility. Here is an example from the 595.91.07 driver, where you can see GB10 is listed.

You will also need to manually install CUDA as well. Here is the guide for that. You will likely need to perform the “Runfile Installation” for CUDA as well.

5

The DGX Spark is using ACPI and not DT so any newer Linux distros will work. There’s now a NixOS based image for the Spark at GitHub - graham33/nixos-dgx-spark: Use Nix and NixOS on your DGX Spark! · GitHub

However, a lot of NVIDIA software is build specifically for Ubuntu 24.04 like the Nvidia Workbench or the Field Diagnostics suite, etc. If you enjoy tinkering with a Linux distro than any will do. If you look for stability and convenience using what NVIDIA offers is the best option!

6

Would be nice to use any distro that’s not debian/ubuntu based.

7

Thank you, it helps a lot!

8

In the fall, there will be windows-based AI PCs, that use a slight variant of the GB0 (the NX1, which will have the same ARM cores and NPU cores and memory interface as the GB1-). It will probably also be cheaper, and supposedly use a lot less power. The same mfrs offering the DGX Spark will be selling this version, and probably more, there will also be laptops.