Has anyone tried an alternative Linux distro?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
21

Looks like NixOS is using the r8169 driver, which is likely an issue!:

[graham@nixos:~]$ lspci | grep Ethernet
0000:01:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0000:01:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0002:01:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0002:01:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0007:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8127 (rev 05)

[graham@nixos:~]$ lspci -k -s 0007:01:00.0
0007:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8127 (rev 05)
	Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
	Kernel driver in use: r8169
	Kernel modules: r8169

It doesnā€™t look like my kernel has it:

[graham@nixos:/tmp]$ ls linux-6.17.5/drivers/net/ethernet/realtek/
8139cp.c   atp.c  Kconfig   r8169_firmware.c  r8169.h       r8169_main.c        rtase
8139too.c  atp.h  Makefile  r8169_firmware.h  r8169_leds.c  r8169_phy_config.c

But the NVIDIA kernel referenced above does have it, via an NVIDIA commit:

Iā€™ll try to build the NVIDIA kernel this weekend. Thanks!

22

I think you nailed it. No module for r8127, itā€™s using r8169 on Fedora.
I guess Iā€™ll try to download 8127 from realtek and blacklist 8169.

23

Well, I did it!

Compiled this branch: GitHub - NVIDIA/NV-Kernels at 24.04_linux-nvidia-6.17-next
Itā€™s based on 6.17.1 which lags behind the latest stable in 6.17 branch, but not as much as 6.11.

I reused config from DGX OS, but set a few parameters:

    1. Scheduler controlled preemption model (PREEMPT_LAZY) (NEW)
  • Module versioning implementation: genksyms
  • No per-page mapcount (EXPERIMENTAL) (NO_PAGE_MAPCOUNT) [N/y/?] (NEW) - SET TO YES!!!
    • looks like this is the most important option as it improves large memory copy operations
  • Enable r8127 as module (disabled by default)
  • CPU Power Management/CPU Frequency scaling/Default CPUFreq governor: schedutil (CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL=y) so the CPU frequency scales down when idle

Left the rest as default. I think I turned on a few related to Mellanox stuff and some ARM-related, but donā€™t remember which ones.

Need to unset these ones if not using Ubuntu: CONFIG_SYSTEM_TRUSTED_KEYS, CONFIG_SYSTEM_REVOCATION_KEYS (or set to new values).

Either disable r8169 in config, or in/etc/modprobe.d/blacklist.conf:

blacklist r8169

Otherwise it will load by default.

Compile, install kernel, sudo poweroff, then power on to avoid Ethernet adapter malfunction.
Check that r8127 is loaded via lspci -k | grep r8127

Install NVIDIA drivers.
Reboot. Note that Ethernet is not glitching anymore!

Make sure nvidia-persistenced is enabled, so power management works properly.

So, with all that, Iā€™m getting the same benchmarks in llama.cpp that I was getting in DGX OS, but the model loading is now 4-5x faster! 20 seconds vs more than a minute for gpt-oss-120b!

Benchmarks:

model size params backend test t/s
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 1956.03 Ā± 9.28
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 60.57 Ā± 0.25
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 @ d4096 1637.34 Ā± 4.86
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 @ d4096 54.14 Ā± 0.08
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 @ d8192 1512.01 Ā± 5.66
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 @ d8192 51.54 Ā± 0.14
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 @ d16384 1307.42 Ā± 3.79
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 @ d16384 47.45 Ā± 0.08
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 @ d32768 1027.31 Ā± 4.79
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 @ d32768 40.55 Ā± 0.13
3 Likes
24

Thanks, this is working well on NixOS. The network interface stays up after a reboot. I found I didnā€™t need to change most of the options you mentioned, since the NVIDIA-provided defconfig seemed reasonable. The only ones I overrode were:

CONFIG_FAULT_INJECTION = lib.mkForce no; # fault injection might add overhead/don't think I need it?
CONFIG_SECURITY_APPARMOR_RESTRICT_USERNS = yes; # NixOS enables AppArmor by default
CONFIG_UBUNTU_HOST = no; # Not Ubuntu!

Hereā€™s the config if anyoneā€™s interested: nixos-dgx-spark/modules/dgx-spark.nix at main Ā· graham33/nixos-dgx-spark Ā· GitHub

2 Likes
25

@eugr what benchmark are you using above exactly? Iā€™d be interested to see if I can reproduce your numbers. Thanks.

26

Sure, here we go (first, build llama.cpp from the source):

To test model loading time and download the model from Huggingface:

build/bin/llama-server -hf ggml-org/gpt-oss-120b-GGUF -fa 1 -ngl 999 -ub 2048 -c 0 --jinja --reasoning-format auto --no-mmap

Then run benchmark:

build/bin/llama-bench -m ~/.cache/llama.cpp/ggml-org_gpt-oss-120b-GGUF_gpt-oss-120b-mxfp4-00001-of-00003.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0
1 Like
27

So, 6.17.1 solved model loading issues without mmap, but mmap performance is still mediocre :( Not a problem with llama.cpp, but absolutely a problem with pyTorch based workflows, like vllm.

Now you can speed it up by using --safetensors-load-strategy eager, but then vllm process takes a lot of additional CPU memory (like 1/4 of the overall model size). For Qwen3-Next-80B in FP8 quant, thatā€™s additional 20GB of RAM!

I created a separate post about it - want to know if NVIDIA is aware and whether there are any plans on fixing this.

28

Iā€™m not familiar with NixOS kernel build process, but how does r8127 get included in your kernel if the config option is not set in arm64 defconfig? Am I missing something?

29

@eugr please see debian.nvidia-6.17/config/annotations for details:

CONFIG_R8127 policy<{ā€˜amd64ā€™: ā€˜nā€™, ā€˜arm64ā€™: ā€˜mā€™}>

It builds the module if ARM is detected.

30

So, nix can use debian annotations to build the kernel?

Definitely doesnā€™t work when using standard Makefile. I based my .config on the one from DGX OS that came with Spark, so it works, but when I tried to generate one using defconfig, it didnā€™t even boot.

31

Sorry for the late response. In this case Iā€™m telling nix to use defconfig, which ends up doing a make defconfig in the kernel source, which pulls in `arch/$(ARCH)/configs/defconfig`. Youā€™re right that it doesnā€™t have CONFIG_R8127=m, so I assume this must be implied by another option

32

BTW, I generated .config using Debianā€™s annotations script that picked up NVIDIA-specific annotations. It differs in some options from mine, but works ok. Just needed to replace AppArmor with Fedora by default. I also changed default CPU governor to schedutil, so it ramps down frequencies at idle.

Turned out NO_MAPCOUNT option doesnā€™t really affect performance. What does is hugepage support, but itā€™s on in NVidia annotations too.

Out of curiosity tried 6.14 kernel from DGX OS packages. The regular version didnā€™t make any difference, 64k pages version improved --no-mmap loading, but hangs on mmap allocations, so rolled it back to the default kernel for now.

1 Like
33

On another note, I tried out pytorch 2.9 with CUDA 13 on NixOS, and am running into a weird issue loading torch CUDA support (which causes torch.cuda.is_available() to return False):

grep -i -A 10 -B 10 error /tmp/ld.out

    271892:

    271892:

    271892:     calling init: /nix/store/ijci9ppcf6sfnhah49q75ncqjsmi7ngz-cuda13.0-libcublas-13.1.0.3-lib/lib/libcublas.so.13

    271892:

    271892:

    271892:     calling init: /nix/store/z3di9xkjrk95sv8fjryk767767vdqh3h-cuda13.0-libcurand-10.4.0.35-lib/lib/libcurand.so.10

    271892:

    271892:

    271892:     calling init: /nix/store/61skgh54ddpqz6xnpwg3sq81fs9k9qj8-cuda13.0-libcusparse_lt-0.8.1.1-lib/lib/libcusparseLt.so.0

    271892:

    271892:     /nix/store/m0b67b3lmjcxa8aplpl75qpb26gr5vsf-python3-3.13.8/bin/python: error: symbol lookup error: undefined symbol: InitializeInjectionCaskEnhanced (fatal)

    271892:

    271892:     calling init: /nix/store/xwxym0y0az7sd48x3gkxi9kl4nw28c3k-cuda13.0-libcufft-12.0.0.61-lib/lib/libcufft.so.12

    271892:

    271892:

    271892:     calling init: /nix/store/8fjxxc3jnb16x4ka014gda3d31rjyviz-cuda13.0-libcusparse-12.6.3.3-lib/lib/libcusparse.so.12

    271892:

    271892:

    271892:     calling init: /nix/store/zmjqwzhgl9hwr8xnk8raj8d50lzkql66-gcc-14.3.0-lib/lib/libgomp.so.1

    271892:

    271892:

Just posting here on the off-chance that someone else has seen it!

34

I havenā€™t seen this issue, but what I found out is that for some workflows, pytorch with cuda 13 is slower than pytorch with cuda 12.9. One example is ComfyUI - I get 96 seconds on cu129 vs 139 seconds on cu130 using default Qwen Image workflow. Both are still using installed Cuda 13 SDK. Doesnā€™t seem to make any difference for vllm though.

35

Thanks for all the tips on getting Fedora 43 working. I decided to skip the ethernet issue by using a USB Ethernet dongle instead of the Ethernet port.

I decided to install the Fedora Server version, and made it headless, with a minimal install and only install packages as needed. I wanted ā€œnvidia-smiā€ to have the same output as the original build. And I got that now:

root@fedora:~# uname -a
Linux fedora 6.17.7-300.fc43.aarch64 #1 SMP PREEMPT_DYNAMIC Sun Nov  2 15:33:04 UTC 2025 aarch64 GNU/Linux
root@fedora:~# nvidia-smi
Fri Nov  7 13:41:20 2025
Ā±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
Ā±----------------------------------------Ā±-----------------------Ā±---------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   40C    P8              5W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
Ā±----------------------------------------Ā±-----------------------Ā±---------------------+

Ā±----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
Ā±----------------------------------------------------------------------------------------+

For others who want to know the setup I did to get the same config (as root):

nvidia-smi -pm 1
systemctl enable --now nvidia-persistenced
grubby --update-kernel=ALL --args=ā€œnvidia-drm.modeset=0ā€
reboot

36

Just be aware, that stock Fedora kernel is significantly slower in GPU workloads. I havenā€™t tried 6.17.7 yet, but I donā€™t think any NVIDIA-specific changes made it there. However, Fedora 43 Server with custom built kernel performs better than stock DGX OS :)

1 Like
37

Thanks for the heads up on the kernel differences. I wish I could spend more time on this. It definitely feels like it is a memory access problem. I do have two devices, one I have left using DGX OS. That does make it easier to figure out the differences. I would be interested to know what they changed to make DGX OS quicker with the GPU workloads!