Has anyone tried an alternative Linux distro?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
21

Looks like NixOS is using the r8169 driver, which is likely an issue!:

[graham@nixos:~]$ lspci | grep Ethernet
0000:01:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0000:01:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0002:01:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0002:01:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0007:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8127 (rev 05)

[graham@nixos:~]$ lspci -k -s 0007:01:00.0
0007:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8127 (rev 05)
	Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
	Kernel driver in use: r8169
	Kernel modules: r8169

It doesnā€™t look like my kernel has it:

[graham@nixos:/tmp]$ ls linux-6.17.5/drivers/net/ethernet/realtek/
8139cp.c   atp.c  Kconfig   r8169_firmware.c  r8169.h       r8169_main.c        rtase
8139too.c  atp.h  Makefile  r8169_firmware.h  r8169_leds.c  r8169_phy_config.c

But the NVIDIA kernel referenced above does have it, via an NVIDIA commit:

https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-6.17-next/drivers/net/ethernet/realtek/r8127

Iā€™ll try to build the NVIDIA kernel this weekend. Thanks!

22

I think you nailed it. No module for r8127, itā€™s using r8169 on Fedora.
I guess Iā€™ll try to download 8127 from realtek and blacklist 8169.

23

Well, I did it!

Compiled this branch: GitHub - NVIDIA/NV-Kernels at 24.04_linux-nvidia-6.17-next
Itā€™s based on 6.17.1 which lags behind the latest stable in 6.17 branch, but not as much as 6.11.

I reused config from DGX OS, but set a few parameters:

    1. Scheduler controlled preemption model (PREEMPT_LAZY) (NEW)
  • Module versioning implementation: genksyms
  • No per-page mapcount (EXPERIMENTAL) (NO_PAGE_MAPCOUNT) [N/y/?] (NEW) - SET TO YES!!!
    • looks like this is the most important option as it improves large memory copy operations
  • Enable r8127 as module (disabled by default)
  • CPU Power Management/CPU Frequency scaling/Default CPUFreq governor: schedutil (CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL=y) so the CPU frequency scales down when idle

Left the rest as default. I think I turned on a few related to Mellanox stuff and some ARM-related, but donā€™t remember which ones.

Need to unset these ones if not using Ubuntu: CONFIG_SYSTEM_TRUSTED_KEYS, CONFIG_SYSTEM_REVOCATION_KEYS (or set to new values).

Either disable r8169 in config, or in/etc/modprobe.d/blacklist.conf:

blacklist r8169

Otherwise it will load by default.

Compile, install kernel, sudo poweroff, then power on to avoid Ethernet adapter malfunction.
Check that r8127 is loaded via lspci -k | grep r8127

Install NVIDIA drivers.
Reboot. Note that Ethernet is not glitching anymore!

Make sure nvidia-persistenced is enabled, so power management works properly.

So, with all that, Iā€™m getting the same benchmarks in llama.cpp that I was getting in DGX OS, but the model loading is now 4-5x faster! 20 seconds vs more than a minute for gpt-oss-120b!

Benchmarks:

model size params backend test t/s
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 1956.03 Ā± 9.28
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 60.57 Ā± 0.25
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 @ d4096 1637.34 Ā± 4.86
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 @ d4096 54.14 Ā± 0.08
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 @ d8192 1512.01 Ā± 5.66
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 @ d8192 51.54 Ā± 0.14
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 @ d16384 1307.42 Ā± 3.79
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 @ d16384 47.45 Ā± 0.08
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA pp2048 @ d32768 1027.31 Ā± 4.79
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA tg32 @ d32768 40.55 Ā± 0.13
24

Thanks, this is working well on NixOS. The network interface stays up after a reboot. I found I didnā€™t need to change most of the options you mentioned, since the NVIDIA-provided defconfig seemed reasonable. The only ones I overrode were:

CONFIG_FAULT_INJECTION = lib.mkForce no; # fault injection might add overhead/don't think I need it?
CONFIG_SECURITY_APPARMOR_RESTRICT_USERNS = yes; # NixOS enables AppArmor by default
CONFIG_UBUNTU_HOST = no; # Not Ubuntu!

Hereā€™s the config if anyoneā€™s interested: nixos-dgx-spark/modules/dgx-spark.nix at main Ā· graham33/nixos-dgx-spark Ā· GitHub

25

@eugr what benchmark are you using above exactly? Iā€™d be interested to see if I can reproduce your numbers. Thanks.

26

Sure, here we go (first, build llama.cpp from the source):

To test model loading time and download the model from Huggingface:

build/bin/llama-server -hf ggml-org/gpt-oss-120b-GGUF -fa 1 -ngl 999 -ub 2048 -c 0 --jinja --reasoning-format auto --no-mmap

Then run benchmark:

build/bin/llama-bench -m ~/.cache/llama.cpp/ggml-org_gpt-oss-120b-GGUF_gpt-oss-120b-mxfp4-00001-of-00003.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0
27

So, 6.17.1 solved model loading issues without mmap, but mmap performance is still mediocre :( Not a problem with llama.cpp, but absolutely a problem with pyTorch based workflows, like vllm.

Now you can speed it up by using --safetensors-load-strategy eager, but then vllm process takes a lot of additional CPU memory (like 1/4 of the overall model size). For Qwen3-Next-80B in FP8 quant, thatā€™s additional 20GB of RAM!

I created a separate post about it - want to know if NVIDIA is aware and whether there are any plans on fixing this.

28

Iā€™m not familiar with NixOS kernel build process, but how does r8127 get included in your kernel if the config option is not set in arm64 defconfig? Am I missing something?

29

@eugr please see debian.nvidia-6.17/config/annotations for details:

CONFIG_R8127 policy<{ā€˜amd64ā€™: ā€˜nā€™, ā€˜arm64ā€™: ā€˜mā€™}>

It builds the module if ARM is detected.

30

So, nix can use debian annotations to build the kernel?

Definitely doesnā€™t work when using standard Makefile. I based my .config on the one from DGX OS that came with Spark, so it works, but when I tried to generate one using defconfig, it didnā€™t even boot.

31

Sorry for the late response. In this case Iā€™m telling nix to use defconfig, which ends up doing a make defconfig in the kernel source, which pulls in `arch/$(ARCH)/configs/defconfig`. Youā€™re right that it doesnā€™t have CONFIG_R8127=m, so I assume this must be implied by another option

32

BTW, I generated .config using Debianā€™s annotations script that picked up NVIDIA-specific annotations. It differs in some options from mine, but works ok. Just needed to replace AppArmor with Fedora by default. I also changed default CPU governor to schedutil, so it ramps down frequencies at idle.

Turned out NO_MAPCOUNT option doesnā€™t really affect performance. What does is hugepage support, but itā€™s on in NVidia annotations too.

Out of curiosity tried 6.14 kernel from DGX OS packages. The regular version didnā€™t make any difference, 64k pages version improved --no-mmap loading, but hangs on mmap allocations, so rolled it back to the default kernel for now.

33

On another note, I tried out pytorch 2.9 with CUDA 13 on NixOS, and am running into a weird issue loading torch CUDA support (which causes torch.cuda.is_available() to return False):

grep -i -A 10 -B 10 error /tmp/ld.out

    271892:

    271892:

    271892:     calling init: /nix/store/ijci9ppcf6sfnhah49q75ncqjsmi7ngz-cuda13.0-libcublas-13.1.0.3-lib/lib/libcublas.so.13

    271892:

    271892:

    271892:     calling init: /nix/store/z3di9xkjrk95sv8fjryk767767vdqh3h-cuda13.0-libcurand-10.4.0.35-lib/lib/libcurand.so.10

    271892:

    271892:

    271892:     calling init: /nix/store/61skgh54ddpqz6xnpwg3sq81fs9k9qj8-cuda13.0-libcusparse_lt-0.8.1.1-lib/lib/libcusparseLt.so.0

    271892:

    271892:     /nix/store/m0b67b3lmjcxa8aplpl75qpb26gr5vsf-python3-3.13.8/bin/python: error: symbol lookup error: undefined symbol: InitializeInjectionCaskEnhanced (fatal)

    271892:

    271892:     calling init: /nix/store/xwxym0y0az7sd48x3gkxi9kl4nw28c3k-cuda13.0-libcufft-12.0.0.61-lib/lib/libcufft.so.12

    271892:

    271892:

    271892:     calling init: /nix/store/8fjxxc3jnb16x4ka014gda3d31rjyviz-cuda13.0-libcusparse-12.6.3.3-lib/lib/libcusparse.so.12

    271892:

    271892:

    271892:     calling init: /nix/store/zmjqwzhgl9hwr8xnk8raj8d50lzkql66-gcc-14.3.0-lib/lib/libgomp.so.1

    271892:

    271892:

Just posting here on the off-chance that someone else has seen it!

34

I havenā€™t seen this issue, but what I found out is that for some workflows, pytorch with cuda 13 is slower than pytorch with cuda 12.9. One example is ComfyUI - I get 96 seconds on cu129 vs 139 seconds on cu130 using default Qwen Image workflow. Both are still using installed Cuda 13 SDK. Doesnā€™t seem to make any difference for vllm though.

35

Thanks for all the tips on getting Fedora 43 working. I decided to skip the ethernet issue by using a USB Ethernet dongle instead of the Ethernet port.

I decided to install the Fedora Server version, and made it headless, with a minimal install and only install packages as needed. I wanted ā€œnvidia-smiā€ to have the same output as the original build. And I got that now:

root@fedora:~# uname -a
Linux fedora 6.17.7-300.fc43.aarch64 #1 SMP PREEMPT_DYNAMIC Sun Nov  2 15:33:04 UTC 2025 aarch64 GNU/Linux
root@fedora:~# nvidia-smi
Fri Nov  7 13:41:20 2025
Ā±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
Ā±----------------------------------------Ā±-----------------------Ā±---------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   40C    P8              5W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
Ā±----------------------------------------Ā±-----------------------Ā±---------------------+

Ā±----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
Ā±----------------------------------------------------------------------------------------+

For others who want to know the setup I did to get the same config (as root):

nvidia-smi -pm 1
systemctl enable --now nvidia-persistenced
grubby --update-kernel=ALL --args=ā€œnvidia-drm.modeset=0ā€
reboot

36

Just be aware, that stock Fedora kernel is significantly slower in GPU workloads. I havenā€™t tried 6.17.7 yet, but I donā€™t think any NVIDIA-specific changes made it there. However, Fedora 43 Server with custom built kernel performs better than stock DGX OS :)

37

Thanks for the heads up on the kernel differences. I wish I could spend more time on this. It definitely feels like it is a memory access problem. I do have two devices, one I have left using DGX OS. That does make it easier to figure out the differences. I would be interested to know what they changed to make DGX OS quicker with the GPU workloads!

38

Got some more cycles to play with the dgx devices. Still havenā€™t built a custom kernel, but hope to do that soon. Thought, I might test the ConnectX-7 setup. With a vanilla Fedora kernel on one end and a DGX OS on the other end. I am getting around 100Gb/sec. Is 100 Gb/sec what I should be getting? I got myself a 200G cable from fs., but I seem to remember reading somewhere it will be a lot less than 200G.


root@fedora:~# ib_write_bw -d rocep1s0f0 -R -s 1048576 -q 8 -Q 8 -D 10 --report_gbits 192.168.211.219

Warning: Large message requested and CQ moderation enabled
Warning: It can lead to inaccurate results
WARNING: BW peak wonā€™t be measured in this run.

                RDMA_Write BW Test

Dual-port       : OFF		Device         : rocep1s0f0
Number of qps   : 8		Transport type : IB
Connection type : RC		Using SRQ      : OFF
PCIe relax order: ON		Lock-free      : OFF
ibv_wr* API     : ON		Using DDP      : OFF
TX depth        : 128
CQ Moderation   : 8
CQE Poll Batch  : 16
Mtu             : 1024[B]
Link type       : Ethernet
GID index       : 3
Max inline data : 0[B]
rdma_cm QPs	 : ON
Data ex. method : rdma_cm

local address: LID 0000 QPN 0x012b PSN 0xd50685
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:218
local address: LID 0000 QPN 0x012c PSN 0xf2ae8b
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:218
local address: LID 0000 QPN 0x012d PSN 0xbab479
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:218
local address: LID 0000 QPN 0x012e PSN 0x562cd4
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:218
local address: LID 0000 QPN 0x012f PSN 0xd94457
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:218
local address: LID 0000 QPN 0x0130 PSN 0xb4986d
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:218
local address: LID 0000 QPN 0x0131 PSN 0x3c95c8
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:218
local address: LID 0000 QPN 0x0132 PSN 0x82b221
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:218
remote address: LID 0000 QPN 0x012b PSN 0x7679b8
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:219
remote address: LID 0000 QPN 0x012c PSN 0xab0079
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:219
remote address: LID 0000 QPN 0x012d PSN 0xc2f799
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:219
remote address: LID 0000 QPN 0x012e PSN 0x330e3d
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:219
remote address: LID 0000 QPN 0x012f PSN 0xac0743
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:219
remote address: LID 0000 QPN 0x0130 PSN 0xab2bb3
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:219
remote address: LID 0000 QPN 0x0131 PSN 0x9bbb60
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:219
remote address: LID 0000 QPN 0x0132 PSN 0x75d0a3
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:211:219


     
Bytes      Iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
1048576    71256            0.00               99.62  		     0.011876
39

It would be good to get some clarity from NVidia on this. For instance, is 200 Gb/s achievable only by using both ports in aggregate, or we should be able to get it from a single port?

40

Iā€™ve asked for clarification, but according to this post it looks like you have to aggregate p1 and P2 ports to achieve max bandwidth even if they are mapped to the same physical interface, which kinda makes sense, because they could only provide x4 PCie per interface.

I may get another Spark later this week and test it.