Ubuntu 26.04 + drivers 610 + cuda-toolkit 13.3 + ZFS on GX10

Finally managed to get fully working setup with Ubuntu 26.04 (clean of that DGX OS’s bloatware) with newest drivers/cuda and ZFS. Why ZFS? Because it has checksums and configurable disk cache size. Compression is not fully useless too (4% on my GGUFs collection), snapshots may help if you like to play roulette with different versions.

So far, tested with llama.cpp, speed is about same as DGX OS+595 drivers+13.3 cuda. Will test with vllm later.

Had to play with reinstalling 24.04 and 26.04 for two days to fix problem with additional 15W power consumption (it was connectx7, of course). Here is my steps:

First, backup your DGX OS drive. Insert big enough usb drive:

mount /dev/sda /mnt

dd if=/dev/nvme0n1p1 of=/mnt/efi.img bs=64k

mkdir /mnt/backuproot

rsync -aSHAXxv --numeric-ids / /mnt/backuproot/

Then install ubuntu 26.04 (desktop, ARM version), check both drivers and third-party software. At filesystem configuration - click on advanced option and select ZFS without encryption (you may choose to encrypt but be ready to enter passphrase every boot).

After install - open terminal, check that network is configured and install ssh:

sudo -i apt install openssh-server

systemctl start ssh.service

Then login through ssh:

sudo -i

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/sbsa/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt full-upgrade
apt install linux-nvidia-hwe-24.04-edge
apt autoremove

apt install cuda-toolkit-13-3 cuda-13-3 libnccl2 libnccl-dev libssl-dev curl cmake autoconf flex bison libtool git automake ccache

echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc

apt install nvtop nvidia-container-toolkit
apt install nvidia-open cuda

echo 'options zfs zfs_arc_max=2147483648'>> /etc/modprobe.d/zfs.conf
echo 'options zfs zfs_arc_min=1073741824'>> /etc/modprobe.d/zfs.conf

wget https://repo.download.nvidia.com/baseos/ubuntu/noble/arm64/pool/dgx/d/dgx-spark-mlnx-hotplug/dgx-spark-mlnx-hotplug_26.01-1_all.deb
dpkg -i dgx-spark-mlnx-hotplug_26.01-1_all.deb
update-initramfs -u

The system will be configured with 2GB max disk cache. After this you should able to clone llama.cpp and compile it.

Note, if you don’t leave the system working 24/7 then do trim nvme drive periodically (it does that once per month automatically):

zpool trim bpool

zpool trim rpool

I did almost the same thing, and Vulkan is faster with the new driver. The llama.cpp Vulkan backend is finally a viable option.

DGX

./target/aarch64-unknown-linux-gnu/release/memtest_vulkan 
https://github.com/GpuZelenograd/memtest_vulkan v0.6.0 by GpuZelenograd
To finish testing use Ctrl+C
TU: error: ../src/freedreno/vulkan/tu_knl.cc:387: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
MESA: error: Opening /dev/dri/card0 failed: Permission denied

1: Bus=0x01:00 DevId=0x2E12   92GB NVIDIA GB10
2: Bus=0x00:00 DevId=0x0000   122GB llvmpipe (LLVM 20.1.2, 128 bits)
                                                   Override index to test:1
TU: error: ../src/freedreno/vulkan/tu_knl.cc:387: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
MESA: error: Opening /dev/dri/card0 failed: Permission denied
Standard 5-minute test of 1: Bus=0x01:00 DevId=0x2E12   92GB NVIDIA GB10
          NO ERRORS           written:   86.2GB  176.8GB/s      checked:   90.0GB  191.7GB/s     00:00:00.957
          NO ERRORS           written:  172.5GB  177.2GB/s      checked:  180.0GB  192.5GB/s     00:00:02.866
          NO ERRORS           written:  517.5GB  176.9GB/s      checked:  540.0GB  192.5GB/s     00:00:08.597
          NO ERRORS           written: 2760.0GB  176.6GB/s      checked: 2880.0GB  192.6GB/s     00:00:39.174

Ubuntu 26

./target/aarch64-unknown-linux-gnu/release/memtest_vulkan 
https://github.com/GpuZelenograd/memtest_vulkan v0.6.0 by GpuZelenograd
To finish testing use Ctrl+C

1: Bus=0x01:00 DevId=0x2E12   92GB NVIDIA GB10
2: Bus=0x00:00 DevId=0x0000   122GB llvmpipe (LLVM 21.1.8, 128 bits)
                                                   Override index to test:1
Standard 5-minute test of 1: Bus=0x01:00 DevId=0x2E12   92GB NVIDIA GB10
          NO ERRORS           written:   86.2GB  179.0GB/s      checked:   90.0GB  215.3GB/s     00:00:00.899
          NO ERRORS           written:  172.5GB  178.4GB/s      checked:  180.0GB  214.8GB/s     00:00:02.704
          NO ERRORS           written:  517.5GB  178.5GB/s      checked:  540.0GB  215.3GB/s     00:00:08.111
          NO ERRORS           written: 2932.5GB  178.2GB/s      checked: 3060.0GB  214.5GB/s     00:00:38.833

Interesting, i’ll have to try. Which vulkan sdk is compatible with nvidia drivers?

Well, bad and good news.

Good news, vllm from eugr/spark-vllm-docker and comfyui work fine (don’t see any changes, same speed with llama-benchy), just have to install few more packages from ubuntu repository.

Bad news, it still messes up with buffers/cache memory (though llama.cpp ignores that, only vllm is affected). Buffers get allocated but not freed automatically, only manual cache drop works. But looks like it is done not by OS but by nvidia driver itself, unloading and loading again nvidia driver does help (“for m in nvidia_uvm nvidia_drm nvidia_modeset nvidia; do modprobe -r “$m” 2>/dev/null || true; done; modprobe nvidia; modprobe nvidia_uvm”).

To get vllm docker working:

apt install python3-pip python3-wheel python3-dev libpython3-dev  zlib1g-dev python3-venv python3-setuptools-whl python3-setuptools python3-pip-whl

apt install docker.io docker-buildx docker-compose-v2

You could play with Ubuntu 26.04 but won’t be able to install the DGX Software Stack because the repository is not available, yet.

It adds these metapackages:

nvidia-system-core

nvidia-system-utils

nvidia-system-extra

nvidia-system-station

on top of Ubuntu base in order to make it a DGX. And the NVIDIA drivers.

See details in the latest dgx-os-7-user-guide.pdf section 6.4

Well, that was kinda point to not install this overbloated mess. There is nothing useful for me, just bunch of metapackages, weird scripts and settings that has almost nothing to do with using local models, a lot of telemetry stuff and really useless dashboards.

Btw, you can install nvidia-system-core to 26.04 and a lot of these packages don’t depend on ubuntu version or CPU architecture at all, just really no point. I’ve took only dgx-spark-mlnx-hotplug package from the repository as it fixes power consumption of the connectx7 ports when in idle mode.

I use this

Maybe it works as is but i rebuild it
./vulkansdk glslang vulkan-headers shaderc -j 10

And build llama.cpp with

export VULKAN_SDK=/opt/vulkan/aarch64
export PATH=$PATH:$VULKAN_SDK/bin
export LD_LIBRARY_PATH=$VULKAN_SDK/lib
export VK_LAYER_PATH=$VULKAN_SDK/etc/explicit_layer.d

NVIDIA GB10: Ubuntu 24.04 + NVIDIA 580 vs Ubuntu 26.04 + NVIDIA 610 in clpeak

Test environment

Backend Ubuntu 24.04 Ubuntu 26.04
GPU NVIDIA GB10 NVIDIA GB10
OpenCL driver 580.159.03 610.43.02
Vulkan driver 580.636.192 610.172.128
Vulkan API 1.4.312 1.4.341
CUDA version reported 13.0 13.3
CUDA toolkit 13.0 13.3
CUDA architecture sm_121 sm_121
Compute units / SMs 48 48
GPU clock reported by OpenCL 2418 MHz 2418 MHz
CUDA VRAM reported 124546 MB 124543 MB
Vulkan VRAM reported 93409 MB 93407 MB

OpenCL

Metric Unit Ubuntu 24.04 / 580 Ubuntu 26.04 / 610 Change
Compute units count 48 48 0.00%
Clock frequency MHz 2418 2418 0.00%
FP32 float GFLOPS 29726.33 29778.45 +0.18%
FP32 float2 GFLOPS 29262.52 29374.07 +0.38%
FP32 float4 GFLOPS 29280.39 29391.47 +0.38%
FP32 float8 GFLOPS 29012.17 28830.79 −0.63%
FP32 float16 GFLOPS 28772.67 28667.33 −0.37%
FP16 compute Status Unsupported Unsupported —
FP64 double GFLOPS 484.95 482.67 −0.47%
FP64 double2 GFLOPS 484.38 482.11 −0.47%
FP64 double4 GFLOPS 483.20 480.97 −0.46%
FP64 double8 GFLOPS 480.86 478.67 −0.46%
FP64 double16 GFLOPS 476.50 474.25 −0.47%
Mixed FP16×FP16+FP32 Status Unsupported Unsupported —
INT32 int GOPS 15084.46 15006.20 −0.52%
INT32 int2 GOPS 15023.41 14961.03 −0.42%
INT32 int4 GOPS 14849.79 14905.26 +0.37%
INT32 int8 GOPS 14905.26 14960.43 +0.37%
INT32 int16 GOPS 14903.11 14963.71 +0.41%
Fast 24-bit int GOPS 14878.55 14940.19 +0.41%
Fast 24-bit int2 GOPS 14905.68 14960.25 +0.37%
Fast 24-bit int4 GOPS 14847.65 14909.46 +0.42%
Fast 24-bit int8 GOPS 14757.36 14813.00 +0.38%
Fast 24-bit int16 GOPS 14556.00 14610.26 +0.37%
INT8 char GOPS 13624.00 13553.30 −0.52%
INT8 char2 GOPS 12614.62 12567.72 −0.37%
INT8 char4 GOPS 12648.80 12608.85 −0.32%
INT8 char8 GOPS 11099.16 11071.12 −0.25%
INT8 char16 GOPS 9320.45 9296.15 −0.26%
INT16 short GOPS 13455.57 13404.96 −0.38%
INT16 short2 GOPS 12697.26 12647.22 −0.39%
INT16 short4 GOPS 12741.41 12691.51 −0.39%
INT16 short8 GOPS 11240.24 11203.36 −0.33%
INT16 short16 GOPS 8919.27 8903.45 −0.18%
INT8 dot product Status Unsupported Unsupported —
Global memory float GB/s 233.38 234.55 +0.50%
Global memory float2 GB/s 241.93 242.22 +0.12%
Global memory float4 GB/s 246.82 246.07 −0.30%
Global memory float8 GB/s 246.99 247.36 +0.15%
Global memory float16 GB/s 124.97 125.21 +0.19%
Local memory float GB/s 6349.66 6377.70 +0.44%
Local memory float2 GB/s 11492.80 11547.33 +0.47%
Local memory float4 GB/s 13214.73 13264.57 +0.38%
Local memory float8 GB/s 7260.44 7285.53 +0.35%
Image memory float4 GB/s 120.91 114.41 −5.38%
enqueueWriteBuffer GB/s 59.08 59.68 +1.02%
enqueueReadBuffer GB/s 59.42 59.77 +0.59%
enqueueWriteBuffer nonblocking GB/s 59.25 59.89 +1.08%
enqueueReadBuffer nonblocking GB/s 59.42 59.84 +0.71%
enqueueMapBuffer GB/s 58.77 58.41 −0.61%
memcpy_from_mapped_ptr GB/s 27.19 24.60 −9.53%
enqueueUnmap GB/s 59.07 58.96 −0.19%
memcpy_to_mapped_ptr GB/s 27.34 24.57 −10.13%
Kernel dispatch latency µs 5.70 5.58 −2.11%
Kernel round-trip latency µs 14.18 14.05 −0.92%

Vulkan

Metric Unit Ubuntu 24.04 / 580 Ubuntu 26.04 / 610 Change
Compute units count 48 48 0.00%
Reported VRAM MB 93409 93407 −0.00%
FP32 float GFLOPS 30142.00 30258.23 +0.39%
FP32 float2 GFLOPS 30034.44 30145.53 +0.37%
FP32 float4 GFLOPS 30048.27 24545.48 −18.31%
FP16 half GFLOPS 15310.84 15370.63 +0.39%
FP16 half2 GFLOPS 14176.33 30367.81 +114.21%
FP16 half4 GFLOPS 15246.73 30270.29 +98.54%
FP64 double GFLOPS 481.56 480.21 −0.28%
FP64 double2 GFLOPS 472.80 474.49 +0.36%
FP64 double4 GFLOPS 458.93 460.74 +0.39%
Mixed FP16×FP16+FP32 mp GFLOPS 23967.58 25258.62 +5.39%
Mixed FP16×FP16+FP32 mp2 GFLOPS 23727.53 25015.35 +5.43%
Mixed FP16×FP16+FP32 mp4 GFLOPS 20210.62 20355.87 +0.72%
BF16 bf16 GFLOPS 29616.61 26837.69 −9.38%
BF16 bf16_2 GFLOPS 29330.19 26415.70 −9.94%
BF16 bf16_4 GFLOPS 28695.49 21514.27 −25.03%
Cooperative matrix FP32 Status Unsupported Unsupported —
Cooperative matrix FP16 TFLOPS 120.38 120.94 +0.47%
Cooperative matrix BF16 TFLOPS 120.36 120.85 +0.41%
Cooperative matrix FP8 E4M3 TFLOPS Unsupported 240.73 New
Cooperative matrix FP8 E5M2 TFLOPS Unsupported 241.04 New
INT32 int GOPS 9985.42 10081.76 +0.96%
INT32 int2 GOPS 9929.33 9964.91 +0.36%
INT32 int4 GOPS 9937.03 9980.77 +0.44%
INT8 dot int8_dp GOPS 13732.91 24984.84 +81.93%
INT8 dot int8_dp2 GOPS 12851.18 28386.58 +120.89%
INT8 dot int8_dp4 GOPS 13656.91 29359.68 +114.98%
Cooperative matrix INT8 TOPS 241.65 242.70 +0.43%
Global memory float GB/s 235.15 234.85 −0.13%
Global memory float2 GB/s 242.71 242.91 +0.08%
Global memory float4 GB/s 242.32 242.84 +0.21%
Local memory float GB/s 6238.58 6372.11 +2.14%
Local memory float2 GB/s 10827.55 11486.60 +6.09%
Local memory float4 GB/s 12225.40 13204.80 +8.01%
Image memory float4 GB/s 222.76 224.19 +0.64%
Host → device transfer GB/s 123.21 122.65 −0.45%
Device → host transfer GB/s 122.09 121.67 −0.34%
Kernel dispatch latency µs 4.59 3.84 −16.34%
Kernel round-trip latency µs 48.18 48.99 +1.68%

CUDA

Metric Unit Ubuntu 24.04 / 580 Ubuntu 26.04 / 610 Change
SMs count 48 48 0.00%
Reported VRAM MB 124546 124543 −0.00%
FP32 float GFLOPS 30568.84 30651.12 +0.27%
FP16 half GFLOPS 30120.23 30210.82 +0.30%
FP16 half2 GFLOPS 30501.67 30617.60 +0.38%
FP64 double GFLOPS 484.77 486.47 +0.35%
Mixed FP16×FP16+FP32 GFLOPS 25473.42 25540.70 +0.26%
BF16×BF16+FP32 GFLOPS 28655.62 28753.13 +0.34%
WMMA FP16 TFLOPS 441.62 436.07 −1.26%
WMMA BF16 TFLOPS 441.58 435.61 −1.35%
WMMA TF32 TFLOPS 121.77 118.64 −2.57%
WMMA FP64 TFLOPS 1.70 1.70 0.00%
FP8 E4M3 MMA TFLOPS 237.19 239.43 +0.94%
FP8 E5M2 MMA TFLOPS 237.22 239.42 +0.93%
FP4 E2M1 Status Compile/load failed Compile/load failed —
MXFP4 E2M1 Status Compile/load failed Compile/load failed —
NVFP4 E2M1 Status Compile/load failed Compile/load failed —
Sparse MXFP4 Status Compile/load failed Compile/load failed —
Sparse NVFP4 Status Compile/load failed Compile/load failed —
INT32 IMAD GOPS 15177.19 15220.15 +0.28%
INT8 dot int8_dp GOPS 47164.96 47130.71 −0.07%
INT8 dot int8_dp2 GOPS 49252.26 49105.77 −0.30%
INT8 dot int8_dp4 GOPS 46378.27 46526.51 +0.32%
INT8 dot int8_dp8 GOPS 47440.56 47492.58 +0.11%
WMMA INT8 TOPS 477.59 474.63 −0.62%
INT8 MMA k32 TOPS 235.57 239.39 +1.62%
Sparse INT8 MMA TOPS 238.45 239.45 +0.42%
INT4 MMA Status Unsupported Unsupported —
Global memory float GB/s 234.97 235.34 +0.16%
Global memory float2 GB/s 242.52 243.31 +0.33%
Global memory float4 GB/s 244.50 244.42 −0.03%
Local memory float GB/s 6306.25 6446.73 +2.23%
Local memory float2 GB/s 11423.36 11807.13 +3.36%
Local memory float4 GB/s 13045.15 13276.49 +1.77%
Image memory float4 GB/s 114.45 114.57 +0.10%
Pinned host → device GB/s 59.00 58.86 −0.24%
Pinned device → host GB/s 58.87 58.91 +0.07%
Kernel dispatch latency Status Not measurable Not measurable —
Kernel round-trip latency µs 4.84 4.99 +3.10%

be aware, ZFS on non-ECC RAM system, like on gb10, may lead to data corruption, specially with overclocked memory.

In the same was that any filesystem without ECC RAM could lead to data corruption

However the situations in which not having ECC RAM would lead specifically to bad bytes being written to disk and not just an assert or crash are… probably pretty thin.

checksumming without ECC is still measurably better than no checksumming and no ECC, as it can still protect you from (or at least make you aware of) actual bitrot on the storage medium.

Just to add some nuance, rather than spread folk-lore.

(I probably still wouldn’t run ZFS on my Spark, but I do on my laptop without ECC RAM, because ZFS is awesome)

Yep, there are probably some risks. But you may lose almost any filesystem because silent memory corruption. ZFS would able to detect data blocks corruption at least in this case.

Anyway, i see refcompressration 1.13 (zstd-5 compression) and dedup ratio 1.05 on my disk. Kinda neat, 17% less data on disk with 1.5GB/s read speed.