Finally managed to get fully working setup with Ubuntu 26.04 (clean of that DGX OS’s bloatware) with newest drivers/cuda and ZFS. Why ZFS? Because it has checksums and configurable disk cache size. Compression is not fully useless too (4% on my GGUFs collection), snapshots may help if you like to play roulette with different versions.
So far, tested with llama.cpp, speed is about same as DGX OS+595 drivers+13.3 cuda. Will test with vllm later.
Had to play with reinstalling 24.04 and 26.04 for two days to fix problem with additional 15W power consumption (it was connectx7, of course). Here is my steps:
First, backup your DGX OS drive. Insert big enough usb drive:
mount /dev/sda /mnt
dd if=/dev/nvme0n1p1 of=/mnt/efi.img bs=64k
mkdir /mnt/backuproot
rsync -aSHAXxv --numeric-ids / /mnt/backuproot/
Then install ubuntu 26.04 (desktop, ARM version), check both drivers and third-party software. At filesystem configuration - click on advanced option and select ZFS without encryption (you may choose to encrypt but be ready to enter passphrase every boot).
After install - open terminal, check that network is configured and install ssh:
sudo -i apt install openssh-server
systemctl start ssh.service
Then login through ssh:
sudo -i
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/sbsa/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt full-upgrade
apt install linux-nvidia-hwe-24.04-edge
apt autoremove
apt install cuda-toolkit-13-3 cuda-13-3 libnccl2 libnccl-dev libssl-dev curl cmake autoconf flex bison libtool git automake ccache
echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc
apt install nvtop nvidia-container-toolkit
apt install nvidia-open cuda
echo 'options zfs zfs_arc_max=2147483648'>> /etc/modprobe.d/zfs.conf
echo 'options zfs zfs_arc_min=1073741824'>> /etc/modprobe.d/zfs.conf
wget https://repo.download.nvidia.com/baseos/ubuntu/noble/arm64/pool/dgx/d/dgx-spark-mlnx-hotplug/dgx-spark-mlnx-hotplug_26.01-1_all.deb
dpkg -i dgx-spark-mlnx-hotplug_26.01-1_all.deb
update-initramfs -u
The system will be configured with 2GB max disk cache. After this you should able to clone llama.cpp and compile it.
Note, if you don’t leave the system working 24/7 then do trim nvme drive periodically (it does that once per month automatically):
zpool trim bpool
zpool trim rpool