Hi everyone,
I’m sharing DGX Spark Memory Saver, a UVM driver patch that reduces GPU page-table memory overhead on DGX Spark systems running a 64 KiB Linux kernel.
In a three-node serving trial, it recovered 1.78-1.86 GiB of additional usable memory per node compared with the stock 4 KiB system. This is on top of the recent finding of memory available from firmware reserves of 2.1 GiB for display scanout.
On a 64 KiB kernel, NVIDIA UVM allocates an entire CPU page for each eligible 256-byte GPU leaf page table. Memory Saver places these tables in separate 4 KiB slots within shared pages, fitting sixteen tables into each page. This is analogous to the linux slab allocator.
On the profiled node, page-table backing storage fell from 3.25 GiB to 0.218 GiB - a saving of 3.03 GiB compared with the stock 64 KiB driver - and giving the ~1.8gb memory savings over the stock 4kb system.
The patch passed concurrent CUDA allocation and reuse tests, full-buffer readback, a 4.5 GiB pinned transfer, BF16 matrix multiplication, CUDA graph replay, and a number of other serving quality checks across three Sparks.
The repository provides:
-
Manual and DKMS installation workflows.
-
Secure Boot signing guidance.
-
Read-only status reporting with JSON output.
-
Removal and rollback instructions.
-
Benchmark reports and CUDA validation programs.
The tested configuration is DGX Spark / GB10, NVIDIA open driver 580.178.04, and Ubuntu kernel 7.0.0-1019-nvidia-64k. The documentation covers prerequisites, validation status, and the installation process.
Get the code and explore the results:
github.com/christopherowen/dgx-spark-memory-saver
Feedback, code review, and reproducible test results are welcome!
Thanks for sharing this, the numbers are really compelling: 512K context and ~2.85M KV tokens with the same r5o image.
Is there any chance this gets upstreamed into NVIDIA’s open GPU kernel modules / the UVM driver, so it ships in an official driver release?
We run three Sparks with Secure Boot enabled and no BMC, so a self-signed DKMS module means enrolling a MOK at the physical console on every node. If it landed in a driver update it would simply arrive signed through the normal DGX OS updates, which would make it much easier to adopt across the cluster.
NVIDIA team, is this something you are looking at?
I cannot speak for nvidia, of course.
I think they don’t need a random hacker’s help to do these things, but I’m willing if they reach out :).
The signing key enrollment is a bit of a pain, but it was necessary for dgx-spark-fan-control too.
Hopefully I left the instructions easy enough to follow as a human or agent. You could disable secure boot as an alternative but that requires console too.
I use one of the tiny glinet/jetkvm network kvm things to do it over the network.
Thanks a lot, that’s very helpful. A network KVM is exactly the missing piece for us, since it covers the MokManager step without being at the console. I’ll pick one up and follow your instructions node by node. Thanks also for publishing everything so clearly!
I’m dealing with the GPU memory fragmentation issues in which I’ll have 100 GiB of RAM free, but get the no memory failure when allocating. I imagine this could be used to reduce the impact. Are there other page sizes available and if so, have you tried them?
Thanks for sharing this. I’d be interested to understand what’s happening on your system. Do you have any logs or diagnostics that point to fragmentation? The exact error, allocation size, application, and how you’re measuring the 100 GiB free would be helpful.
Memory Saver reduces the memory used by GPU page tables, giving workloads some additional headroom. It could help indirectly by reducing memory pressure, but I wouldn’t want to promise it will resolve the allocation failure you’re seeing without understanding the cause.
The published results compare 4 KiB and 64 KiB kernels, with the patch running on the 64 KiB kernel. Linux on arm64 also has a 16 KiB configuration, though the project doesn’t currently have validated results for that on Spark.
Maybe you could offer some details on your kernel, driver, and firmware versions?
Did you try this ?
sync && sudo sh -c ‘echo 3 > /proc/sys/vm/drop_caches’
Has anyone tried this on an Asus Ascent GX10? Couldn’t tell at first glance if this was only for DGX Spark…
I have the Nvidia one, but I think there shouldn’t be a compatibility issue.
Please report back if you try
Memory Saver 0.4.1 is out
Two releases since the intro:
0.4.0: one package for reviewed driver versions. The builder picks the source manifest and patch from a compatibility table, based on the NVIDIA RM installed for the target kernel. R580 580.178.04 remains the validated profile. R610 610.57.04 is now conditional: it needs a separate RM DMA-alignment correction and system pools disabled (details). The allocator patch is unchanged. New read-only tools: ./scripts/driver-build check verifies a host’s packages, sources and headers before building, and ./scripts/status --json reports what’s installed and loaded.
0.4.1: installs again after this week’s apt upgrade. Ubuntu republished the R580 modules package for 7.0.0-1019-nvidia-64k as 7.0.0-1019.19~24.04.2+2, a packaging resync with the same modules, and removed +1 from the archive. 0.4.0 insisted on exactly +1, so on an updated system it refused to build or DKMS-install. 0.4.1 matches the package on the kernel version and accepts any packaging suffix; the source, patch and module identity checks are unchanged. The kernel guide’s install commands now take the archive’s current revision.
Upgrading from 0.2.0, 0.3.0 or 0.4.0: remove the old DKMS registration with GPU clients stopped, then register 0.4.1 (steps). If you already enrolled a MOK for an earlier version, the same key signs 0.4.1, so there’s no new console step.
It’s running on all four of my Sparks with DKMS since today, signed and loaded with packing on and serving in production. Its UVM module compiles to the same code as the 0.2.0 build these results came from; only the comments changed.
Still open from earlier in the thread: no validated results on 16 KiB pages yet, and no reports from an Asus Ascent GX10. @nathanpwhite, if you try it, ./scripts/driver-build check will tell you before anything is built.