Introducing DGX Spark Memory Saver: more memory headroom for CUDA workloads

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi everyone,

I’m sharing DGX Spark Memory Saver, a UVM driver patch that reduces GPU page-table memory overhead on DGX Spark systems running a 64 KiB Linux kernel.

In a three-node serving trial, it recovered 1.78-1.86 GiB of additional usable memory per node compared with the stock 4 KiB system. This is on top of the recent finding of memory available from firmware reserves of 2.1 GiB for display scanout.

On a 64 KiB kernel, NVIDIA UVM allocates an entire CPU page for each eligible 256-byte GPU leaf page table. Memory Saver places these tables in separate 4 KiB slots within shared pages, fitting sixteen tables into each page. This is analogous to the linux slab allocator.

On the profiled node, page-table backing storage fell from 3.25 GiB to 0.218 GiB - a saving of 3.03 GiB compared with the stock 64 KiB driver - and giving the ~1.8gb memory savings over the stock 4kb system.

The patch passed concurrent CUDA allocation and reuse tests, full-buffer readback, a 4.5 GiB pinned transfer, BF16 matrix multiplication, CUDA graph replay, and a number of other serving quality checks across three Sparks.

The repository provides:

  • Manual and DKMS installation workflows.

  • Secure Boot signing guidance.

  • Read-only status reporting with JSON output.

  • Removal and rollback instructions.

  • Benchmark reports and CUDA validation programs.

The tested configuration is DGX Spark / GB10, NVIDIA open driver 580.178.04, and Ubuntu kernel 7.0.0-1019-nvidia-64k. The documentation covers prerequisites, validation status, and the installation process.

Get the code and explore the results:
github.com/christopherowen/dgx-spark-memory-saver

Feedback, code review, and reproducible test results are welcome!

2

Thanks for sharing this, the numbers are really compelling: 512K context and ~2.85M KV tokens with the same r5o image.

Is there any chance this gets upstreamed into NVIDIA’s open GPU kernel modules / the UVM driver, so it ships in an official driver release?

We run three Sparks with Secure Boot enabled and no BMC, so a self-signed DKMS module means enrolling a MOK at the physical console on every node. If it landed in a driver update it would simply arrive signed through the normal DGX OS updates, which would make it much easier to adopt across the cluster.

NVIDIA team, is this something you are looking at?

3

I cannot speak for nvidia, of course.

I think they don’t need a random hacker’s help to do these things, but I’m willing if they reach out :).

The signing key enrollment is a bit of a pain, but it was necessary for dgx-spark-fan-control too.

Hopefully I left the instructions easy enough to follow as a human or agent. You could disable secure boot as an alternative but that requires console too.

I use one of the tiny glinet/jetkvm network kvm things to do it over the network.

4

Thanks a lot, that’s very helpful. A network KVM is exactly the missing piece for us, since it covers the MokManager step without being at the console. I’ll pick one up and follow your instructions node by node. Thanks also for publishing everything so clearly!

5

I’m dealing with the GPU memory fragmentation issues in which I’ll have 100 GiB of RAM free, but get the no memory failure when allocating. I imagine this could be used to reduce the impact. Are there other page sizes available and if so, have you tried them?

6

Thanks for sharing this. I’d be interested to understand what’s happening on your system. Do you have any logs or diagnostics that point to fragmentation? The exact error, allocation size, application, and how you’re measuring the 100 GiB free would be helpful.

Memory Saver reduces the memory used by GPU page tables, giving workloads some additional headroom. It could help indirectly by reducing memory pressure, but I wouldn’t want to promise it will resolve the allocation failure you’re seeing without understanding the cause.

The published results compare 4 KiB and 64 KiB kernels, with the patch running on the 64 KiB kernel. Linux on arm64 also has a 16 KiB configuration, though the project doesn’t currently have validated results for that on Spark.

Maybe you could offer some details on your kernel, driver, and firmware versions?

7

Did you try this ?

sync && sudo sh -c ‘echo 3 > /proc/sys/vm/drop_caches’

8

Has anyone tried this on an Asus Ascent GX10? Couldn’t tell at first glance if this was only for DGX Spark…

9

I have the Nvidia one, but I think there shouldn’t be a compatibility issue.

Please report back if you try