Framework-agnostic DGX Spark setup: less power, more memory, same speed
First off, a huge “thank you” to this community and to the Chinese open-weight labs (DeepSeek, Qwen, GLM, Kimi): you’re the reason the Spark turned into a genuinely useful box, despite everything below. 🙏
There’s already a thread for capping the GPU and one for capping the CPU. So here’s the one that caps both. 🙂
I wish someone had told me all this six months ago. It would’ve saved me a pile of kWh, a few “why is it off again?” mornings, and way too many evenings staring at temperature graphs. So here it is, the short version I wanted back then. Works on any Spark, any engine, one node or ten.
1. Debloat (headless)
Boot to text mode first, so the desktop never starts (undo: graphical.target):
sudo systemctl set-default multi-user.target
Disable: bluetooth cups cups-browsed avahi-daemon ModemManager apport dgx-dashboard dgx-dashboard-admin multipathd rsyslog.
Mask: snapd packagekit nvidia-dgx-telemetry nvidia-dgx-sol udisks2.
udisks2is D-Bus activated, sodisablealone doesn’t stop it; it needsmask.- PipeWire needs
systemctl --user maskon the services and their sockets. - Firmware/OTA (over-the-air) updates re-enable some of these, so re-check afterwards.
- Don’t
apt purge --auto-removethe desktop:nvidia-system-station-coredepends onubuntu-desktop, and the cascade takes Spark platform packages (fstab, serial console, GRUB config) with it. A dry-run wanted to remove 1078 packages. Leave the desktop installed; withmulti-user.targetit never starts anyway.
Leaf packages that are safe to purge (~1–2 GB of disk)
Office, games and desktop apps a headless box never runs. Purge without --auto-remove, and dry-run first (-s):
sudo apt-get -s purge 'libreoffice*' \
aisleriot gnome-mahjongg gnome-mines gnome-sudoku \
rhythmbox totem cheese simple-scan transmission-gtk \
gnome-maps gnome-weather gnome-contacts gnome-music gnome-clocks \
nvidia-system-station-games nvidia-system-station-apps
Desktop snaps (before masking snapd):
sudo snap remove firefox thunderbird snap-store
If the dry-run lists anything nvidia-spark-*, dgxstation-* or ubuntu-desktop, stop.
2. GPU cap: 1800 MHz
Decode is memory-bound. Above ~1800 MHz you buy only power and heat. Prefill does scale with clock, so going much lower costs prefill speed; 1800 is the top of the energy sweet spot. Credit to @peter.h177, who measured the whole curve at the wall (his thread).
@peter.h177's sweep: 2× Spark, both nodes capped, wall power for both
| GPU cap | Decode tok/s | Wall power | Wh per 1M tokens |
|---|---|---|---|
| 2400 (uncapped) | 51.34 | 330 W | 1,688 |
| 2200 | 51.43 | 274 W | 1,480 |
| 1900 | 50.74 | 252 W | 1,381 |
| 1700 | 49.80 | 242 W | 1,350 |
| 1400 | 47.74 | 234 W | 1,362 |
| 800 | 34.90 | 211 W | 1,679 |
3. CPU cap: 3000 MHz on the P-cores (performance cores, Cortex-X925) only
Inference engines keep 2–3 threads at 100% on the P-cores, mostly spin-waiting: instead of sleeping until the GPU is done, they poll it in a tight loop to shave microseconds off every step. A spinning core burns full power doing nothing, and power grows much faster than clock (higher clock needs higher voltage), so the top bins are the most expensive way to wait. At 3000 MHz: same throughput, 8–12% less energy, hottest thermal zone 17–18 °C lower.
Don’t go below 2808 MHz, the E-core (efficiency core, Cortex-A725) maximum: the scheduler then moves the hot threads to the E-cores and you start losing speed.
My measurements (2026-11-03)
2× GX10, TP2 (tensor parallel over both nodes), GPU cap 1800 MHz in every run, only the X925 cap changed. Benchmark: TEB (tool-eval-bench) hardmode, 97 agentic scenarios, parallel 3, seed 42. Power and energy from the smart plugs’ own counters; temperature = max of all thermal_zone*.
TensorFold (EXL3 4-bit)
| X925 cap | TEB score | TEB wall | Energy, both nodes | Avg power head / worker | Peak temp head / worker |
|---|---|---|---|---|---|
| 3900 (stock) | 91 | 661 s | 41.9 Wh | 116 / 113 W | 84.2 / 78.5 °C |
| 3000 | 91 | 661 s | 38.5 Wh | 106 / 104 W | 67.5 / 65.1 °C |
| 2600 | 91 | 691 s | 37.6 Wh | 100 / 97 W | 62.8 / 60.7 °C |
vLLM (W4A16, MTP)
| X925 cap | TEB score | TEB wall | Energy, both nodes | Avg power head / worker | Peak temp head / worker |
|---|---|---|---|---|---|
| 3900 (stock) | 90 | 962 s | 60.6 Wh | 123 / 104 W | 89.7 / 78.0 °C |
| 3000 | 88 | 932 s | 53.2 Wh | 108 / 97 W | 71.9 / 63.7 °C |
| 2600 | 94 | 932 s | 49.9 Wh | 99 / 95 W | 64.0 / 59.9 °C |
- TensorFold: all 97 outputs byte-identical across caps (same score), so the time difference is pure speed. At 2600 the hot threads moved to the E-cores: +4.5%.
- vLLM is not deterministic here, and it’s not batching: two identical runs with one request at a time still gave identical outputs in only 5 of 48 scenarios. Texts match for a few hundred characters, then a near-tie token flips (“Condition:” vs “Conditions:”) and MTP drafting carries the difference forward. So its ±3% time and 88–94 score spread are run-to-run noise; no cap made it slower.
4. Firmware OTA2610+: +1.5 GiB
The UEFI (firmware) display reservation is now configurable: 2 GB by default, Auto = 512 MiB when headless. The freed memory is ordinary OS memory, so every framework gets it. It can be set from the OS (NVIDIA docs), applied on the next boot.
Check that Auto (512 MiB) is active:
od -An -tx1 /sys/firmware/efi/efivars/NvidiaDispFrmStatus-284e19b9-235f-4aab-ae4c-190456d1bd76
# 06 00 00 00 00 00 02 00 00 -> Auto, 0x200 = 512 MiB
5. 64K-page kernel: memory, not speed
- With 4K pages Linux spends ~2 GiB of the 128 GB on page bookkeeping. A 64K kernel cuts that to ~128 MiB.
- But NVIDIA’s UVM (Unified Virtual Memory) driver then backs every 256-byte GPU page table with a whole 64 KiB page: ~3.25 GiB instead of ~0.2 GiB. A stock 64K kernel leaves you less free memory. That’s not Linux and not your engine, it’s NVIDIA’s UVM driver: fine on GPUs with their own VRAM, wasteful on the Spark’s VRAM-less GB10 with 64K pages, a combo NVIDIA doesn’t validate. And a community patch fixes it by simply packing 16 tables per page. NVIDIA should be especially embarrassed by this one: their own unified-memory box, losing gigabytes to their own driver.
- The memory-saver patch packs those tables: net +1.8 GiB over 4K. It’s a patched
nvidia-uvmDKMS (Dynamic Kernel Module Support) module, so with Secure Boot on it needs your own enrolled key. - No speed gain: prefill and decode stay level.
6. Idle is the real waste
A Spark with a model loaded and the CX7 (ConnectX-7 NIC) link up idles at ~46 W at the wall, ~18–20 W of it the CX7 alone: ASPM (Active State Power Management) is off in its firmware, and EEE (Energy-Efficient Ethernet) doesn’t exist at 200G. Wake-on-LAN doesn’t work. The only real idle fix is poweroff plus a smart plug. A Mac Studio or a laptop idles at ~10 W.
7. Useful command-line tools
-
dgxtop:
topfor the Spark: GPU, CPU, unified memory, per process. -
sparkview: GPU monitor that understands the GB10’s unified memory (memory pressure instead of a fake “VRAM” number).
-
nvtop: classic GPU usage and per-process view.
-
btop: per-core load, so you can see which cores spin.
-
s-tui: CPU frequency, load, temperature and power on one screen, handy to see the P-core cap work.
-
ttyplot: plots any number stream, e.g. the hottest thermal zone.
ttyplot: hottest thermal zone, live
while :; do sort -n /sys/class/thermal/thermal_zone*/temp | tail -1 | awk '{print $1/1000}'; sleep 1; done \ | ttyplot -t 'max thermal zone' -u C -
plugtop: btop-style monitor for Shelly/Tasmota smart plugs, i.e. real wall power.
-
vllm-monitor: live vLLM dashboard: tok/s, KV cache, queue.
-
EngineMetrics: single-file TUI (terminal UI) for vLLM and TensorFold metrics.
-
tool-eval-bench: agentic tool-calling benchmark, the one behind the tables above.
The units
jetspark-gpu-clock-cap.service + /etc/default/gpu-clock-cap
# /etc/default/gpu-clock-cap
GPU_MAX_MHZ=1800
[Unit]
Description=GB10 fixed MHz GPU clock cap
Wants=nvidia-persistenced.service
After=nvidia-persistenced.service
StartLimitIntervalSec=0
[Service]
Type=oneshot
EnvironmentFile=/etc/default/gpu-clock-cap
ExecStart=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -lgc 0,${GPU_MAX_MHZ}
ExecStop=/usr/bin/nvidia-smi -rgc
RemainAfterExit=yes
Restart=on-failure
RestartSec=5
TimeoutStartSec=60
TimeoutStopSec=30
[Install]
WantedBy=multi-user.target
jetspark-cpu-clock-cap.service + /etc/default/cpu-clock-cap
# /etc/default/cpu-clock-cap (GB10 P-cores are cpu5-9 and cpu15-19)
CPU_X925_CORES=5-9,15-19
CPU_X925_MAX_MHZ=3000
[Unit]
Description=GB10 fixed MHz CPU clock cap (P-cores, Cortex-X925)
After=nv-cpu-governor.service
StartLimitIntervalSec=0
[Service]
Type=oneshot
EnvironmentFile=/etc/default/cpu-clock-cap
ExecStart=/usr/bin/cpupower -c ${CPU_X925_CORES} frequency-set -u ${CPU_X925_MAX_MHZ}MHz
ExecStop=/usr/bin/cpupower -c ${CPU_X925_CORES} frequency-set -u 3900MHz
RemainAfterExit=yes
Restart=on-failure
RestartSec=5
TimeoutStartSec=60
TimeoutStopSec=30
[Install]
WantedBy=multi-user.target
cpupower comes with linux-tools for your kernel; DGX OS already needs it for nv-cpu-governor.service. Verify under load with cat /sys/devices/system/cpu/cpu17/cpufreq/scaling_cur_freq, not scaling_max_freq.
Bottom line: skip the fancy fan curves and thermal-hysteresis scripts. Pareto applies, ~80% of the benefit for ~20% of the effort, and here it’s just two tiny oneshot units.
It’s a shame to cap a machine this powerful, but burning extra watts for zero extra throughput is just pointless. And it’s not just the power bill: at stock clocks the hottest zone spikes to ~90 °C, and a Spark that hits a heat spike simply switches itself off, mid-job, no logs. So here’s the million-dollar question: is it NVIDIA’s engineering culture, or greedy stakeholders who want people buying new “better” and “optimized” hardware over and over? Probably a bit of both.