LoRA fine-tuning on DGX Spark: 2.2–2.9x faster than Unsloth at the same loss on Gemma 4 and Qwen 3.5, plus 35B MoE and multi-GPU (Apache 2.0)

Hi all,

I wrote a LoRA fine-tuning engine for the Spark on top of Transformers, PEFT and TRL, and I’m sharing what I found along the way. Most of it is about how the GB10 behaves and is useful whatever you train with. The engine, Meep², is Apache 2.0, with the scripts and results behind every number. The first results look very promising: on Gemma 4 and Qwen 3.5, one epoch takes a third to a half of the time Unsloth needs with checkpointing off, at the same held-out loss.

What I found, in the order I ran into it:

1. Loading a big model can need twice its size. Transformers reads each file into memory and then copies it to the GPU, and on the Spark both are the same memory. Reading each file straight to the GPU, and dropping it from the page cache, avoids it.

2. PyTorch shows less free memory than you have. torch.cuda.mem_get_info() counts the file cache as used. MemAvailable in /proc/meminfo is the real figure.

3. Running out of memory can freeze the whole box. The GPU shares memory with everything else, so an overflow stalls the machine and the OOM killer may stop other processes. Limiting PyTorch with torch.cuda.set_per_process_memory_fraction, to MemAvailable minus a few GiB, turns it into a normal error.

4. With 128 GB you often don’t need gradient checkpointing. It saves memory by computing twice, which costs time. Better to run a few real training steps first and measure the peak than to guess.

5. Mixture-of-experts models are slow on the GB10. PyTorch computes each expert separately here. A Triton kernel that does all the experts at once, with LoRA only on the tokens each one receives, is much faster.

6. FP8 is worth it for storing frozen weights, not for the expert maths. It saves memory and time on dense models at almost the same loss; computing the expert products in FP8 was slower.

7. Padding wastes most time on this year’s models. Gemma 4 and Qwen 3.5 are harder to train without padding, so padding is the common default. Packing conversations end to end is where most of the speed below comes from.

Numbers. One epoch of LoRA on no_robots (9,100 conversations, 1,137 steps of 8), same conversations in the same order for every engine, every adapter scored by the same evaluation on 400 held-out conversations.

Model Engine Minutes Peak GiB Held-out loss
Gemma 4 E4B TRL, defaults 141.2 19.3 1.9303
Unsloth, checkpointing off 54.4 54.7 1.9326
Meep² 24.7 32.3 1.9295
Qwen 3.5 9B TRL, defaults 137.8 21.6 1.7815
Unsloth, checkpointing off 101.4 90.7 1.7811
Meep² 35.5 39.5 1.7813

One run per arm, seed 42. TRL at defaults (padded, every layer checkpointed). Unsloth 2026.10.7 with checkpointing off and padded batches, on Transformers 5.17 / TRL 0.29.1, the newest it accepts; compiler off on Qwen 3.5. Meep² and TRL on Transformers 5.20.2 / TRL 1.16.0. On Llama 3.1 8B, which needs none of point 7, Meep² and Unsloth are level: 1,633 against 1,662 training tokens/s.

The engine. pip install meepsquared (PyTorch 2.11 with CUDA first; already there in nvcr.io/nvidia/pytorch). Give it the model, the data and the method; it applies the above and prints what it decided and why. LoRA, QLoRA, FP8 weights, GRPO through TRL, several GPUs in one machine.

If you run it, I’d like to see your numbers: on other models, other data, other settings, two Sparks, or the same runs giving different results. The repository has a “Benchmark result” issue template for exactly that, and I’ll add what you measure to the published results. And I’d like to hear what others have found on the Spark.

4 Likes