Hey guys, I was playing around with larger models on the DGX Spark with NVFP4 quantization, and wanted to share a result from it, a stack that works for LoRA fine-tuning and serving the Nemotron-3-Super-120B MoE model on a single spark.
The main problem is that NVIDIA’s Nemotron-3 NVFP4 checkpoints are efficient to run, but they are not directly compatible with standard LoRA tooling. The weights are stored as packed NVFP4 values with FP8 block scales and a per-tensor FP32 scale, and the model also mixes in FP8 Mamba/shared-expert layers. Off-the-shelf PEFT paths do not really handle that cleanly.
So this project takes a different route:
- keep the NVFP4 base weights frozen in their compressed form
- dequantize on the fly to bf16 inside a custom autograd path
- train only the LoRA adapters
- merge the adapter back into the NVFP4 base
- serve the merged checkpoint with vLLM on DGX Spark
The target hardware is a single DGX Spark (GB10, 128 GB unified memory, CUDA 13).
What the repo includes
- NVFP4-aware LoRA layers and loader code
- training scripts for Nano-30B and Super-120B
- merge + re-quantization pipeline
- validation scripts for merge correctness
- serving recipes for vLLM
- reproducibility notes, troubleshooting, and benchmark artifacts
Headline results
Training on a single DGX Spark
| Model | Base format | LoRA targets | Peak memory | Step time | Final loss |
|---|---|---|---|---|---|
| Nemotron-3 Super 120B | NVFP4 | up_proj, down_proj |
93.2 GB | 135.6 s | 0.81 |
| Nemotron-3 Nano 30B | NVFP4 | up_proj, down_proj |
36.1 GB | 43.8 s | 1.00 |
| Nemotron-3 Nano 30B | BF16 | up_proj, down_proj |
67.5 GB sampled | 4.2 s | 0.98 |
A useful result here is that the Nano NVFP4 vs BF16 ablation showed very similar training loss, while NVFP4 used much less memory. The tradeoff is speed: NVFP4 training is slower, but it makes models fit that otherwise would not fit on this box at all.
Serving results
| Serving path | Throughput on DGX Spark |
|---|---|
| Super-120B base, vLLM CUTLASS | ~11-14 tok/s |
| Super-120B merged FT, vLLM CUTLASS | ~12-14 tok/s |
| Super-120B merged FT, concurrency 3 | up to ~40 tok/s aggregate |
| Nano-30B FT, concurrency 8 | up to ~339 tok/s aggregate |
One of the practical conclusions was that for Super-120B, the most reliable path was merge-then-serve rather than dynamic LoRA attachment at request time.
Why this was interesting
The part I found most interesting was that this was less a “fine-tuning script” problem and more a systems problem:
- quantized storage format compatibility
- on-the-fly dequantization without keeping a persistent bf16 shadow
- unified-memory behavior on GB10
- kernel/toolchain quirks on Blackwell-class local hardware
- getting a serving path that preserves FT behavior without giving up native NVFP4 speed
If anyone is experimenting with NVFP4 on the DGX Spark, or local fine-tuning of large MoE models, it would be great to connect as I’m still finding my feet with this setup!