Fine-tuning Nemotron-3-Nano-30B-A3B on ASUS Ascent GX10 / GB10 (DGX Spark): CUDA OOM at load, box freezes, 4-bit QLoRA fix

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Hi All,
After struggling with OOM and system freezes attempting to train the Nemotron 30B Nano on our Asus GX10 box we finally had a breakthrough with the assistance of Claude and Grok Build performing deep analysis and troubleshooting. Posting this in case others have a similar issue and this can perhaps help this solve the same issue.**

**TL;DR:**** Following NVIDIA’s DGX Spark “Fine-tune with NeMo” playbook ( Fine-tune with NeMo | DGX Spark ) with the **30B hybrid Mamba-2 + 128-expert MoE** `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` on **one ASUS Ascent GX10** (NVIDIA GB10 superchip), the stock BF16 recipe **OOMs while *loading* the model — not training — and can freeze the whole box.** Fix: **4-bit QLoRA with `force_hf: true` +`device_map: 0`**, source-built Mamba kernels, and a few transformers-5.5.0/Nemotron-H shims. Load then peaks ~20 GB, training ~35 GB of the 122 GB pool, and it saves a full 128-expert LoRA adapter. **Full step-by-step
write-up + copy-paste runbook are attached (.pdf/.docx).**

**Environment (confirmed on):** **ASUS Ascent GX10** — NVIDIA GB10 (Grace-Blackwell, sm_121), aarch64,~122 GB unified memory, CUDA 13, NGC PyTorch 2.10 (25.11), transformers 5.5.0, NeMo AutoModel 0.3.0. *We verified all of this specifically on the GX10; the same GB10 superchip is in other boxes (e.g. DGX Spark) so it should apply, but we’ve only confirmed the GX10.*

**Problems we hit:

  • `torch.OutOfMemoryError: … 116.15 GiB memory in use` during **model load**, in the MoE expert-merge (`moe/state_dict_mixin.py:471 torch.stack`) — the BF16 native path holds the model **and** merged experts ~2×.

- **Box freezes / SSH hangs / hard reboot needed** — the GX10 / GB10 single unified pool can’t spill a CUDA OOM to swap (swap death-spiral); **stale HBM** also shrinks run-over-run after a failed attempt.

- `mamba-ssm is required … cannot be imported` — PyPI Mamba/causal-conv1d wheels are x86-only; build from source.

- 4-bit **also** OOMs at load until you add `device_map: 0` + `low_cpu_mem_usage`; the fused Mamba kernel breaks if `out_proj` is 4-bit; `does not support Flash Attention 2.0`; sm_121 Mamba-2 Triton `illegal instruction`.

**How we solved it (high level):
4-bit QLoRA through NeMo AutoModel’s `force_hf` loader, streaming/quantizing per layer (`device_map: 0`) so load peaks ~20 GB; keep `out_proj`/`lm_head` out of 4-bit and out of LoRA;

`attn_implementation: eager`; `activation_checkpointing: false`; plus three small launcher patches for the transformers-5.5.0 + Nemotron-H quirks. The attached **runbook** is copy-paste; the attached **write-up** has every error → cause → fix verbatim.

*Tags: dgx-spark, gb10, gx10, nemotron, nemo-automodel, qlora, out-of-memory, mamba-ssm, aarch64*

MANUAL_RUNBOOK.pdf (64.3 KB)

FORUM_POST.pdf (71.1 KB)

2 Likes