Anyone have a solution for LoRA training of recent MoE models like Qwen3.5-35B-A3B or Gemma-4-26B-A4B *and* successfully running in vLLM?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi - I’m not having much luck getting a LoRA for recent SOTA MoE models to work with vLLM.

I tried this solution: Bf16 LoRA Fine-Tuning of Qwen3.5-35B-A3B on DGX Spark — No Quantization Required but it didn’t load successfully into vLLM apparently due to a mismatch between the Unsloth LoRA format of fused expert tensors. I also tried training on just the attention layers but the loss stayed really high. If you try to use Unsloth Studio directly you run out of memory (at least for Qwen3.5-35B-A3B and Gemma-4-26B-A4B).

If anyone has worked out how to do this without going OOM, and produced a LoRA that actually affects the output of the model, and could run in vLLM, I would love to copy your approach.

thanks

2

LOOK AT MY GITHUB REPO.

4

AutoModel is a good fit for smaller models training. You can start with Automodel container. We have a ready-to-run Gemma 4 26B-A4B LoRA recipe
(Automodel/examples/vlm_finetune/gemma4/gemma4_26b_a4b_moe_peft.yaml at main · NVIDIA-NeMo/Automodel · GitHub)

torchrun --nproc-per-node=8 examples/vlm_finetune/finetune.py \
   -c examples/vlm_finetune/gemma4/gemma4_26b_a4b_moe_peft.yaml

For Qwen3.5-35B-A3B, see the VLM fine-tuning recipe
(Automodel/examples/vlm_finetune/qwen3_5_moe/qwen3_5_35b.yaml at main · NVIDIA-NeMo/Automodel · GitHub) and the MoE LoRA configuration example (Automodel/examples/llm_benchmark/qwen/qwen3.5_moe_lora.yaml at main · NVIDIA-NeMo/Automodel · GitHub). The LoRA peft: block can be applied to the fine-tuning recipe and adapted to the target dataset.

AutoModel saves PEFT checkpoints as Hugging Face-compatible adapter_config.json and adapter_model.safetensors files; see the checkpointing guide (Automodel/docs/guides/checkpointing.mdx at main · NVIDIA-NeMo/Automodel · GitHub).

The resulting adapter can be served on top of the original base model using vLLM’s LoRA serving interface
(LoRA Adapters - vLLM):

 vllm serve <base-model> \\
   --enable-lora \\
   --max-lora-rank 16 \\
   --lora-modules automodel-adapter=/path/to/checkpoint/model

Please use a recent vLLM release that supports the exact Gemma4 or Qwen3.5 model class. The AutoModel examples keep the vision tower frozen and apply LoRA to the language-side modules, which is the recommended deployment path.