The bottleneck is not memory, it’s the interconnect. ~23 GB/s between nodes vs ~600 GB/s inside the box. MoE all-to-all is very sensitive to this. Just for reference: the biggest cluster reported in the forums (8x GB10) runs Qwen3.5-397B-FP8 inference at 31-35 t/s, and MoE gains flatten past TP=4: Kimi 2.6 and Qwen 3.5 397B FP8 on 8xGB10. Also keep in mind FP8 training doesn’t exist on sm_121 (TransformerEngine has no backend, confirmed no roadmap): How to enable nvfp4.
On frameworks: Megatron-LM for MoE at this scale (expert parallelism), FSDP/DeepSpeed won’t survive the weight-gather traffic on 200GbE. Megatron/NeMo does work on GB10, with caveats:
- Megatron Bridge uses endless VRAM on GB10 (use
vlm_step, reduce num_workers) - Is Megatron training unsupported on GB10? (the low MFU report was config error, not sm_121)
- Training on 3 nodes without a switch (NCCL subnet-aware env vars are mandatory past 2 nodes)