How to properly load a 190GB model across 3 nodes? (GLM-5.2, PP=3, vLLM)

Accelerated Computing DGX Systems (Data Center)
1

Hi everyone,

I am trying to figure out how to correctly deploy the nvidia/GLM-5.2-NVFP4 model across my cluster.

Hardware & Setup:

  • 3x DGX nodes (connected in a ring topology / InfiniBand setup).

  • Target configuration: --tensor-parallel-size 1 and --pipeline-parallel-size 3.

The Issue: I’ve tried running this using both vLLM + Ray and native vLLM + torchrun. However, every time the engine reaches the “Starting to load model…” phase, my server completely freezes and becomes unresponsive.

I strongly get the impression that the system is trying to load the entire 190GB model into the system RAM on the head node (or simultaneously on every node) before dividing the weights. This causes a massive memory bottleneck/OOM, instantly hanging the machine.

Questions:

  1. Am I doing this completely wrong?

  2. How should I approach loading a model of this size across multiple nodes in a Pipeline Parallel setup without choking the RAM?

  3. Should I be pre-sharding the model offline first? If so, are there any specific guides or tips for vLLM?