Is 32× A100 40 GB with 1 GPU per node a reasonable distributed-training testbed?

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

We are designing a distributed-training experiment and currently have access to 32 NVIDIA A100 40 GB GPUs.

For both Jetstream2 and Google Cloud, the 1× A100 40 GB per node configurations are the lowest-cost options available to us, so this is the configuration we are considering for the main experiment.

Jetstream2 Google Cloud
Nodes 32 32
GPU per node 1× A100 40 GB 1× A100 40 GB
Total GPUs 32 32
VM type g3.xl a2-highgpu-1g
CPU per VM 32 vCPUs 12 vCPUs
RAM per VM 120 GB 85 GB
Network Physical GPU hosts have 2×100 GbE; actual VM throughput will be measured Up to 24 Gbps per VM

Our goal is to create a realistic distributed-training baseline using the resources we can afford and access at this scale.

The main concern is that using one GPU per node forces GPU-to-GPU collective communication onto the inter-node network. Many larger training systems instead use multiple GPUs per node connected with NVLink/NVSwitch.

Given these constraints, would 32 nodes with 1× A100 40 GB each still be considered a reasonable distributed-training experiment?

In particular, we would appreciate advice on:

  • Whether this represents a realistic and meaningful distributed-training configuration.
  • Whether 1 GPU per node would overemphasize networking enough to make the experiment misleading.
  • What measurements or controls we should include to properly characterize the system, such as NCCL collective bandwidth, all-reduce time, network latency/throughput, scaling efficiency, and compute-vs-communication time.
  • Any changes we could make without significantly increasing hardware cost that would make the experiment more representative.

We understand this is not an ideal high-end training cluster. We are mainly trying to determine whether it is a defensible representation of distributed training given the hardware available to us, and how to account for its limitations.

2

Yes, 32 nodes with 1× A100 per node is a defensible distributed-training testbed, but with important caveats.

Why it’s reasonable:

  • 32 A100s (1.28 PB/s peak compute) is a substantial cluster for many research workloads
  • 1 GPU per node is the standard configuration in cloud and academic settings due to cost
  • This setup realistically represents the communication-bound regime that many practitioners encounter
  • Your platforms (Jetstream2 with 2×100 GbE, GCP with 24 Gbps) offer measurable inter-node bandwidth

The key limitation:
Yes, 1 GPU per node forces all collective communication (all-reduce, all-gather) onto the inter-node network. This differs from high-end clusters with 8× GPUs per node connected by NVLink, where intra-node communication is ~600 GB/s and only the final reduction crosses the network. Your setup will be more network-bound.

Is it misleading?
Not if you characterize it properly. This configuration is actually representative of a large class of real distributed-training deployments (cloud, academic, smaller industry labs). The key is to measure and report:

  1. NCCL collective bandwidth: Run nccl-tests all-reduce benchmark to measure actual achieved bandwidth per operation
  2. All-reduce time: Profile your training loop’s collective operations (use torch.profiler or NCCL tracing)
  3. Compute-to-communication ratio: Measure the fraction of time spent in collectives vs. forward/backward passes
  4. Scaling efficiency: Report throughput (samples/sec) at 1, 4, 8, 16, 32 nodes to quantify communication overhead
  5. Network saturation: Monitor actual network utilization during training (use iperf3 or cloud monitoring)

Low-cost improvements (without major hardware changes):

  • Use gradient accumulation to amortize collective communication cost
  • Overlap communication with computation (PyTorch DDP with find_unused_parameters=False)
  • Consider ring all-reduce or other topology-aware collectives if your framework supports it
  • Profile with NCCL tracing to identify bottleneck operations
  • If possible, use Jetstream2 over GCP (2×100 GbE vs. 24 Gbps is an 8× difference in network capacity)

Bottom line:
This is a valid distributed-training experiment. It’s not a high-end cluster, but it’s realistic and defensible if you measure and report the communication characteristics. The results will be representative of the communication-bound regime, which is valuable for understanding scaling behavior in that regime.