Enquires about running jobs using multiple GPUs

[ngeneva]

Hi @NEA_taywb

Thanks for your interest in Modulus, and good question. You are correct. Batch sizes in Modulus are defined such that this is the batch size per process. In physics driven problems we generate training input “data” on initialization, which is simply batch_size * batch_per_epoch. So if you don’t half this number going from one to two GPUs, you’re actually increasing your dataset by a factor of two.

This is weak scaling and maintaining the same performance (wall clock) is the ideal. Theres some additional information in this thread here that can hopefully clear things up.

Testing performance on multiple GPUs

Hi @bvss891 In Modulus the batch size defined in the config are by default the local (per GPU) batch size. What this means is that if you keep the batch size the same between 1 and 2 GPUs is that you’ve gone from a global batch size of 10000 → 20000. Thus this would be a weak scaling test and the best you could ask for is the same performance speed between 1->2 GPUs. This of course won’t happen because to the communication between the GPUs adding some overhead. The exact overhead depends on ho…