Meaning of 'Message Size' in NCCL operations

Developer Tools Nsight Systems Profiling Linux Targets
1

Hi! I have been profiling my LLM training using Nsight Systems to better understand the communication patterns. When I hover over the ncclAllReduce event it shows some additional information such as Message Size. What exactly does the Message size refer to in this context?
Is it #GPUs x #elements/GPU x Bytes/element?

image
image1920×697 106 KB

Thanks for your help :)

2

@tilsche can you respond to this please.

3

Hi, this is the [number of elements per GPU (i.e. the *count argument of the API call) * size of each element in bytes]. It does not consider the number of GPUs in the communicator.

4

Thanks! That’s really helpful :))