How to Monitor and Optimize GPU and CPU Resource Usage?

Gaming and Visualization Technologies General Topics and Other SDKs GPU - Hardware
1

Hello everyone,

I have been training and fine-tuning large language models using PyTorch recently. During this process, I am looking to better understand and monitor the inter-GPU communication, the transfer of parameters and operators, as well as the usage of GPU memory and CPU memory.

Specifically, I am facing the following challenges:

  1. How can I monitor the communication and data transfer in a multi-GPU environment?
  2. What are the best practices to accurately measure and optimize the memory usage of the GPU in my model?
  3. Are there tools or techniques that can help me better monitor the usage of CPU memory during training?

I have tried using nvidia-smi to observe GPU usage, but I find the information provided somewhat limited. I am looking for more detailed analysis, especially in a distributed training context.