I am profiling a Restnet benchmark in MVperf, and try to collect NVlink profiling trace data from a compute node with 4 GPUs.
The command I use as follow:
nvprof --aggregate-mode off --event-collection-mode continuous --print-gpu-trace --metrics nvlink_total_data_transmitted,nvlink_total_data_received,nvlink_transmit_throughput,nvlink_receive_throughput -f --export-profile timeline.prof --log-file log1.txt python keras_imagenet_main.py --data_dir=/global/homes/s/sd904/models/official/resnet/keras/imagenet --model_dir=/global/homes/s/sd904/models/official/resnet/keras/model --use_synthetic_data=true --train_steps=1 --num_gpus=4
The log file can show the number of data transmitted/received (see log1.txt snapshot in below). However, I cannot identify GPU Device ID, so I don’t know which GPU device sends/receives the data; the log file only indicated the device name (Tesla V100-SXM2).
How can I identify the data be transmitted/received from/to specific GPU devices by Nvlink?
Here is the snapshot in log1.txt:
==35439== Profiling result:
Start Duration Grid Size Block Size Regs* SSMem* DSMem* Device Context Stream Instantaneous Metric Name
1.43335s - - - - - - Tesla V100-SXM2 - - 576 nvlink_total_data_transmitted
1.43335s - - - - - - Tesla V100-SXM2 - - 544 nvlink_total_data_received
1.43967s - - - - - - Tesla V100-SXM2 - - 768 nvlink_total_data_transmitted
1.43967s - - - - - - Tesla V100-SXM2 - - 1024 nvlink_total_data_received
1.44394s - - - - - - Tesla V100-SXM2 - - 576 nvlink_total_data_transmitted
1.44394s - - - - - - Tesla V100-SXM2 - - 768 nvlink_total_data_received
Thanks!
Duan