In profiles of Pytorch CNN training in Nsight Systems I can see strange gaps between every three or so training iterations.
On the above screenshot, are a few iterations of Pytorch Googlenet training. Each iteration is marked with an NVTX region. You can see the gaps are afther the 3rd, the 6th and the 9th iterations.
The only activity captured by the nsys profiler in these gaps are OS runtime “sem_timedwait”-s.
The training ran on a p3.2xlarge instance on AWS cloud in a Docker container.
OS Ubuntu 18.04.3 LTS,
NVIDIA driver 430.64,
Any ideas of what is going on during these gaps and what is the cause?