GPU Utilization

I am using 1 Tesla K 80; and training with 25,000 images. But I am tracking the GPU utilization every second where I find this pattern where every other second the utilization is high (90%), but every other second the GPU Utilization is low (0%).

The current speeds are around 5 images per second / 250 annotations per second. It is taking 35 minutes per epoch for 25000 images.

I am wondering why is there this high and low of the GPU utilization during training. My goal is to keep the GPU utilization on high all the time so training is done more quickly.

From my experience for experiments, it is related to different training work. I can see some of jobs get high GPU utiliztion. Some are lower.
You can also try increasing the batch size.