GPU memory allocated but GPU usage 0%

GPUs : 2 Nos - Tesla K80
CUDA: 10.0
Tensorflow-gpu - 1.15
keras: 2.2.4

During Training of model, I am seeing one GPU memory allocated but GPU Utilisation most of the time shows 0%, only few seconds it shows 100%, then again shows 0%. It is reflecting in epoch duration that it takes long time to training, that means one GPU is not utilised only memory allocated. Whereas other GPU is always 100% utilisation.

Please refer below screenshot. ProcessId is also shows for both GPU and memory of both GPU are allocated


What is the issue here? Why GPU memory used but GPU Util ?