GPU memory allocated but GPU usage 0%

GPUs : 2 Nos - Tesla K80
CUDA: 10.0
Tensorflow-gpu - 1.15
keras: 2.2.4

During Training of model, I am seeing one GPU memory allocated but GPU Utilisation most of the time shows 0%, only few seconds it shows 100%, then again shows 0%. It is reflecting in epoch duration that it takes long time to training, that means one GPU is not utilised only memory allocated. Whereas other GPU is always 100% utilisation.

Please refer below screenshot. ProcessId is also shows for both GPU and memory of both GPU are allocated

image

What is the issue here? Why GPU memory used but GPU Util ?