We have a tensorflow python application on video analytic. The program CPU usage around 90 but the GPU usage is about 20. How to balance the usage to increase GPU usage and decrease cpu usage.
I would begin by examining the input data pipeline. For example, try running the input pipeline with a trivial single-layer network. If the training speed is similar to that of your full model, then the input pipeline is a bottleneck.
How you optimize your input pipeline depends on the details of what you are doing. In general, you want to avoid loading data through numpy. Converting your model to use the tf.data preprocessing primitives often provides a good speedup. You might also want to checkout DALI (NVIDIA DALI Documentation — NVIDIA DALI 1.16.1 documentation) which can handle many common image prepossessing steps on the GPU itself.