Hi everyone,
I would like to understand in technical terms why batching images improves the throughput of the GPU.
Thank you very much.
Hi maycondouglasd, please see the following whitepaper which includes a discussion on batching inference:
In short, batching more data in parallel helps keep the GPU’s pipeline full and increased occupancy/utilization.