Hi everyone,

I would like to understand in technical terms why batching images improves the throughput of the GPU.

Thank you very much.

Hi maycondouglasd, please see the following whitepaper which includes a discussion on batching inference:

In short, batching more data in parallel helps keep the GPU’s pipeline full and increased occupancy/utilization.