I am trying to run NVIDIA sample code for a custom Faster RCNN model. When I increase batch size, the time taken for inference execution goes up almost linearly.
So, Batch Size :1, Time : 430 ms
Batch Size: 4, Time: 1723 ms
Nvidia Documents state the following :
“Often the time taken to compute results for batch size N=1 is almost identical to batch sizes up to N=16”
Needed help to figure out what am I missing ?
Thanks in advance.