Reading over the TRT 8.6 docs, I see the following statement:
What does this mean in practice? Above it says 500 images are sufficient for calibration. In theory, setting the batch size as large as possible would mean that I use 1 batch of 500 images. That doesn’t seem right to me, so what is a practical batch size to use for a sample size of 500 images?
Hi @cyrus.behr,
The advice is to use as large a batch as possible.
How many images comprise this batch will depend on the model and the resources it requires vs the resources available on your machine. In other words, if your GPU has enough resources to process 500 images in a single batch, we advise that you use 500 images.
It’s important to understand that 500 is an empirical value that may not be large enough for certain models, data sets and tasks. You might want to try using more images for calibration if the accuracy is low. You can examine the dynamic ranges TensorRT computes for each tensor when using progressively larger calibration sets and make sure they stabilize.
During calibration TensorRT collects statistics of the dynamic range of each intermediate activation tensor. To provide a good estimate of the range, the calibration dataset needs to be a good representation of the "real’ dataset (images that will be inputs during deployment).
