I am performing semantic segmentation on a Jetson Nano board (Jetpack 32.2.3), with a small U-Net model implemented with pytorch 1.3.0.
As images are fetched from the SD card and forwarded through the model, I find that the inference speed suddenly slows down. That is, for the first 7~8 images, inference takes around 25ms each. Then, the time shoots up to around 150ms on the next image, and sustains an inference time of around 280ms then on.
Inference time is measured by wrapping the model as follows:
... def inference(self, x): tic = time.perf_counter() with torch.no_grad(): pred = self.model(x) toc = time.perf_counter() return pred, toc-tic ...
Hence I am pretty sure that the variation in image fetch time is not the problem.
Could you provide suggestions or guesses about the cause?