Model inference speed suddenly slows down


I am performing semantic segmentation on a Jetson Nano board (Jetpack 32.2.3), with a small U-Net model implemented with pytorch 1.3.0.

As images are fetched from the SD card and forwarded through the model, I find that the inference speed suddenly slows down. That is, for the first 7~8 images, inference takes around 25ms each. Then, the time shoots up to around 150ms on the next image, and sustains an inference time of around 280ms then on.

Inference time is measured by wrapping the model as follows:

def inference(self, x):
    tic = time.perf_counter()
    with torch.no_grad():
        pred = self.model(x)
    toc = time.perf_counter()
    return pred, toc-tic

Hence I am pretty sure that the variation in image fetch time is not the problem.

Could you provide suggestions or guesses about the cause?



Sorry for the late update.

Another possible cause is the memory allocation.
If the buffer is not be reused, you may need to release/allocate essential memory for each frame.

Would you mind to help us checking this first?
Please help to add a sleep code after inference and monitor the system status with tegrastats.

sudo tegrastats