Execution time variation of a CNN inference in Jetson AGX

Hello there,
I have a question please, I’m working on some CNN benchmarking on the Nvidia Jetson AGX, I’ve measured the execution time of an inference (with a SqueezeNet CNN) several times on the same program and I noticed that the execution time vary from iteration to another. to measure the execution times I used the time() function of python. I use Tensorflow-gpu 1.14 to run the CNN inference.
I would like to know what’s the reason behind the variation in execution times and how we can tackle this problem.
Thank you in advance.


Have you maximized the device performance with following command first?

(Please noticed that these command need to be applied in order)

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

The default setting of Jetson is dynamic clock, which may leads to various execution time.
So it’s recommended to maximize the device performance before benchmarking.