nvpmodel 0 (maxn) is slower than 2 (15W)

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

After upgrade my jetpack to 4.1.1, the performance is very strange.

nvidia@jetson-0423318029348:~/aibox/models$ sudo nvpmodel -q
NV Power Mode: MODE_15W
2

/usr/src/tensorrt/bin/trtexec --onnx=resnet50_y540_x960.onnx --fp16 --useSpinWait --allowGPUFallback

Get the result like
Average over 10 runs is 30.9526 ms (host walltime is 31.0191 ms, 99% percentile time is 48.1313).
Average over 10 runs is 28.4402 ms (host walltime is 28.4723 ms, 99% percentile time is 28.4887).
Average over 10 runs is 29.1165 ms (host walltime is 29.1563 ms, 99% percentile time is 34.8661).
Average over 10 runs is 29.1349 ms (host walltime is 29.1717 ms, 99% percentile time is 35.1668).
Average over 10 runs is 28.4567 ms (host walltime is 28.49 ms, 99% percentile time is 28.4929).
Average over 10 runs is 28.4391 ms (host walltime is 28.4709 ms, 99% percentile time is 28.5036).
Average over 10 runs is 29.0794 ms (host walltime is 29.1119 ms, 99% percentile time is 33.9968).
Average over 10 runs is 28.982 ms (host walltime is 29.1315 ms, 99% percentile time is 33.2626).
Average over 10 runs is 28.4499 ms (host walltime is 28.4874 ms, 99% percentile time is 28.5548).
Average over 10 runs is 28.4519 ms (host walltime is 28.4846 ms, 99% percentile time is 28.4777).

Switch to 30w maxn model

nvidia@jetson-0423318029348:~/aibox/models$ sudo nvpmodel -q
NV Power Mode: MAXN
0

The result is all over the map

Average over 10 runs is 45.3409 ms (host walltime is 45.3926 ms, 99% percentile time is 56.7173).
Average over 10 runs is 44.6976 ms (host walltime is 45.1909 ms, 99% percentile time is 49.4582).
Average over 10 runs is 45.2987 ms (host walltime is 45.3244 ms, 99% percentile time is 56.3724).
Average over 10 runs is 45.1983 ms (host walltime is 45.2301 ms, 99% percentile time is 55.2765).
Average over 10 runs is 31.9611 ms (host walltime is 31.9845 ms, 99% percentile time is 44.1454).
Average over 10 runs is 21.0879 ms (host walltime is 21.1093 ms, 99% percentile time is 21.3068).
Average over 10 runs is 15.14 ms (host walltime is 15.159 ms, 99% percentile time is 15.1698).
Average over 10 runs is 15.6232 ms (host walltime is 15.6428 ms, 99% percentile time is 19.8605).
Average over 10 runs is 15.1527 ms (host walltime is 15.1705 ms, 99% percentile time is 15.1916).
Average over 10 runs is 15.5959 ms (host walltime is 15.615 ms, 99% percentile time is 19.5466).

Whey maxn is so strange and performance is not stable?

2

Since the execution time monotonically decreases during MAX-N run, it would appear that the clocks are spinning up during this time. MAX-N increases the frequency governor limits to their maximums, but DVFS scaling is still enabled. To lock the clocks at their maximum rates, run sudo ~/jetson_clocks.sh

3

@dusty-nv

That does the magic! Thanks!