Hello,
I noticed the performance of OpenVX nodes on Jetson TX1 is very unstable. It can be seen on demos and examples from VisionWorks where the FPS of the algorithms is rapidly changing. Could you explain me this behavior, please?
Hi JanTV1,
Would you please provide more inputs for the performance issue you saw?
Which demo and example from VisionWorks where the FPS of the algorithms is rapidly changing?
Thanks
Hi Kayccc,
I see this behaviour in all OpenVX samples and demos. For example here is output from opencv_npp_interop:
Graph Time : 2.49734 ms
Gaussian Blur 1 Time : 0.135579 ms
Gaussian Blur 2 Time : 0.12942 ms
Alpha Comp Time : 0.11125 ms
Display Time : 8.51667 ms
Graph Time : 8.0763 ms
Gaussian Blur 1 Time : 0.199157 ms
Gaussian Blur 2 Time : 0.190947 ms
Alpha Comp Time : 0.086615 ms
Display Time : 20.1079 ms
Graph Time : 4.82594 ms
Gaussian Blur 1 Time : 0.198842 ms
Gaussian Blur 2 Time : 0.192842 ms
Alpha Comp Time : 0.12974 ms
Display Time : 21.156 ms
Graph Time : 1.49219 ms
Gaussian Blur 1 Time : 0.064736 ms
Gaussian Blur 2 Time : 0.997262 ms
Alpha Comp Time : 0.091718 ms
Display Time : 3.64547 ms
Graph Time : 1.2226 ms
Gaussian Blur 1 Time : 0.063525 ms
Gaussian Blur 2 Time : 0.05821 ms
Alpha Comp Time : 0.175885 ms
Display Time : 5.24479 ms
Graph Time : 2.97125 ms
Gaussian Blur 1 Time : 0.136316 ms
Gaussian Blur 2 Time : 0.129209 ms
Alpha Comp Time : 0.087813 ms
Display Time : 7.44578 ms
Graph Time : 2.19911 ms
Gaussian Blur 1 Time : 0.13563 ms
Gaussian Blur 2 Time : 0.129262 ms
Alpha Comp Time : 0.098177 ms
Display Time : 4.3937 ms
Graph Time : 2.19833 ms
Gaussian Blur 1 Time : 0.136684 ms
Gaussian Blur 2 Time : 0.12942 ms
Alpha Comp Time : 0.088021 ms
Display Time : 4.23333 ms
Graph Time : 2.8175 ms
Gaussian Blur 1 Time : 0.197842 ms
Gaussian Blur 2 Time : 0.191211 ms
Alpha Comp Time : 0.216458 ms
Display Time : 7.92557 ms
Graph Time : 3.28156 ms
Gaussian Blur 1 Time : 0.198736 ms
Gaussian Blur 2 Time : 0.191368 ms
Alpha Comp Time : 0.462656 ms
Display Time : 11.5259 ms
Graph Time : 3.04745 ms
Gaussian Blur 1 Time : 0.198263 ms
Gaussian Blur 2 Time : 0.191525 ms
Alpha Comp Time : 0.165469 ms
Display Time : 16.0173 ms
Graph Time : 4.51974 ms
Gaussian Blur 1 Time : 0.136526 ms
Gaussian Blur 2 Time : 0.131474 ms
Alpha Comp Time : 0.095313 ms
Display Time : 8.93724 ms
Graph Time : 6.78906 ms
Gaussian Blur 1 Time : 0.198052 ms
Gaussian Blur 2 Time : 0.192053 ms
Alpha Comp Time : 0.725364 ms
Display Time : 15.2932 ms
Graph Time : 6.24589 ms
Gaussian Blur 1 Time : 0.198736 ms
Gaussian Blur 2 Time : 0.191367 ms
Alpha Comp Time : 0.124479 ms
Display Time : 14.2217 ms
Graph Time : 1.41422 ms
Gaussian Blur 1 Time : 0.065367 ms
Gaussian Blur 2 Time : 0.058157 ms
Alpha Comp Time : 0.116041 ms
Display Time : 3.39958 ms
Graph Time : 2.31536 ms
Gaussian Blur 1 Time : 0.062263 ms
Gaussian Blur 2 Time : 0.05821 ms
Alpha Comp Time : 0.10276 ms
Display Time : 3.96115 ms
Graph Time : 3.62276 ms
Gaussian Blur 1 Time : 0.134893 ms
Gaussian Blur 2 Time : 0.12863 ms
Alpha Comp Time : 0.099271 ms
Display Time : 9.71896 ms
Graph Time : 6.65156 ms
Gaussian Blur 1 Time : 0.134736 ms
Gaussian Blur 2 Time : 0.129052 ms
Alpha Comp Time : 0.105573 ms
Display Time : 14.8395 ms
Graph Time : 0.660156 ms
Gaussian Blur 1 Time : 0.056947 ms
Gaussian Blur 2 Time : 0.052737 ms
Alpha Comp Time : 0.084167 ms
Display Time : 3.18318 ms
Graph Time : 0.733645 ms
Gaussian Blur 1 Time : 0.102737 ms
Gaussian Blur 2 Time : 0.097894 ms
Alpha Comp Time : 0.08125 ms
Display Time : 4.24104 ms
Graph Time : 1.77859 ms
Gaussian Blur 1 Time : 0.102683 ms
Gaussian Blur 2 Time : 0.097999 ms
Alpha Comp Time : 0.122448 ms
Display Time : 4.15891 ms
Graph Time : 2.16865 ms
Gaussian Blur 1 Time : 0.134632 ms
Gaussian Blur 2 Time : 0.132525 ms
Alpha Comp Time : 0.097083 ms
Display Time : 4.21479 ms
Graph Time : 2.06766 ms
Gaussian Blur 1 Time : 0.134632 ms
Gaussian Blur 2 Time : 0.128579 ms
Alpha Comp Time : 0.211927 ms
Display Time : 5.39031 ms
Graph Time : 2.19979 ms
Gaussian Blur 1 Time : 0.135473 ms
Gaussian Blur 2 Time : 0.132473 ms
Alpha Comp Time : 0.100834 ms
Display Time : 4.42786 ms
Graph Time : 0.99375 ms
Gaussian Blur 1 Time : 0.13579 ms
Gaussian Blur 2 Time : 0.128737 ms
Alpha Comp Time : 0.083021 ms
Display Time : 5.0349 ms
You can see that Graph times and nodes times are changing.
It’s the same for the other examples. I also tried write my own programs with OpenVX and I found out that the performance of the OpenVX nodes (Gaussian, Sobel, Optical flow, Magnitude etc.) is changing even if the input parameters are the same.
This issue remains even if the performance times are bigger. This is output from my code with Optical Flow OpenVX node (Input arrays have still the same size - I can provide my source code if needed):
Graph time : 8.903854 ms
Create pyramid node time : 1.732525 ms
Optical flow node time : 6.619206 ms
Graph time : 8.489531 ms
Create pyramid node time : 0.788263 ms
Optical flow node time : 7.297048 ms
Graph time : 30.400417 ms
Create pyramid node time : 1.285051 ms
Optical flow node time : 28.865617 ms
Graph time : 20.589323 ms
Create pyramid node time : 1.370684 ms
Optical flow node time : 18.076307 ms
Graph time : 52.204948 ms
Create pyramid node time : 3.082155 ms
Optical flow node time : 48.891712 ms
Graph time : 7.580105 ms
Create pyramid node time : 0.816736 ms
Optical flow node time : 5.912312 ms
Graph time : 18.277135 ms
Create pyramid node time : 1.269104 ms
Optical flow node time : 15.583413 ms
Graph time : 49.841302 ms
Create pyramid node time : 1.713947 ms
Optical flow node time : 47.287239 ms
Graph time : 4.255677 ms
Create pyramid node time : 0.571421 ms
Optical flow node time : 2.885313 ms
Graph time : 3.693438 ms
Create pyramid node time : 0.623682 ms
Optical flow node time : 2.884524 ms
Graph time : 9.755000 ms
Create pyramid node time : 0.781157 ms
Optical flow node time : 8.521364 ms
Graph time : 11.145937 ms
Create pyramid node time : 0.973315 ms
Optical flow node time : 9.720257 ms
Graph time : 20.982084 ms
Create pyramid node time : 1.246525 ms
Optical flow node time : 19.294464 ms
Graph time : 56.840417 ms
Create pyramid node time : 2.091262 ms
Optical flow node time : 54.576972 ms
Thank you!
Hi JanTV1,
(1) Power management software on the platform adjusts the clocks based on the load; and this is the major cause of the variation. Setting system clocks to max brings somewhat more deterministic behavior. One can check GPU profile log to see the actual execution time of these operations.
(2) Also note that TX1 system resources are shared across various system tasks in addition to the algorithms we run on VisionWorks. Background tasks might lead to delays in vision algorithm execution.
Thanks
Hi kayccc,
I set the fixed GPU clock and shut down the Ubuntu GUI and the performance is more stable now.
Thank you very much!