I started to time some OpenCV code I ported over to the Jetson and try to decide what to optimise to get things running at my camera’s frame rate (~30FPS). The code uses color and depth images. It contains a mix of computer vision, model fitting (PCA and similar) and visualisations via imshow() calls.
I am currently at 8 to 20 FPS depending on the mix of image processing and visualisation operations being performed. The code is single threaded so things can definitely be improved.
I have noticed quite a bit of slow down using imshow to display images (10ms+) and the “lag” seems to be very variable. Are the highgui functionality hardware accelerated? I noticed that the namedWindow Windows do not support OpenGL.
Any pointers towards more technical information about the 2.4.8 version of OpenCV4Tegra available for the Jetson would be much appreciated as I can’t just dive into the code to see whether the GPU or CPU or both are being utilized.
Note that I don’t have access to VisionWorks (but would love to get access :D) so I haven’t used any functions from VW.
It’s true, we recently became aware that displaying images using OpenCV4Tegra on Jetson TK1 is not optimized at all. The reason is that we optimized OpenCV4Tegra for several years but it was only for Android, where highgui isn’t available for image display, so that part was never optimized. Now that OpenCV4Tegra has been ported to Linux, this function is missing out on optimizations. I looked into whether it is worth us (NVIDIA) optimizing the imshow() function for Linux, but the conclusion was that it would take significant work, since there are a massive number of library dependencies and image formats and pixel formats and display drivers that would need to be dealt with, that we decided it is not a worthwhile investment, since VisionWorks will provide a better solution when it becomes available next year.
So unfortunately, for now you have to either stick with the un-optimized imshow() implementation, or write your own OpenGL code to render OpenCV images efficiently for your specific image type. If you have experience with OpenGL then it should be fairly trivial, and a customer told me recently he hacked together some code to display an OpenCV image and is considering eventually cleaning it up & uploading it, but I don’t know when or if that will happen.
I managed to optimise my code through threading (4 cores FTW) and also using the GPU for some OpenCV operations. Running at ~30FPS but unsure about the latency. If it is too poor, I will look into writing a simple GLUT-based image renderer.
Sorry but NVIDIA Visionworks Toolkit is still in development and not yet released to developers. In the meantime, please use the Tegra accelerated OpenCV4Tegra library.