Recommended libraries for real-time video image display with overlayed annotations

Platform: Jetson Nano
JetPack version: 4.6.1

Many thanks for prior advice regarding using OpenCV in conjunction with the Argus API for real-time video image processing. I have been able to achieve a throughput of 30 frames per second using the following multi-threaded architecture:

  • Each colored box represents an independent thread of control.
  • The threads communicate with each other using message queues.
  • The Argus Video Frame message object contains a pointer to a EGLStream::Frame object
  • The Video Image message object contains a cv::Mat object

The primary improvements I’m trying to make involve the userInterface thread, which uses OpenCV objects and methods for displaying annotated video data. The improvements I would like to achieve include:

  • Higher frame rate. The video image display frequency ranges from 10 to 12 frames per second, which gives the false impression the application can’t keep up with the native frame rate of the camera (30 fps). I’m using cv::imshow() for displaying the annotated video data.
  • Better-looking annotations. I annotate the video using cv::rectangle() to draw a rectangle around a person’s face, and cv::putText() with cvv::rectangle() to display warning and instructions-for-use messages inside of opaque rectangles (e.g., a red rectangle for a warning message). I would like to use a better-looking font than what OpenCV provides, and use static images for displaying the results of the imageProcessing thread.

From other forum topics I’ve read, the NvEglRenderer class looks promising (see also sample code /usr/src/jetson_multimedia_api/samples/13_multi_camera/main.cpp), but that class appears to be limited to low-level rendering operations (e.g., method setOverlayText() does not allow the caller to specify parameters such as the font family, size, style, or color).

Are there other classes or APIs that would work with NvEglRenderer (or a similar high-performance rendering engine) to yield the performance and aesthetic improvements I would like to achieve?

Many thanks!

In the patch, it demonstrates mapping NvBuffer to cv::Mat in RGBA, convert to the other cv::Mat in BGR and render through cv::imshow():
NVBuffer (FD) to opencv Mat - #6 by DaneLLL

If you would like to use NvEglRenderer, would need to convert the data in cv::Mat in BGR back to the cv::Mat in RGBA, call NvBufferMemSyncForDevice(), and render the NvBuffer. It is uncertain whether this has better performance than cv::imshow(). Would be great if you can give it a try and share us the result.


Thanks for your reply. Referring to the data flow diagram in this thread, I would probably modify the design to pass the EGLStream::Frame object directly to the userInterface thread to avoid copying and/or converting data structures.

In addition to a faster rendering engine, I’m looking for classes for creating nicely-formatted strings. That is, I’d rather not re-invent the wheel, so to speak. I’m definitely struggling to find faster equivalents of cv::putText() and cv::getTextSize().


This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.