OpenCV VideoCapture capture one frame take more than 600ms in Jetson Nano

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hi
This is my test code:
#include
#include <opencv2/opencv.hpp>

int main() {
    // open the first webcam plugged in the computer
    cv::VideoCapture camera(0);
    if (!camera.isOpened()) {
        std::cerr << "ERROR: Could not open camera" << std::endl;
        return 1;
    }

    // create a window to display the images from the webcam
    cv::namedWindow("Webcam");

    // this will contain the image from the webcam
    cv::Mat frame;

    // display the frame until you press a key
    while (1) {
        // capture the next frame from the webcam
        int64 t0 = cv::getTickCount();
        camera.read(frame);
        int64 t1 = cv::getTickCount();
        int64 pro1_t = (t1 - t0) * 1000 / (int64)cv::getTickFrequency();
        qDebug() << "capture time " << pro1_t << "ms";
        if (!frame.empty())
        {
            // show the image on the window
            cv::imshow("Webcam", frame);
            // wait (10ms) for a key to be pressed
            if (cv::waitKey(10) >= 0)
                break;
        }
    }
    return 0;
}

This is the output:
ion@ion-desktop:~/build-test_camera-Desktop-Debug$ ./test_camera **
** [ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
** capture time 9 ms**
** capture time 473 ms**
** capture time 617 ms**
** capture time 667 ms**

My camera is: e-CAM50. frame size: 2592 x 1944. Frame rate: 28 fps.
Jetson Nano installed Jetpack 4.3 from SDKManger.
Qt installed on Jetson Nano to build the program.

My Questions:

  1. Why a warning message of “[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1”?
  2. Why capture time from 9ms (1st frame) increased to more than 600ms (3rd frame)?

Thanks.

2

When changed the line below:
cv::VideoCapture camera(0);
to use gestreamer in OpenCV as below:

cv::VideoCapture camera("v4l2src device=/dev/video0 \
                            ! video/x-raw,width=2592,height=1944,format=UYVY \
                            ! videoconvert \
                            ! video/x-raw,format=BGR \
                            ! appsink", CAP_GSTREAMER);

The capture time reduced a bit, but it is far greater than it should be 35ms.
This is output:

[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
capture time 8 ms
capture time 419 ms
capture time 549 ms
capture time 561 ms
capture time 586 ms
capture time 579 ms
capture time 573 ms

The warning message is still there.
What’s wrong with it and how to fix it?
Thanks

4

Hi,
Please execute sudo jetson_clocks and see if it helps. OpenCV is CPU-based frameworks and significant loading may cap the performance.

5

execute sudo jetson_clocks NOT help.

6

About the warning, it is normal. A live stream has no duration so current position cannot be computed. This is harmelss.

Note that you are using quite high resolution and this might be much for Nano at high framerate. I don’t see any framerate specified in your code, you may specify one provided by your camera in 2592x1944 resolution.

I’d guess the main bottleneck is cv::imshow. It is CPU only and not very efficient on Jetson. Try commenting it and see if timing imporves. If yes, you would try to display with a cv::videoWriter using a pipeline to a videosink such as here.

The other possible bottleneck is videoconvert that runs on CPU. One possible optimization could be to do part of the conversion with HW, so that videoconvert has just to remove the extra 4th byte :

v4l2src device=/dev/video0 ! video/x-raw,width=2592,height=1944,format=UYVY ! nvvidconv ! video/x-raw(memory:NVMM), format=UYVY ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink

Note that there are 2 nvvidconv because it expects at least one of its input or output to be in NVMM memory. The first one copies into NVMM memory and second one converts into BGRx into standard memory.

1 Like
7

Hi Honey_Patouceul
Thank you for comment and suggestion.
The high resolution is required in our application. The camera can only support frame rate at 28fps when frame size is at 2592x1944.
The bottleneck is NOT the cv::imshow as the time I measured is just read frame.

Your gstreamer code is significantly improved the video capture speed. Now it takes about 40 to 50ms per frame.
As I am new to gstreamer and Jetson nano, would you please give me your gstreamer code suggestion to get Y plan of the image to see if it can speed up further?
Thanks

8

You may convert to GRAY8 with nvvidconv, this should be accepted by opencv videoio and you would read one channel Mat in opencv:

v4l2src device=/dev/video0 ! video/x-raw,width=2592,height=1944,format=UYVY ! nvvidconv ! video/x-raw(memory:NVMM), format=UYVY' ! nvvidconv ! video/x-raw, format=GRAY8 ! appsink
1 Like
9

Thanks