How to increase the speed of Opencv "capture >> frame"

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi folks, I am testing the TX2’s onboard CSI camera using Opencv3.2 with Gstreamer pipeline.
Here is part of the code:

...

src_selection = "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)I420, framerate=(fraction)60/1 ! nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink";


VideoCapture capture;
capture.open(src_selection);
CV_Assert(capture.isOpened());

Mat frame_cpu;
clock_t t1;

while(1)
{
   t1 = clock();

   capture >> frame_cpu;
   if (frame_cpu.empty())
      {
        break;
      }

    t1 = clock() - t1;
    cout << "it takes " << (((float)t1)/CLOCKS_PER_SEC)*1000 << " ms to capture a frame. The capture rate can reach " << 1/(((float)t1)/CLOCKS_PER_SEC) \
<< " FPS" << endl;

...

}

...

I insert the time stamps to calculate the “capture rate”.
It gives an average of 15ms to capture a frame to user memory.
This statistic seems to have nothing to do with the frame rate set in the Gstreamer pipeline.
I have set the frame rate to 1fps and 120fps, both give an average of 15ms.
However when I change the resolution to 1280x720 at 60fps, it only takes 1 or 2ms to capture.
Can anyone help explain this?

After capture, I implement some image processing using Opencv for Cuda and OpenGL imshow. All these are in a single thread. (I will implement multi-thread later.)
The image processing and displaying takes less than 5ms in total and this should be fast enough for now. So the bottleneck for me is this capture process. I was told not to use Opencv VideoCapture and use some other framework. I think libargus may be a good choice but I am not familiar with it.

Hope some experienced guys here can give me some advice. Thanks in advance.

2

Hi,
There ae three sensor modes for onboard ov5693

Available Sensor modes :
2592 x 1944 FR=30.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

It should pick 1280x720 120fps when you set to 1280x720 at 60 fps.

3

Hi DaneLLL, are you saying no matter how I set the resolution or frame rate, the driver will pick up 1 nearest mode from the 3 available modes?

I kind of had the same thought, but when I tried to set the frame rate to 1 FPS, I can definitely feel the very low frame rate and it can not be 30 FPS.

1920x1080 is not in the 3 modes you mentioned. However when I check the Mat size using Opencv, it shows the incoming frame is 1920x1080 when I use the same pipeline as in the posted code.

4

Hi,
For 1920x1080, the sensor is set to 2592x1458. TX1/TX2 receives 2592x1458 frames and downscales them into 1920x1080.

5

I think you are measuring 2 things: waiting for frame and reading it. If running at 60 fps, you cannot expect a new frame less than 16 ms after previous one.
Consider the following code with your measurement:

#include <iostream>

#include <opencv2/opencv.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>

int main()
{
    const char* gst =  "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)I420, framerate=(fraction)120/1 ! \
			nvvidconv flip-method=6 ! video/x-raw, format=(string)BGRx ! \
			videoconvert ! video/x-raw, format=(string)BGR ! \
			appsink";

    cv::VideoCapture cap(gst);

    if(!cap.isOpened())
    {
	std::cout<<"Failed to open camera."<<std::endl;
	return -1;
    }

    unsigned int width = cap.get(CV_CAP_PROP_FRAME_WIDTH); 
    unsigned int height = cap.get(CV_CAP_PROP_FRAME_HEIGHT); 
    unsigned int pixels = width*height;
    std::cout <<"Frame size : "<<width<<" x "<<height<<", "<<pixels<<" Pixels "<<std::endl;

    //cv::namedWindow("MyCameraPreview", CV_WINDOW_AUTOSIZE);
    cv::Mat frame_in(width, height, CV_8UC3);
    clock_t t1;
    while(1)
    {
	t1 = clock();
    	if (!cap.read(frame_in)) {
		std::cout<<"Capture read error"<<std::endl;
		break;
	}
	else {
		t1 = clock() - t1;
   		std::cout << "it takes " << (((float)t1)/CLOCKS_PER_SEC)*1000 << " ms to capture a frame. The capture rate can reach " << 1/(((float)t1)/CLOCKS_PER_SEC) \
<< " FPS" << std::endl;

   		//cv::imshow("MyCameraPreview",frame_in);
    		cv::waitKey(1000/120); // let imshow draw and wait for 8 ms for 120 fps
	}	
    }

    cap.release();

    return 0;

}

If you decrease the 8 ms wait, you will see your measurements showing increasing read times, while in fact it’s just waiting.

Boosting my TX2 with

sudo nvpmodel -m0
sudo /home/ubuntu/jetson_clocks.sh

it gives:

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected 1280x720 FrameRate = 120.000000 ...

Frame size : 1280 x 720, 921600 Pixels 
it takes 4.586 ms to capture a frame. The capture rate can reach 218.055 FPS
it takes 5.688 ms to capture a frame. The capture rate can reach 175.809 FPS
it takes 7.229 ms to capture a frame. The capture rate can reach 138.332 FPS
it takes 1.721 ms to capture a frame. The capture rate can reach 581.058 FPS
it takes 1.149 ms to capture a frame. The capture rate can reach 870.322 FPS
it takes 1.182 ms to capture a frame. The capture rate can reach 846.024 FPS
it takes 2.264 ms to capture a frame. The capture rate can reach 441.696 FPS
it takes 2.165 ms to capture a frame. The capture rate can reach 461.894 FPS
it takes 1.166 ms to capture a frame. The capture rate can reach 857.633 FPS
it takes 1 ms to capture a frame. The capture rate can reach 1000 FPS
it takes 2.465 ms to capture a frame. The capture rate can reach 405.68 FPS
it takes 2.104 ms to capture a frame. The capture rate can reach 475.285 FPS
it takes 1.485 ms to capture a frame. The capture rate can reach 673.401 FPS
it takes 0.909 ms to capture a frame. The capture rate can reach 1100.11 FPS
it takes 2.123 ms to capture a frame. The capture rate can reach 471.032 FPS
it takes 1.528 ms to capture a frame. The capture rate can reach 654.45 FPS
it takes 0.699 ms to capture a frame. The capture rate can reach 1430.62 FPS
it takes 0.738 ms to capture a frame. The capture rate can reach 1355.01 FPS
it takes 1.088 ms to capture a frame. The capture rate can reach 919.118 FPS
it takes 2.554 ms to capture a frame. The capture rate can reach 391.543 FPS
it takes 1.764 ms to capture a frame. The capture rate can reach 566.893 FPS
it takes 1.446 ms to capture a frame. The capture rate can reach 691.563 FPS
it takes 1.058 ms to capture a frame. The capture rate can reach 945.18 FPS
it takes 2.449 ms to capture a frame. The capture rate can reach 408.33 FPS
it takes 2.04 ms to capture a frame. The capture rate can reach 490.196 FPS
it takes 1.849 ms to capture a frame. The capture rate can reach 540.833 FPS
it takes 0.802 ms to capture a frame. The capture rate can reach 1246.88 FPS
it takes 2.245 ms to capture a frame. The capture rate can reach 445.434 FPS

For your use case, this pipeline

const char* gst =  "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)2592, height=(int)1458, format=(string)I420, framerate=(fraction)60/1 ! \
	            nvvidconv flip-method=6 ! video/x-raw, width=(int)1920, height=(int)1080, format=(string)BGRx ! \
	            videoconvert ! video/x-raw, format=(string)BGR ! \
	            appsink";

with a timeout of 16ms gives:

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected 2592x1458 FrameRate = 60.000000 ...

Frame size : 1920 x 1080, 2073600 Pixels 
it takes 4.065 ms to capture a frame. The capture rate can reach 246.002 FPS
it takes 15.032 ms to capture a frame. The capture rate can reach 66.5247 FPS
it takes 14.558 ms to capture a frame. The capture rate can reach 68.6908 FPS
it takes 11.892 ms to capture a frame. The capture rate can reach 84.0901 FPS
it takes 12.732 ms to capture a frame. The capture rate can reach 78.5423 FPS
it takes 12.526 ms to capture a frame. The capture rate can reach 79.8339 FPS
it takes 12.699 ms to capture a frame. The capture rate can reach 78.7464 FPS
it takes 12.571 ms to capture a frame. The capture rate can reach 79.5482 FPS
it takes 12.51 ms to capture a frame. The capture rate can reach 79.9361 FPS
it takes 12.235 ms to capture a frame. The capture rate can reach 81.7327 FPS
it takes 12.508 ms to capture a frame. The capture rate can reach 79.9488 FPS
it takes 12.376 ms to capture a frame. The capture rate can reach 80.8016 FPS
it takes 12.233 ms to capture a frame. The capture rate can reach 81.7461 FPS
it takes 12.321 ms to capture a frame. The capture rate can reach 81.1622 FPS
it takes 12.113 ms to capture a frame. The capture rate can reach 82.5559 FPS
it takes 12.101 ms to capture a frame. The capture rate can reach 82.6378 FPS
it takes 12.176 ms to capture a frame. The capture rate can reach 82.1288 FPS
it takes 12.119 ms to capture a frame. The capture rate can reach 82.5151 FPS
it takes 12.674 ms to capture a frame. The capture rate can reach 78.9017 FPS
it takes 12.561 ms to capture a frame. The capture rate can reach 79.6115 FPS
it takes 12.626 ms to capture a frame. The capture rate can reach 79.2017 FPS
it takes 12.661 ms to capture a frame. The capture rate can reach 78.9827 FPS
it takes 12.796 ms to capture a frame. The capture rate can reach 78.1494 FPS
it takes 12.994 ms to capture a frame. The capture rate can reach 76.9586 FPS
it takes 12.702 ms to capture a frame. The capture rate can reach 78.7278 FPS
it takes 12.791 ms to capture a frame. The capture rate can reach 78.18 FPS
it takes 12.421 ms to capture a frame. The capture rate can reach 80.5088 FPS
it takes 12.451 ms to capture a frame. The capture rate can reach 80.3148 FPS
it takes 12.337 ms to capture a frame. The capture rate can reach 81.057 FPS
it takes 12.054 ms to capture a frame. The capture rate can reach 82.96 FPS
it takes 12.014 ms to capture a frame. The capture rate can reach 83.2362 FPS
it takes 12.063 ms to capture a frame. The capture rate can reach 82.8981 FPS
it takes 12.085 ms to capture a frame. The capture rate can reach 82.7472 FPS
it takes 12.082 ms to capture a frame. The capture rate can reach 82.7678 FPS
it takes 12.069 ms to capture a frame. The capture rate can reach 82.8569 FPS
it takes 12.052 ms to capture a frame. The capture rate can reach 82.9738 FPS
it takes 12.063 ms to capture a frame. The capture rate can reach 82.8981 FPS

This is quite confirming your findings, but I’m unsure about what these results mean as the timeout seems to have no effect with this mode.

6

The question may be how is made a framerate of 60 fps with a native mode of 30 fps.
This pipeline

const char* gst =  "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)2592, height=(int)1458, format=(string)I420, framerate=(fraction)30/1 ! \
		    nvvidconv flip-method=6 ! video/x-raw, width=(int)1920, height=(int)1080, format=(string)BGRx ! \
		    videoconvert ! video/x-raw, format=(string)BGR ! \
		    appsink";

in camera native mode of 30 fps, with a 33ms timeout in the loop, gives

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected 2592x1458 FrameRate = 30.000000 ...

Frame size : 1920 x 1080, 2073600 Pixels 
it takes 7.944 ms to capture a frame. The capture rate can reach 125.881 FPS
it takes 11.985 ms to capture a frame. The capture rate can reach 83.4376 FPS
it takes 14.432 ms to capture a frame. The capture rate can reach 69.2905 FPS
it takes 1.618 ms to capture a frame. The capture rate can reach 618.047 FPS
it takes 1.65 ms to capture a frame. The capture rate can reach 606.061 FPS
it takes 1.614 ms to capture a frame. The capture rate can reach 619.579 FPS
it takes 1.606 ms to capture a frame. The capture rate can reach 622.665 FPS
it takes 1.662 ms to capture a frame. The capture rate can reach 601.685 FPS
it takes 1.609 ms to capture a frame. The capture rate can reach 621.504 FPS
it takes 1.63 ms to capture a frame. The capture rate can reach 613.497 FPS
it takes 1.655 ms to capture a frame. The capture rate can reach 604.23 FPS
it takes 1.684 ms to capture a frame. The capture rate can reach 593.824 FPS
it takes 1.619 ms to capture a frame. The capture rate can reach 617.665 FPS
it takes 1.647 ms to capture a frame. The capture rate can reach 607.165 FPS
it takes 2.013 ms to capture a frame. The capture rate can reach 496.771 FPS
it takes 2.368 ms to capture a frame. The capture rate can reach 422.297 FPS
it takes 4.945 ms to capture a frame. The capture rate can reach 202.224 FPS
it takes 3.804 ms to capture a frame. The capture rate can reach 262.881 FPS
it takes 5.039 ms to capture a frame. The capture rate can reach 198.452 FPS
it takes 4.038 ms to capture a frame. The capture rate can reach 247.647 FPS
7

Hello,

I would like to use OpenCV with the built in camera for the TX2, but am receiving errors when running my program. I borrowed some code from above, I hope that is alright.

#include <opencv2/objdetect/objdetect.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include<opencv2/highgui/highgui.hpp>
#include <opencv2/videoio/videoio.hpp>
#include <opencv2/core/core.hpp>
#include <stdlib.h>
#include <iostream>

using namespace cv;
using namespace std;

int main()
{

const char* gst =  "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)I420, framerate=(fraction)120/1 ! \
			nvvidconv flip-method=6 ! video/x-raw, format=(string)BGRx ! \
			videoconvert ! video/x-raw, format=(string)BGR ! \
			appsink";

cv::VideoCapture cap(gst);
if(!cap.isOpened()) { // check if we succeeded
cout << "Failed to open camera " << endl;
return -1;
}
for(;;)
{
Mat frame;
cap >> frame; // get a new frame from camera
imshow("original", frame);
waitKey(1);
}
// the camera will be deinitialized automatically in VideoCapture destructor
cap.release();

return 0;

}

compiled with: g++ -std=c++11 face_test_cpu.cpp -I/usr/include -lopencv_core -lopencv_highgui -lopencv_imgproc -lopencv_objdetect -lopencv_videoio -lopencv_video -o face_test

Output:
Failed to open camera

8

Hi Alex,
That’s another topic, but for using a gstreamer pipeline in opencv, you have to use a gstreamer enabled opencv, which opencv4tegra is not. You have to build and install your own version.