On-board Camera FPS OpenCV

vchinta · September 28, 2017, 12:03am

Hi,

I read a lot of threads here (built OpenCV 3.1 and changed cboot.bin) and finally got my on-board camera working. But now I’ve run into another problem - the camera (or code) is not grabbing frames at the required/specified rate.

I believe this line sets the gst pipeline fps to 120.

VideoCapture cap("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, \
                        height=(int)720, format=(string)I420, framerate=(fraction)120/1 ! \
    			nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! \
    			videoconvert ! video/x-raw, format=(string)BGR ! appsink");

And using this code, I should get a 120 fps recording of 3 seconds right?

for(int i = 0; i <= 360; i++)
{
    if(!cap.read(src))
    {
        cout << "ERROR! blank frame grabbed\n";
        return -1;
    }

    writer.write(src);	
}

But I don’t. The video is 3 seconds long because I’m specifically counting 361 frames, but the actual time elapsed during the recording is much more than 3 seconds, close to 30 seconds. Please see recorded video here - https://drive.google.com/open?id=0Bwew736KYms5WVEtOUMzQnJ1c1E

Can anyone tell me what’s happening here?
Is the TX1 just that slow at iterating through the for-loop?
Or should I be using some sort of CUDA code? Or Nvidia’s argus/libargus?
Will I face this issue even if I move to external CSI-2 cameras? (which is the plan)

Please help!
Venkat

WayneWWW · September 28, 2017, 2:14am

Please check the camera config by using v4l2-ctl and see if it is 120fps.

Raise the clk to maximum through jetson_clock.sh and try again.

After trying these, if still cannot achieve 120fps, the videoconvert in the pipeline maybe the bottleneck.

vchinta · September 28, 2017, 5:09pm

This is the output of v4l2-ctl --all I get.

$ v4l2-ctl --all
Driver Info (not using libv4l2):
	Driver name   : tegra-video
	Card type     : vi-output, ov5693 6-0036
	Bus info      : platform:54080000.vi:2
	Driver version: 4.4.38
	Capabilities  : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Priority: 2
Video input : 0 (Camera 2: no power)
Format Video Capture:
	Width/Height      : 1280/720
	Pixel Format      : 'RG10'
	Field             : None
	Bytes per Line    : 2560
	Size Image        : 1843200
	Colorspace        : sRGB
	Transfer Function : Default
	YCbCr Encoding    : Default
	Quantization      : Default
	Flags             : 

Camera Controls

                   frame_length (int)    : min=0 max=32767 step=1 default=1984 value=761 flags=slider
                    coarse_time (int)    : min=2 max=32761 step=1 default=1978 value=755 flags=slider
              coarse_time_short (int)    : min=2 max=32761 step=1 default=1978 value=755 flags=slider
                     group_hold (intmenu): min=0 max=1 default=0 value=0
                     hdr_enable (intmenu): min=0 max=1 default=0 value=0
                       otp_data (str)    : min=0 max=1024 step=2 value='93b2ec0110238540000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000057000000000000000000170000000000' flags=read-only, has-payload
                        fuse_id (str)    : min=0 max=16 step=2 value='93b2ec0110238540' flags=read-only, has-payload
                           gain (int)    : min=256 max=4096 step=1 default=256 value=4096 flags=slider
                    bypass_mode (intmenu): min=0 max=1 default=0 value=1
                override_enable (intmenu): min=0 max=1 default=0 value=1
                   height_align (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align (intmenu): min=0 max=2 default=0 value=0
               write_isp_format (bool)   : default=0 value=0

I cannot see fps on there. I tried the same command on my laptop and I can see fps details.

I tried to get the fps using

double fps = (double) cap.get(CV_CAP_PROP_FPS);

but the function just returns 0.

I also felt that it might be the write operation that is taking too long. So I moved them to separate threads using C++ and the fps improved to approx 100. Maybe if I raise the clk to max I can get 120.

But the write operation still took a long time. So should I be using a gstreamer pipeline for output as well? Is there documentation for using gstreamer as output pipeline on TX1? Even though for now I’m writing to file, the end goal is to output frames via USB.

Thanks!

vchinta · September 28, 2017, 5:47pm

If you would like to take a look at the complete code I’m using:

#include <opencv2/opencv.hpp>
#include <iostream>
#include <stdio.h>
#include <thread>

using namespace cv;
using namespace std;

Mat inFrames[150];
VideoCapture in;
VideoWriter out;
bool grabbed[150];

void frameGrabber();
void frameWriter();
	
int main(int, char**)
{
    in.open("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, \
                        height=(int)720, format=(string)I420, framerate=(fraction)120/1 ! \
    			nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! \
    			videoconvert ! video/x-raw, format=(string)BGR ! appsink");

    if(!in.isOpened())
    {
	cout << "\nERROR! Unable to open camera\n";
	return -1;
    }
	
    Mat frame;
    in >> frame;

    if (frame.empty()) 
    {
        cout << "\nERROR! Blank frame grabbed\n";
        return -1;
    }	
    
    int codec = CV_FOURCC('X', 'V', 'I', 'D');    
    
/*  This line does not work as mentioned in previous reply
 *
 *  double fps = (double) in.get(CV_CAP_PROP_FPS);
 *  cout << "Camera FPS: " << fps << endl;
 *
 *  VideoWriter will not work if fps passed is 0
 *  So I manually set fps to 120 below.
 */
    
    double fps = 120;
    string filename = "./live.avi";
    out.open(filename, codec, fps, frame.size());
    if (!out.isOpened())
    {
        cout << "\nERROR! Could not open output file location\n";
        return -1;
    }
    
    double t = (double) getTickCount(), t1, t2;

    thread fG_t(frameGrabber);
    thread fW_t(frameWriter);

    fG_t.join();

    t1 = ((double)getTickCount() - t)/getTickFrequency();
    cout << "Finished Grabbing in: " << t1 << "s\n";	

    in.release();

    fW_t.join();

    t2 = ((double)getTickCount() - t)/getTickFrequency();
    cout << "Finished Writing in: " << t2 << "s\n";
	
    return 0;
}

void frameGrabber()
{  
    for(int i = 0; i < 150; i++)
    {
    	in >> inFrames[i];
    	grabbed[i] = true;
    }
}

void frameWriter()
{	
    for(int i = 0; i < 150; i++)
    {
	while(!grabbed[i]);
	out.write(inFrames[i]);
	imshow("Live", inFrames[i]);
	waitKey(8);
    }
}

And CMakeLists.txt:

cmake_minimum_required(VERSION 2.8)
project(onBrdVid)
find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
set (CMAKE_CXX_FLAGS "-std=c++11 ${CMAKE_CXX_FLAGS}")
add_executable(onBrdVid main.cpp)
target_link_libraries(onBrdVid ${OpenCV_LIBS} -lpthread)

On my TX1, frameGrabber ends in about 1.4 s and frameWriter in about 15 s.

WayneWWW · September 29, 2017, 5:54am

There is another thread discussing an enhancement of videoconvert latency, but it is on opencv3.3.

https://devtalk.nvidia.com/default/topic/1024245/jetson-tx2/opencv-3-3-and-integrated-camera-problems-/post/5210735/#5210735

Please try it.

Honey_Patouceul · September 29, 2017, 8:04am

The issue doesn’t seem to be with the capture pipeline and videoconvert, but rather with writer encoding into xvid.

You may improve a bit with boosting your TX2 if not yet done:

sudo nvpmodel -m0
sudo /home/ubuntu/jetson_clocks.sh

You may also use tegrastats for checking the ressource usage.

Not sure how much efficient is your synchronization method. Does it give same results if you launch writer thread only after reader has filled buffers ?

vchinta · October 2, 2017, 11:10pm

Like you said, Wayne, videoconvert was the issue.

I built OpenCV 3.3 and got rid of videoconvert from the pipeline. Much faster now - about 114 fps without even raising the clocks to maximum.

Thanks a lot!

TomK · October 18, 2021, 6:58pm