MIPI CS Camera and OpenCV

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX1
1

Hello,
Togheter with my friend, we work on a project that includes Jetson Tx1 with OpenCV(3.1).
We have leopard camera LI-IMX274-MIPI-CS. We have drivers and get image in a similar way like on onboard camera(given my NVidia 5MP MIPI Camera) by using gst-launch.

We try to get image like in this topic: https://devtalk.nvidia.com/default/topic/943129/jetson-tx1/highgui-error-v4l-v4l2-while-opening-camera-with-opencv4tegra-l4t-r24/post/4921383/#4921383

Code:

VideoCapture cap("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)24/1 ! nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"); //open the default camera

if(!cap.isOpened()) { // check if we succeeded
cerr << "Fail to open camera " << endl;
return -1;
}
for(;;)
{
Mat frame;
cap >> frame; // get a new frame from camera
imshow("original", frame);
waitKey(1);
}
// the camera will be deinitialized automatically in VideoCapture destructor
cap.release();

But we get error: “Fail to open camera”.

Maybe someone also use Leopard TX1 CS MIPI camera with OpenCV and know how to do it.

2

Did you try the gst-launch command working first?

gst-launch-1.0 nvcamerasrc fpsRange=“30.0 30.0” ! ‘video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1’ ! nvtee ! nvvidconv flip-method=2 ! ‘video/x-raw(memory:NVMM), format=(string)I420’ ! nvoverlaysink -e

3

Yes this command working.

The issue Is That we dont know how to convert it to stream that will work in OpenCV.

4

Your command can’t launch by the gst-launch-1.0, you need use gst-launch-1.0 to make sure your command is correct first. What I mean is below command have syntax error. What’s your purpose?

gst-launch-1.0 nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)24/1 ! nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink

5

Hi Shane,

Is your gst-launch string V4L2 compatible? I believe OpenCV expects V4L2.

I think in OpenCV, the mat container/format is the universal input tool…perhaps there is a different nvcamerasrc gst pipeline that can deliver objects in mat format to OpenCV?

Barrett

6

Problem solved.
That was problem with OpenCV and Gstreamer connection. Thanks for help :)

7

After launch camera in OpenCV I see That is probably a problem with speed of transmission (in Full HD 24fps is ok, but lags in 30 and more fps). Is it the problem of conversion ??

8

That depend on your pipeline. If your pipeline need memory copy that may cause the performance problem. You need to break down which gst element cause the problem.

9

Hi rfracer,

I’m facing similar performance issues. Could you please share your current OpenCV VideoCapture object instantiation code ? What is the net fps performance that you get with this code ?

In my case following line from with in OpenCV application -

VideoCapture cap1("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)I420, framerate=(fraction)60/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink");

creates considerable lag. Further more with “tegrastats” tool I see that this line jacks up the EMC freq really high, and I get few errors during my run. The video frames on HDMI also seem to lag considerably.

RAM 1977/3995MB (lfb 210x4MB) cpu [47%,53%,81%,17%]@1734 EMC 28%@1600 AVP 2%@12 NVDEC 192 MSENC 192 GR3D 41%@153 EDP limit 1734
RAM 1977/3995MB (lfb 210x4MB) cpu [22%,41%,66%,37%]@1734 EMC 26%@1600 AVP 2%@12 NVDEC 192 MSENC 192 GR3D 35%@153 EDP limit 1734
RAM 1976/3995MB (lfb 210x4MB) cpu [35%,87%,28%,55%]@1734 EMC 28%@1600 AVP 2%@12 NVDEC 192 MSENC 192 GR3D 39%@153 EDP limit 1734
RAM 1977/3995MB (lfb 210x4MB) cpu [47%,75%,40%,89%]@1734 EMC 32%@1600 AVP 2%@12 NVDEC 192 MSENC 192 GR3D 43%@153 EDP limit 1734

I see following warnings

(cv_hello:4814): GStreamer-CRITICAL **: gst_element_get_static_pad: assertion 'GST_IS_ELEMENT (element)' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_pad_get_current_caps: assertion 'GST_IS_PAD (pad)' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_structure_get_int: assertion 'structure != NULL' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_structure_get_int: assertion 'structure != NULL' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_structure_get_fraction: assertion 'structure != NULL' failed

I feel that I am having surplus memcpy operation in my opencv gstreamer pipe. Could someone please point out how to optimize that ?

On the other hand, when I preview video directly from gst-launch command line, my CPU and external memory controller freq doesnt get cranked to high value. I do not see any drop in fps either.

gst-launch-1.0 nvcamerasrc fpsRange="60.0 60.0" sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)60/1' ! nvtee ! nvvidconv flip-method=0 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e

RAM 1936/3995MB (lfb 228x4MB) cpu [44%,37%,18%,24%]@518 EMC 17%@665 AVP 26%@12 NVDEC 192 MSENC 192 GR3D 0%@76 EDP limit 1734
RAM 1936/3995MB (lfb 228x4MB) cpu [32%,42%,35%,31%]@403 EMC 20%@665 AVP 21%@12 NVDEC 192 MSENC 192 GR3D 0%@76 EDP limit 1734
RAM 1936/3995MB (lfb 228x4MB) cpu [32%,43%,35%,37%]@710 EMC 22%@665 AVP 16%@12 NVDEC 192 MSENC 192 GR3D 14%@76 EDP limit 1734
RAM 1937/3995MB (lfb 228x4MB) cpu [35%,34%,43%,36%]@307 EMC 22%@665 AVP 16%@12 NVDEC 192 MSENC 192 GR3D 0%@76 EDP limit 1734

Thanks,

10

Maybe you could try using I420 instead of BGRx between nvvidconv and videoconvert.
You could also check this: [url]https://devtalk.nvidia.com/default/topic/1001696/jetson-tx1/failed-to-open-tx1-on-board-camera/post/5117370/#5117370[/url]

11

I am also working on OpenCV 3.1 and having trouble.
I tried the command lines above, and they do not work.
My version is:
VideoCapture input1("nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)I420, framerate=(fraction)30/1’ ! nvvidconv flip-method=0 ! ‘video/x-raw(memory:NVMM), format=(string)I420’ ! appsink ");

I get video from the command line:
gst-launch-1.0 nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)I420, framerate=(fraction)30/1’ ! nvvidconv flip-method=0 ! ‘video/x-raw(memory:NVMM), format=(string)I420’ ! nvoverlaysink -e

I have also tried this with the BGR with the same error.

The error is: GStreamer-CRITICAL **: gst_element_make_from_uri: assertion ‘gst_uri_is_valid (uri)’ failed

Any help would be much appreciated.

12

Hi powerbeam
You may need convert to RGB for opencv. Below topic could be help for you.

https://devtalk.nvidia.com/default/topic/987537/jetson-tx1/videocapture-fails-to-open-onboard-camera-l4t-24-2-1-opencv-3-1/post/5064902/#5064902

13

Thank you ShaneCCC. And thank you Honey_Patouceul. This works:
nvcamerasrc sensor-id=0! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1 !
nvvidconv flip-method=0 ! video/x-raw, format=(string)I420!
videoconvert ! video/x-raw, format=(string)BGR !
appsink