MIPI CS Camera and OpenCV

Hello,
Togheter with my friend, we work on a project that includes Jetson Tx1 with OpenCV(3.1).
We have leopard camera LI-IMX274-MIPI-CS. We have drivers and get image in a similar way like on onboard camera(given my NVidia 5MP MIPI Camera) by using gst-launch.

We try to get image like in this topic: https://devtalk.nvidia.com/default/topic/943129/jetson-tx1/highgui-error-v4l-v4l2-while-opening-camera-with-opencv4tegra-l4t-r24/post/4921383/#4921383

Code:

VideoCapture cap("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)24/1 ! nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"); //open the default camera

if(!cap.isOpened()) { // check if we succeeded
cerr << "Fail to open camera " << endl;
return -1;
}
for(;;)
{
Mat frame;
cap >> frame; // get a new frame from camera
imshow("original", frame);
waitKey(1);
}
// the camera will be deinitialized automatically in VideoCapture destructor
cap.release();

But we get error: “Fail to open camera”.

Maybe someone also use Leopard TX1 CS MIPI camera with OpenCV and know how to do it.

Did you try the gst-launch command working first?

gst-launch-1.0 nvcamerasrc fpsRange=“30.0 30.0” ! ‘video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1’ ! nvtee ! nvvidconv flip-method=2 ! ‘video/x-raw(memory:NVMM), format=(string)I420’ ! nvoverlaysink -e

Yes this command working.

The issue Is That we dont know how to convert it to stream that will work in OpenCV.

Your command can’t launch by the gst-launch-1.0, you need use gst-launch-1.0 to make sure your command is correct first. What I mean is below command have syntax error. What’s your purpose?

gst-launch-1.0 nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)24/1 ! nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink

Hi Shane,

Is your gst-launch string V4L2 compatible? I believe OpenCV expects V4L2.

I think in OpenCV, the mat container/format is the universal input tool…perhaps there is a different nvcamerasrc gst pipeline that can deliver objects in mat format to OpenCV?

Barrett

Problem solved.
That was problem with OpenCV and Gstreamer connection. Thanks for help :)

After launch camera in OpenCV I see That is probably a problem with speed of transmission (in Full HD 24fps is ok, but lags in 30 and more fps). Is it the problem of conversion ??

That depend on your pipeline. If your pipeline need memory copy that may cause the performance problem. You need to break down which gst element cause the problem.

Hi rfracer,

I’m facing similar performance issues. Could you please share your current OpenCV VideoCapture object instantiation code ? What is the net fps performance that you get with this code ?

In my case following line from with in OpenCV application -

VideoCapture cap1("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)I420, framerate=(fraction)60/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink");

creates considerable lag. Further more with “tegrastats” tool I see that this line jacks up the EMC freq really high, and I get few errors during my run. The video frames on HDMI also seem to lag considerably.

RAM 1977/3995MB (lfb 210x4MB) cpu [47%,53%,81%,17%]@1734 EMC 28%@1600 AVP 2%@12 NVDEC 192 MSENC 192 GR3D 41%@153 EDP limit 1734
RAM 1977/3995MB (lfb 210x4MB) cpu [22%,41%,66%,37%]@1734 EMC 26%@1600 AVP 2%@12 NVDEC 192 MSENC 192 GR3D 35%@153 EDP limit 1734
RAM 1976/3995MB (lfb 210x4MB) cpu [35%,87%,28%,55%]@1734 EMC 28%@1600 AVP 2%@12 NVDEC 192 MSENC 192 GR3D 39%@153 EDP limit 1734
RAM 1977/3995MB (lfb 210x4MB) cpu [47%,75%,40%,89%]@1734 EMC 32%@1600 AVP 2%@12 NVDEC 192 MSENC 192 GR3D 43%@153 EDP limit 1734

I see following warnings

(cv_hello:4814): GStreamer-CRITICAL **: gst_element_get_static_pad: assertion 'GST_IS_ELEMENT (element)' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_pad_get_current_caps: assertion 'GST_IS_PAD (pad)' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_structure_get_int: assertion 'structure != NULL' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_structure_get_int: assertion 'structure != NULL' failed

(cv_hello:4814): GStreamer-CRITICAL **: gst_structure_get_fraction: assertion 'structure != NULL' failed

I feel that I am having surplus memcpy operation in my opencv gstreamer pipe. Could someone please point out how to optimize that ?

On the other hand, when I preview video directly from gst-launch command line, my CPU and external memory controller freq doesnt get cranked to high value. I do not see any drop in fps either.

gst-launch-1.0 nvcamerasrc fpsRange="60.0 60.0" sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)60/1' ! nvtee ! nvvidconv flip-method=0 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
RAM 1936/3995MB (lfb 228x4MB) cpu [44%,37%,18%,24%]@518 EMC 17%@665 AVP 26%@12 NVDEC 192 MSENC 192 GR3D 0%@76 EDP limit 1734
RAM 1936/3995MB (lfb 228x4MB) cpu [32%,42%,35%,31%]@403 EMC 20%@665 AVP 21%@12 NVDEC 192 MSENC 192 GR3D 0%@76 EDP limit 1734
RAM 1936/3995MB (lfb 228x4MB) cpu [32%,43%,35%,37%]@710 EMC 22%@665 AVP 16%@12 NVDEC 192 MSENC 192 GR3D 14%@76 EDP limit 1734
RAM 1937/3995MB (lfb 228x4MB) cpu [35%,34%,43%,36%]@307 EMC 22%@665 AVP 16%@12 NVDEC 192 MSENC 192 GR3D 0%@76 EDP limit 1734

Thanks,

Maybe you could try using I420 instead of BGRx between nvvidconv and videoconvert.
You could also check this: https://devtalk.nvidia.com/default/topic/1001696/jetson-tx1/failed-to-open-tx1-on-board-camera/post/5117370/#5117370

I am also working on OpenCV 3.1 and having trouble.
I tried the command lines above, and they do not work.
My version is:
VideoCapture input1("nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)I420, framerate=(fraction)30/1’ ! nvvidconv flip-method=0 ! ‘video/x-raw(memory:NVMM), format=(string)I420’ ! appsink ");

I get video from the command line:
gst-launch-1.0 nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)I420, framerate=(fraction)30/1’ ! nvvidconv flip-method=0 ! ‘video/x-raw(memory:NVMM), format=(string)I420’ ! nvoverlaysink -e

I have also tried this with the BGR with the same error.

The error is: GStreamer-CRITICAL **: gst_element_make_from_uri: assertion ‘gst_uri_is_valid (uri)’ failed

Any help would be much appreciated.

Hi powerbeam
You may need convert to RGB for opencv. Below topic could be help for you.

https://devtalk.nvidia.com/default/topic/987537/jetson-tx1/videocapture-fails-to-open-onboard-camera-l4t-24-2-1-opencv-3-1/post/5064902/#5064902

Thank you ShaneCCC. And thank you Honey_Patouceul. This works:
nvcamerasrc sensor-id=0! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1 !
nvvidconv flip-method=0 ! video/x-raw, format=(string)I420!
videoconvert ! video/x-raw, format=(string)BGR !
appsink