Opencv VideoCapture With Gstreamer V4L2

Hi,
I have a hdmi -> csi convert board TC358743 . Here is formats output with

v4l2-ctl -d /dev/video0 --list-formats
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RGGB'
	Name        : 8-bit Bayer RGRG/GBGB

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'BG10'
	Name        : 10-bit Bayer BGBG/GRGR

	Index       : 3
	Type        : Video Capture
	Pixel Format: 'RG12'
	Name        : 12-bit Bayer RGRG/GBGB

	Index       : 4
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2

	Index       : 5
	Type        : Video Capture
	Pixel Format: 'NV16'
	Name        : Y/CbCr 4:2:2

	Index       : 6
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2

And success test with gst-launch-1.0 with follow command

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, width=1920, height=1080, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12' ! nvtee ! nvivafilter cuda-process=true pre-process=true post-process=true customer-lib-name="libnvsample_cudaprocess.so" ! 'video/x-raw(memory:NVMM), format=(string)NV12' ! nvoverlaysink display-id=0 -e

Then I can get the image from my own hdmi device .

And now I want to use opencv VideoCapture with Gstreamer to capture the video . Here is my code

VideoCapture capture("v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=(string)UYVY ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)I420 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink");
  if (!capture.isOpened())
  {
        cout << "capture device failed to open!" << endl;
    return 1;
  }

But I get the output is capture device failed to open! .
My opencv version is 3.3 with gstreamer plugin.

Video I/O:
--     DC1394 1.x:                  NO
--     DC1394 2.x:                  NO
--     FFMPEG:                      YES
--       avcodec:                   YES (ver 56.60.100)
--       avformat:                  YES (ver 56.40.101)
--       avutil:                    YES (ver 54.31.100)
--       swscale:                   YES (ver 3.1.101)
--       avresample:                NO
--     GStreamer:                   
--       base:                      YES (ver 1.8.3)
--       video:                     YES (ver 1.8.3)
--       app:                       YES (ver 1.8.3)
--       riff:                      YES (ver 1.8.3)
--       pbutils:                   YES (ver 1.8.3)
--     OpenNI:                      NO
--     OpenNI PrimeSensor Modules:  NO
--     OpenNI2:                     NO
--     PvAPI:                       NO
--     GigEVisionSDK:               NO
--     Aravis SDK:                  NO
--     UniCap:                      NO
--     UniCap ucil:                 NO
--     V4L/V4L2:                    NO/YES
--     XIMEA:                       NO
--     Xine:                        NO
--     Intel Media SDK:             NO
--     gPhoto2:                     NO

How can I modify to make it work.
Anything suggestions are appreciate !

Hi,

Is gst-launch able to preview below pipeline to monitor?

v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=(string)UYVY ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)I420 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! nvoverlaysink

Hi WayneWWW,
Thank you for your help .
I can’t get the preview of your code .
Here I run

gst-launch-1.0 --gst-debug-level=1 v4l2src device=/dev/video0 ! 'video/x-raw,width=1920,height=1080,format=(string)UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)I420' ! nvvidconv ! 'video/x-raw, format=(string)BGRx' ! videoconvert ! 'video/x-raw, format=(string)BGR' ! nvoverlaysink

Then I get following debug output :

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:04.792662464  4857       0x5bc850 ERROR           omxvideosink gstomxvideosink.c:1242:gst_omx_video_sink_setcaps:<nvoverlaysink-nvoverlaysink0> Unsupported format BGR
0:00:04.793011200  4857       0x5bc850 ERROR           omxvideosink gstomxvideosink.c:1242:gst_omx_video_sink_setcaps:<nvoverlaysink-nvoverlaysink0> Unsupported format BGR
0:00:04.859536992  4857       0x5bc850 ERROR           omxvideosink gstomxvideosink.c:1242:gst_omx_video_sink_setcaps:<nvoverlaysink-nvoverlaysink0> Unsupported format BGR
0:00:04.914503616  4857       0x5bc850 ERROR           omxvideosink gstomxvideosink.c:1242:gst_omx_video_sink_setcaps:<nvoverlaysink-nvoverlaysink0> Unsupported format BGR
0:00:04.914767968  4857       0x5bc850 ERROR           omxvideosink gstomxvideosink.c:1242:gst_omx_video_sink_setcaps:<nvoverlaysink-nvoverlaysink0> Unsupported format BGR
0:00:04.914855584  4857       0x5bc850 ERROR           omxvideosink gstomxvideosink.c:1242:gst_omx_video_sink_setcaps:<nvoverlaysink-nvoverlaysink0> Unsupported format BGR
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2948): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 0:00:04.799141472
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

It seems that nvoverlaysink can’t support with BGR format.

And I can get preview of the following code

gst-launch-1.0 --gst-debug-level=1 v4l2src device=/dev/video0 ! 'video/x-raw,width=1920,height=1080,format=(string)UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)I420' ! nvoverlaysink

And if I directly open

VideoCapture capture(0)

I get the following error :

VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV
capture device failed to open!

You may check which version libs are used in your opencv application with something like:

std::cout << cv::getBuildInformation() << std::endl;

or

ldd you_opencv_app

If you still have a previous version installed in /usr, then this version will be used as it is in standard path, unless you set LD_LIBRARY_PATH with the path to your gstreamer enabled version.

Hi Honey_Patouceul,
Thank you for your help !
Here are my opencv libs version output with cv::getBuildInformation()

General configuration for OpenCV 3.3.0 =====================================
  Version control:               unknown

  Platform:
    Timestamp:                   2018-03-25T16:27:27Z
    Host:                        Linux 4.4.38-tegra aarch64
    CMake:                       3.5.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               Release

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 5.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):
    Linker flags (Debug):
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          gtk-x11-2.0 gdk-x11-2.0 pangocairo-1.0 atk-1.0 cairo gdk_pixbuf-2.0 gio-2.0 pangoft2-1.0 pango-1.0 fontconfig freetype gthread-2.0 /usr/lib/aarch64-linux-gnu/libpng.so /usr/lib/aarch64-linux-gnu/libz.so /usr/lib/aarch64-linux-gnu/libtiff.so /usr/lib/aarch64-linux-gnu/libjasper.so /usr/lib/aarch64-linux-gnu/libjpeg.so gstbase-1.0 gstreamer-1.0 gobject-2.0 glib-2.0 gstvideo-1.0 gstapp-1.0 gstriff-1.0 gstpbutils-1.0 avcodec-ffmpeg avformat-ffmpeg avutil-ffmpeg swscale-ffmpeg dl m pthread rt /usr/lib/aarch64-linux-gnu/libtbb.so
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 core flann imgproc ml objdetect photo video dnn imgcodecs shape videoio highgui superres ts features2d calib3d stitching videostab python2
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev java python3 viz

  GUI: 
    QT:                          NO
    GTK+ 2.x:                    YES (ver 2.24.30)
    GThread :                    YES (ver 2.48.1)
    GtkGlExt:                    NO
    OpenGL support:              NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver )
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 - 4.0.6)
    JPEG 2000:                   /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)
    OpenEXR:                     NO
    GDAL:                        NO
    GDCM:                        NO

  Video I/O:
    DC1394 1.x:                  NO
    DC1394 2.x:                  NO
    FFMPEG:                      YES
      avcodec:                   YES (ver 56.60.100)
      avformat:                  YES (ver 56.40.101)
      avutil:                    YES (ver 54.31.100)
      swscale:                   YES (ver 3.1.101)
      avresample:                NO
    GStreamer:                   
      base:                      YES (ver 1.8.3)
      video:                     YES (ver 1.8.3)
      app:                       YES (ver 1.8.3)
      riff:                      YES (ver 1.8.3)
      pbutils:                   YES (ver 1.8.3)
    OpenNI:                      NO
    OpenNI PrimeSensor Modules:  NO
    OpenNI2:                     NO
    PvAPI:                       NO
    GigEVisionSDK:               NO
    Aravis SDK:                  NO
    UniCap:                      NO
    UniCap ucil:                 NO
    V4L/V4L2:                    NO/YES
    XIMEA:                       NO
    Xine:                        NO
    Intel Media SDK:             NO
    gPhoto2:                     NO

  Parallel framework:            TBB (ver 4.4 interface 9002)

  Trace:                         YES ()

  Other third-party libraries:
    Use Intel IPP:               NO
    Use Intel IPP IW:            NO
    Use VA:                      NO
    Use Intel VA-API/OpenCL:     NO
    Use Lapack:                  NO
    Use Eigen:                   YES (ver 3.2.92)
    Use Cuda:                    NO
    Use OpenCL:                  NO
    Use OpenVX:                  NO
    Use custom HAL:              YES (carotene (ver 0.0.1))

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.12)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
    numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0)
    packages path:               lib/python2.7/dist-packages

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.5.2)

  Python (for build):            /usr/bin/python2.7

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Matlab:                        Matlab not found or implicitly disabled

  Documentation:
    Doxygen:                     NO

  Tests and samples:
    Tests:                       YES
    Performance tests:           YES
    C/C++ Examples:              YES

  Install path:                  /usr

  cvconfig.h is in:              /home/nvidia/opencv/opencv-3.3.0/build

when I run my app It show that

OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in icvStartPipeline, file /home/nvidia/opencv/opencv-3.3.0/modules/videoio/src/cap_gstreamer.cpp, line 441
terminate called after throwing an instance of 'cv::Exception'
  what():  /home/nvidia/opencv/opencv-3.3.0/modules/videoio/src/cap_gstreamer.cpp:441: error: (-2) GStreamer: unable to start pipeline
 in function icvStartPipeline

Aborted (core dumped)

And Here is my capcature open code .

VideoCapture capture("v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=(string)UYVY ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)I420 ! nvvidconv! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink");

I don’t have such camera for trying. It looks ok to me so far.
I have confirmed with a ZED camera (USB) on my TX2 R28.2 with opencv-3.4.1 that your pipeline works with this sample code:

#include <iostream>

#include <opencv2/opencv.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/cudaimgproc.hpp>

int main()
{
	/* Onboard OV5693 */
	/* const char* gst =  "nvcamerasrc  ! video/x-raw(memory:NVMM), format=(string)I420, width=(int)640, height=(int)480, framerate=(fraction)30/1 ! \
				nvvidconv    ! video/x-raw,              format=(string)BGRx ! \
				videoconvert ! video/x-raw,              format=(string)BGR  ! \
				appsink"; */
						

	/* ZED camera */
	/* const char* gst = 	"v4l2src device=/dev/video1 ! video/x-raw, width=2560, height=720, format=YUY2, framerate=60/1 ! "
				"videoconvert ! video/x-raw, width=2560, height=720, format=BGR ! "
				"appsink"; */

	/* Your case adapted to ZED camera on my TX2 */
	const char* gst = 	"v4l2src device=/dev/video1 ! video/x-raw, width=2560, height=720, format=YUY2, framerate=15/1 ! "
				"nvvidconv ! video/x-raw(memory:NVMM), format=I420 ! "
				"nvvidconv ! video/x-raw, format=BGRx ! "
				"videoconvert ! video/x-raw, format=BGR ! "
				"appsink"; 

	// std::cout << cv::getBuildInformation() << std::endl; 
	
	// setenv("GST_DEBUG", "*:3", 0); 

	cv::VideoCapture cap(gst);
	if(!cap.isOpened()) {
		std::cout<< "Failed to open camera." << std::endl;
		return (-1);
	}
    
	unsigned int width  = cap.get(CV_CAP_PROP_FRAME_WIDTH); 
	unsigned int height = cap.get(CV_CAP_PROP_FRAME_HEIGHT); 
	unsigned int fps    = cap.get(CV_CAP_PROP_FPS);
	unsigned int pixels = width*height;
	std::cout <<" Frame size : "<<width<<" x "<<height<<", "<<pixels<<" Pixels "<<fps<<" FPS"<<std::endl;

	cv::namedWindow("MyCameraPreview", CV_WINDOW_AUTOSIZE);
	cv::Mat frame_in;
    
	while(1)
	{
    		if (!cap.read(frame_in)) {
			std::cout<<"Capture read error"<<std::endl;
			break;
		}
		else  {
			cv::imshow("MyCameraPreview",frame_in);
			cv::waitKey(1); // let imshow draw 
		}	
	}

	cap.release();

	return 0;
}

You may try your case with it, be sure to compile with

-I<path_to_opencv-3.3.0_headers>

before any -I /usr/include
and link with

-L<path_to_opencv-3.3.0_libs>

before any -L /usr/lib
and export LD_LIBRARY_PATH to have <path_to_opencv-3.3.0_libs> in the shell env before running:

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:<path_to_opencv-3.3.0_libs>

If it still fails, you may uncomment the line with GST_DEBUG for enabling debug mode and look for errors.

You might also run your application with strace and check if you see some error (such as permissions) that might explain the exception.

Last, I have no experience with other CSI camera than onboard cam and I don’t know how much customized is your device tree or kernel for this camera, but if you have something like nvcamera-deamon running, you may also try to use “–set-ctrl bypass_mode=0” when configuring your device with v4l2-ctl. Do you see kernel errors ?

Hi Honey_Patouceul,
It is much appreciate of your kind help !
For your instructions , first I check my kernel error with

dmesg |grep error

And show me that

[    1.342403] pca953x: probe of 0-0077 failed with error -121
[    4.742070] ina3221x: probe of 0-0042 failed with error -121
[    4.743106] ina3221x: probe of 0-0043 failed with error -121
[    4.992203] xhci-tegra 3530000.xhci: Direct firmware load for tegra18x_xusb_firmware failed with error -2

It seems that no relationship with kernel problem
Then I compile with -I<opencv3.3.0_headers> -L<opencv3.3.0_libs> , but still error .
Because before I compile opencv 3.3.0 I have installed opencv3.3.1 with Jetpack . So I remove all the header files and libs of opencv in /usr/include and /usr/libs . and reinstall opencv 3.3.0 ,Finally it works pretty well .
Thanks again!