V4L2 with opencv in python

I want to display the camera stream directly using opencv, but it seems to not be able to support my pixel format:

(python:5901): GStreamer-CRITICAL **: gst_element_get_state: assertion 'GST_IS_ELEMENT (element)' failed
VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV
VIDEOIO ERROR: V4L: can't open camera by index 1

This is the code I run to reproduce this error:

import cv2

I do not want to use gstreamer to convert my video to BGR because that causes some lag in the video, but it works:

cv2.VideoCapture("nvcamerasrc fpsRange='1.0 60.0' sensor-id=1 ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)15/1 ! nvvidconv flip-method=6 ! video/x-raw, format=(string)I420 ! videoconvert ! video/x-raw, format=(string)BGR ! appsink")

There is no lag running

gst-launch-1.0 nvcamerasrc fpsRange="15.0 15.0" sensor-id=1 ! 'video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=(string)I420, framerate=(fraction)15/1' ! nvtee ! nvvidconv flip-method=2 ! 'video/x-raw, format=(string)I420' ! xvimagesink -e

but when I try to run this in opencv:

cv2.VideoCapture("gst-launch-1.0 nvcamerasrc fpsRange='15.0 15.0' sensor-id=1 ! video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=(string)I420, framerate=(fraction)15/1 ! nvtee ! nvvidconv flip-method=2 ! video/x-raw, format=(string)I420 ! xvimagesink -e")

cap.read() returns a None type. I have also tried using the v4l2capture library in python using the code:

import numpy as np
import cv2
import os
import v4l2capture
import select

video = v4l2capture.Video_device("/dev/video1")
while True:
    select.select((video,), (), ()) # Wait for the device to fill the buffer.
    image_data = video.read_and_queue()

    frame = np.frombuffer(image_data, dtype=np.uint8).reshape(1920,1080,2)
    cv_frame = cv2.imdecode(frame, cv2.COLOR_BAYER_RG2BGR)

    cv2.imshow('frame', frame[:,:,0])
    key = cv2.waitKey(1)
        if key & 0xFF == ord('q'):

the two channels on “frame” (frame[:,:,0] and frame[:,:,1]) show nothing but static looking images, but this responds in real time. “cv_frame” returns all zeros.
After running

v4l2-ctl -d /dev/video1 --list-formats-ext

I get

ioctl: VIDIOC_ENUM_FMT        Index       : 0
        Type        : Video Capture
        Pixel Format: 'RG10'
        Name        : 10-bit Bayer RGRG/GBGB
                Size: Discrete 4056x3040
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 2028x1520
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (60.000 fps)


v4l2-ctl --all

I get

Driver Info (not using libv4l2):
        Driver name   : tegra-video
        Card type     : vi-output, imx477 40-0010
        Bus info      : platform:15700000.vi:0
        Driver version: 4.4.38
        Capabilities  : 0x84200001
                Video Capture
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture
                Extended Pix Format
Priority: 2
Video input : 0 (Camera 0: no power)
Format Video Capture:
        Width/Height      : 4056/3040
        Pixel Format      : 'RG10'
        Field             : None
        Bytes per Line    : 8192
        Size Image        : 24903680
        Colorspace        : sRGB
        Transfer Function : Default
        YCbCr Encoding    : Default
        Quantization      : Default
        Flags             :

Camera Controls

                   frame_length (int)    : min=1551 max=45714 step=1 default=4291 value=4291 flags=slider
                    coarse_time (int)    : min=1 max=45704 step=1 default=4281 value=4281 flags=slider
              coarse_time_short (int)    : min=1 max=45704 step=1 default=4281 value=4281 flags=slider
                     group_hold (intmenu): min=0 max=1 default=0 value=0
                     hdr_enable (intmenu): min=0 max=1 default=0 value=0
                           gain (int)    : min=256 max=90624 step=1 default=256 value=256 flags=slider
                    bypass_mode (intmenu): min=0 max=1 default=0 value=0
                override_enable (intmenu): min=0 max=1 default=0 value=0
                   height_align (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align (intmenu): min=0 max=2 default=0 value=0
               write_isp_format (int)    : min=1 max=1 step=1 default=1 value=1
       sensor_signal_properties (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
        sensor_image_properties (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
      sensor_control_properties (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
              sensor_dv_timings (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
                   sensor_modes (int)    : min=0 max=30 step=1 default=30 value=3 flags=read-only

I think I have to debayer my data but I’m not sure if I’m doing this correctly. Maybe I am not reshaping my arrays correctly? I recompiled opencv 4.0 from source using



Here is the output from

print cv2.getBuildInformation()
General configuration for OpenCV 4.0.0-dev =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            /home/nvidia/src/opencv-master/opencv_contrib-master/modules
    Version control (extra):     unknown

    Timestamp:                   2018-12-20T06:22:58Z
    Host:                        Linux 4.4.38-tegra-leopard aarch64
    CMake:                       3.5.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 5.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffast-math -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffast-math -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffast-math -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffast-math -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      
    Linker flags (Debug):        
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          dl m pthread rt /usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so cudart nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda-9.0/lib64
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv datasets dnn dnn_objdetect dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc java_bindings_generator line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 python_bindings_generator reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cnn_3dobj hdf java js matlab ovis sfm ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         NO

    QT:                          YES (ver 5.5.1)
      QT OpenGL support:         YES (Qt5::OpenGL 5.5.1)
    GTK+:                        NO
    OpenGL support:              YES (/usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so)
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.6)
    JPEG 2000:                   /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)
    OpenEXR:                     build (ver 1.7.1)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      YES (ver 2.2.4)
    FFMPEG:                      YES
      avcodec:                   YES (ver 56.60.100)
      avformat:                  YES (ver 56.40.101)
      avutil:                    YES (ver 54.31.100)
      swscale:                   YES (ver 3.1.101)
      avresample:                NO
      base:                      YES (ver 1.8.3)
      video:                     YES (ver 1.8.3)
      app:                       YES (ver 1.8.3)
      riff:                      YES (ver 1.8.3)
      pbutils:                   YES (ver 1.8.3)
    v4l/v4l2:                    linux/videodev2.h

  Parallel framework:            pthreads

  Trace:                         YES (built-in)

  Other third-party libraries:
    Lapack:                      YES (/usr/lib/liblapack.so /usr/lib/libcblas.so /usr/lib/libatlas.so)
    Eigen:                       YES (ver 3.2.92)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.5.1)

  NVIDIA CUDA:                   YES (ver 9.0, CUFFT CUBLAS FAST_MATH)
    NVIDIA GPU arch:             62
    NVIDIA PTX archs:

  OpenCL:                        YES (no extra features)
    Include path:                /home/nvidia/src/opencv-master/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.12)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
    numpy:                       /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.15.3)
    install path:                lib/python2.7/dist-packages/cv2/python-2.7

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.5.2)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.5m.so (ver 3.5.2)
    numpy:                       /usr/local/lib/python3.5/dist-packages/numpy/core/include (ver 1.15.3)
    install path:                lib/python3.5/dist-packages/cv2/python-3.5

  Python (for build):            /usr/bin/python2.7

    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local

Any guidance would be much appreciated.

This usage is wrong. Please refer to other users’ thread for gstreamer + openCV.

cv2.VideoCapture("gst-launch-1.0 nvcamerasrc fpsRange='15.0 15.0' sensor-id=1 ! video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=(string)I420, framerate=(fraction)15/1 ! nvtee ! nvvidconv flip-method=2 ! video/x-raw, format=(string)I420 ! xvimagesink -e")

Hi WayneWWW, you’re right that it was the incorrect usage. I corrected it with

cv2.VideoCapture("nvcamerasrc fpsRange='15.0 15.0' sensor-id=1 ! video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=(string)I420, framerate=(fraction)15/1 ! nvtee ! nvvidconv flip-method=2 ! video/x-raw, format=(string)I420 ! appsink", cv2.CAP_GSTREAMER)

That took care of most of the issues I was facing.

Is there a way to use the raw V4L format in opencv by simply calling




Are you using VideoCapture to capture on-board camera(csi)?

Yes, that was my attempt to do so.

Hi bsmith18,

This should not work. VideoCapture would go for UVC driver, which is not compatible with csi camera.

To use csi camera, please always use gstreamer pipeline.

Hi WayneWWW,

Is there a way to access the video stream directly from memory into a GpuMat?

This thread may help.