I want to display the camera stream directly using opencv, but it seems to not be able to support my pixel format:
(python:5901): GStreamer-CRITICAL **: gst_element_get_state: assertion 'GST_IS_ELEMENT (element)' failed
VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV
VIDEOIO ERROR: V4L: can't open camera by index 1
This is the code I run to reproduce this error:
import cv2
cv2.VideoCapture(1)
I do not want to use gstreamer to convert my video to BGR because that causes some lag in the video, but it works:
cv2.VideoCapture("nvcamerasrc fpsRange='1.0 60.0' sensor-id=1 ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)15/1 ! nvvidconv flip-method=6 ! video/x-raw, format=(string)I420 ! videoconvert ! video/x-raw, format=(string)BGR ! appsink")
There is no lag running
gst-launch-1.0 nvcamerasrc fpsRange="15.0 15.0" sensor-id=1 ! 'video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=(string)I420, framerate=(fraction)15/1' ! nvtee ! nvvidconv flip-method=2 ! 'video/x-raw, format=(string)I420' ! xvimagesink -e
but when I try to run this in opencv:
cv2.VideoCapture("gst-launch-1.0 nvcamerasrc fpsRange='15.0 15.0' sensor-id=1 ! video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=(string)I420, framerate=(fraction)15/1 ! nvtee ! nvvidconv flip-method=2 ! video/x-raw, format=(string)I420 ! xvimagesink -e")
cap.read() returns a None type. I have also tried using the v4l2capture library in python using the code:
import numpy as np
import cv2
import os
import v4l2capture
import select
video = v4l2capture.Video_device("/dev/video1")
video.set_format(1920,1080)
video.create_buffers(1)
video.queue_all_buffers()
video.start()
while True:
select.select((video,), (), ()) # Wait for the device to fill the buffer.
image_data = video.read_and_queue()
frame = np.frombuffer(image_data, dtype=np.uint8).reshape(1920,1080,2)
cv_frame = cv2.imdecode(frame, cv2.COLOR_BAYER_RG2BGR)
cv2.imshow('frame', frame[:,:,0])
key = cv2.waitKey(1)
if key & 0xFF == ord('q'):
break
video.close()
cv2.destroyAllWindows()
the two channels on “frame” (frame[:,:,0] and frame[:,:,1]) show nothing but static looking images, but this responds in real time. “cv_frame” returns all zeros.
After running
v4l2-ctl -d /dev/video1 --list-formats-ext
I get
ioctl: VIDIOC_ENUM_FMT Index : 0
Type : Video Capture
Pixel Format: 'RG10'
Name : 10-bit Bayer RGRG/GBGB
Size: Discrete 4056x3040
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 2028x1520
Interval: Discrete 0.017s (60.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.017s (60.000 fps)
running
v4l2-ctl --all
I get
Driver Info (not using libv4l2):
Driver name : tegra-video
Card type : vi-output, imx477 40-0010
Bus info : platform:15700000.vi:0
Driver version: 4.4.38
Capabilities : 0x84200001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Priority: 2
Video input : 0 (Camera 0: no power)
Format Video Capture:
Width/Height : 4056/3040
Pixel Format : 'RG10'
Field : None
Bytes per Line : 8192
Size Image : 24903680
Colorspace : sRGB
Transfer Function : Default
YCbCr Encoding : Default
Quantization : Default
Flags :
Camera Controls
frame_length (int) : min=1551 max=45714 step=1 default=4291 value=4291 flags=slider
coarse_time (int) : min=1 max=45704 step=1 default=4281 value=4281 flags=slider
coarse_time_short (int) : min=1 max=45704 step=1 default=4281 value=4281 flags=slider
group_hold (intmenu): min=0 max=1 default=0 value=0
hdr_enable (intmenu): min=0 max=1 default=0 value=0
gain (int) : min=256 max=90624 step=1 default=256 value=256 flags=slider
bypass_mode (intmenu): min=0 max=1 default=0 value=0
override_enable (intmenu): min=0 max=1 default=0 value=0
height_align (int) : min=1 max=16 step=1 default=1 value=1
size_align (intmenu): min=0 max=2 default=0 value=0
write_isp_format (int) : min=1 max=1 step=1 default=1 value=1
sensor_signal_properties (u32) : min=0 max=0 step=0 default=0 flags=read-only, has-payload
sensor_image_properties (u32) : min=0 max=0 step=0 default=0 flags=read-only, has-payload
sensor_control_properties (u32) : min=0 max=0 step=0 default=0 flags=read-only, has-payload
sensor_dv_timings (u32) : min=0 max=0 step=0 default=0 flags=read-only, has-payload
sensor_modes (int) : min=0 max=30 step=1 default=30 value=3 flags=read-only
I think I have to debayer my data but I’m not sure if I’m doing this correctly. Maybe I am not reshaping my arrays correctly? I recompiled opencv 4.0 from source using
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_CUDA=ON -D CUDA_ARCH_BIN="6.2" -D CUDA_ARCH_PTX="" -D WITH_CUBLAS=ON -D ENABLE_FAST_MATH=ON -D CUDA_FAST_MATH=ON -D ENABLE_NEON=ON -D WITH_LIBV4L=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_GSTREAMER=ON -D WITH_GSTREAMER_0_10=OFF -D WITH_OPENCL=ON -D BUILD_opencv_python2=ON -D BUILD_opencv_python3=ON -D WITH_FFMPEG=ON -D WITH_CUFFT=ON -D WITH_V4L=ON -D WITH_V4L2=ON -D WITH_LIBV4L2=ON -D OPENCV_EXTRA_MODULES_PATH=../opencv_contrib-master/modules ..
.
Here is the output from
print cv2.getBuildInformation()
General configuration for OpenCV 4.0.0-dev =====================================
Version control: unknown
Extra modules:
Location (extra): /home/nvidia/src/opencv-master/opencv_contrib-master/modules
Version control (extra): unknown
Platform:
Timestamp: 2018-12-20T06:22:58Z
Host: Linux 4.4.38-tegra-leopard aarch64
CMake: 3.5.1
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: RELEASE
CPU/HW features:
Baseline: NEON FP16
required: NEON
disabled: VFPV3
C/C++:
Built as dynamic libs?: YES
C++ Compiler: /usr/bin/c++ (ver 5.4.0)
C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffast-math -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffast-math -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffast-math -ffunction-sections -fdata-sections -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffast-math -ffunction-sections -fdata-sections -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release):
Linker flags (Debug):
ccache: NO
Precompiled headers: YES
Extra dependencies: dl m pthread rt /usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so cudart nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda-9.0/lib64
3rdparty dependencies:
OpenCV modules:
To be built: aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv datasets dnn dnn_objdetect dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc java_bindings_generator line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 python_bindings_generator reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
Disabled: world
Disabled by dependency: -
Unavailable: cnn_3dobj hdf java js matlab ovis sfm ts viz
Applications: apps
Documentation: NO
Non-free algorithms: NO
GUI:
QT: YES (ver 5.5.1)
QT OpenGL support: YES (Qt5::OpenGL 5.5.1)
GTK+: NO
OpenGL support: YES (/usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so)
VTK support: NO
Media I/O:
ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
WEBP: build (ver encoder: 0x020e)
PNG: /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
TIFF: /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.6)
JPEG 2000: /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)
OpenEXR: build (ver 1.7.1)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
DC1394: YES (ver 2.2.4)
FFMPEG: YES
avcodec: YES (ver 56.60.100)
avformat: YES (ver 56.40.101)
avutil: YES (ver 54.31.100)
swscale: YES (ver 3.1.101)
avresample: NO
GStreamer:
base: YES (ver 1.8.3)
video: YES (ver 1.8.3)
app: YES (ver 1.8.3)
riff: YES (ver 1.8.3)
pbutils: YES (ver 1.8.3)
v4l/v4l2: linux/videodev2.h
Parallel framework: pthreads
Trace: YES (built-in)
Other third-party libraries:
Lapack: YES (/usr/lib/liblapack.so /usr/lib/libcblas.so /usr/lib/libatlas.so)
Eigen: YES (ver 3.2.92)
Custom HAL: YES (carotene (ver 0.0.1))
Protobuf: build (3.5.1)
NVIDIA CUDA: YES (ver 9.0, CUFFT CUBLAS FAST_MATH)
NVIDIA GPU arch: 62
NVIDIA PTX archs:
OpenCL: YES (no extra features)
Include path: /home/nvidia/src/opencv-master/3rdparty/include/opencl/1.2
Link libraries: Dynamic load
Python 2:
Interpreter: /usr/bin/python2.7 (ver 2.7.12)
Libraries: /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
numpy: /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.15.3)
install path: lib/python2.7/dist-packages/cv2/python-2.7
Python 3:
Interpreter: /usr/bin/python3 (ver 3.5.2)
Libraries: /usr/lib/aarch64-linux-gnu/libpython3.5m.so (ver 3.5.2)
numpy: /usr/local/lib/python3.5/dist-packages/numpy/core/include (ver 1.15.3)
install path: lib/python3.5/dist-packages/cv2/python-3.5
Python (for build): /usr/bin/python2.7
Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO
Install to: /usr/local
-----------------------------------------------------------------
Any guidance would be much appreciated.