How does the camera control work ?

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi guys,

I have recently purchased a new camera from E-conSystem which is great but I have a question I can’t find an answer to.
I’d using it with cv2.VideoCapture() and I used to apply this command for the onboard camera:

cap = cv2.VideoCapture("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)640, height=(int)360,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink")

Now I just use “cv2.VideoCapture(0)” cause I have installed the packages and I have the /dev/video0 to use the camera.

My question is: how to use properly the option to set the right mode of the camera ?

The guide only indicate the option with Gstreamer:

gst-launch-1.0 v4l2src device=/dev/video0! "video/x-raw, format=(string)UYVY,
width=(int)2304, height=(int)1296" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)I420,
width=(int)1920, height=(int)1080" ! nvoverlaysink overlay-w=1920 overlay-h=1080 sync=false

but I don’t understand how the option really works…particulary the “appsink” or the “nvoverlaysink” I don’t get those…

If someone could explain me or point to some documentation I would be grateful.

Thanks

2

Depending on your camera, you can access it with several ways. You may read this post for general info about these in case of devkit onboard camera.

If your new cam is USB, its driver should create a node such as /dev/video1 (on devkit if you have the onboard camera it is /dev/video0) as you plug it in. So you can check with

ls -l /dev/video*

what is its dev node.

Once you’ve got it, you can access with V4L interface from opencv with

cv2.VideoCapture(1)

1 stands for /dev/video1. If your opencv frame processing expects BGR, you may have to convert from UYVY to BGR. Opencv cvtColor function should be able to do that. You may also use gstreamer for converting into BGR.

The guide’s command was using a gstreamer pipeline accessing the camera on node /dev/video with V4L interface (v4l2src). Seems this camera provides 2304x1296 in UYVY format (quoted are the capabilities). Then it is copied into contiguous memory (NVMM) and converted to 1920x1080 in I420 format (I think by ISP) (nvvidconv). nvoverlaysink reads this in contiguous memory displays on monitor.

3

My camera replaced the onboard one, it uses the MIPI connector (J27 I think).

The thing is when I use just the command “cv2.VideoCapture(0)” I have a window of size 480x640, but my camera can have a way better resolution so I want to use this power to stream and then do some treatment on it.

So you say that I should do something like:

cap=cv2.VideoCapture(0)
ok,frame=cap.read()
cv2.cvtColor(frame,UYVY2BGR)

image_treatment(frame)

My question was more in the option to be used directly in the VideoCapture() function. Cause somehow I managed lauching using this command:

"v4l2src device=/dev/video0! video/x-raw(memory:NVMM), format=(string)I420,
width=(int)1920, height=(int)1080" ! nvoverlaysink overlay-w=1920 overlay-h=1080 sync=false ! appsink"

but it freezes immediatly… so I feel I can use the option to have the right capture option but I don’t know how

4

If your processing expects BGR frames, then yes you would have to convert.

You can do this in OpenCv. Probably it will only be available as CPU implementation. There are some conversions available in cuda with GPU, but never found how to get GpuMat in python (Mat are implemented as numpy arrays).
I am not so familiar with python api for opencv, but for converting you may try conversion code cv.COLOR_YUV2BGR_UYVY.

For adjusting your frame size and framerate, you may look at what formats/framerates V4L thinks your camera can provide with:

v4l2-ctl --list-formats-ext -d /dev/video0

Then you may be able to your prefered mode with command v4l2-ctl looking at the post mentioned above and restart your app to see if it has changed.

You may also do all the conversion with gstreamer. You may first try such pipeline for displaying in a X window:

gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw, width=640, height=480, framerate=30/1, format=UYVY' ! videoconvert ! xvimagesink

When it works you can experiment different modes. When you can see the expected mode, for using in opencv you would just change the end of the pipeline so that instead of displaying with xvimagesink, it is converted into BGR format and sent to appsink (your opencv application):

cap=cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw, width=640, height=480, framerate=30/1, format=UYVY ! videoconvert ! video/x-raw, format=BGR ! appsink')
5

Thank you for the precision !

When I use your gst-launch command I only obtain a fixed image…and this message on the terminal:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:4:7:1\,\ framerate\=\(fraction\)60/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:4:7:1\,\ framerate\=\(fraction\)60/1"
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:4:7:1\,\ framerate\=\(fraction\)60/1"
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:4:7:1\,\ framerate\=\(fraction\)60/1"
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:4:7:1\,\ framerate\=\(fraction\)60/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:4:7:1\,\ framerate\=\(fraction\)60/1"

I don’t really get it.

Then if I use the command of the guide:

gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw, format=(string)UYVY,
width=(int)2304, height=(int)1296" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)I420,
width=(int)1920, height=(int)1080" ! nvoverlaysink overlay-w=1920 overlay-h=1080 sync=false

I have an image that takes the whole screen and I can’t access anything, I have to “kill” the process to stop it…

6

nvoverlaysink overlays on the X windows, so it is normal it takes the whole screen if your monitor is running in 1080p.

So you may try other display sinks in X window (If these don’t work, you may try to add sync=false):
xvimagesink with format conversion in CPU with videoconvert:

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=(string)UYVY, width=(int)2304, height=(int)1296' ! videoconvert ! xvimagesink

xvimagesink with format conversion in ISP (rescaling to 1080p):

gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw, format=(string)UYVY, width=(int)2304, height=(int)1296" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw, format=I420" ! xvimagesink

EGL sink, it uses NVMM memory:

gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw, format=(string)UYVY, width=(int)2304, height=(int)1296" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080" ! nvegltransform ! nveglglessink

Once you get it running, you would modify the end of the pipeline for providing ‘video/x-raw, format=BGR’ to appsink in opencv videoCapture. For example, if the second example works, it would be:

v4l2src device=/dev/video0 ! video/x-raw, format=(string)UYVY, width=(int)2304, height=(int)1296 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)1920, height=(int)1080 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink
7

Okay awesome, thank you very much Honey.

8

A last question:
how do you know which mode the camera uses when you just apply “cv2.VideoCapture(0)” ??

9 
v4l2-ctl -d /dev/video0 --get-fmt-video

should show current mode.
Using v4l2-ctl --set-fmt-video=… you may be able to change mode. You may have to also use option --set-ctrl bypass_mode=0.

10

Here is the error log I get when I put the following command:

cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)1280, height=(int)960 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink')

VIDEOIO ERROR: V4L: device v4l2src device=/dev/video0 ! video/x-raw, format=(string)UYVY, width=1920, height=1080 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)BGRx, width=1280, height=960 ! nvvidconv ! video/x-raw, format=BGRx! appsink: Unable to query number of channels

(python:14047): GStreamer-CRITICAL **: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(python:14047): GStreamer-CRITICAL **: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python:14047): GStreamer-CRITICAL **: gst_structure_copy: assertion 'structure != NULL' failed

(python:14047): GStreamer-CRITICAL **: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(python:14047): GStreamer-CRITICAL **: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python:14047): GStreamer-CRITICAL **: gst_structure_copy: assertion 'structure != NULL' failed

(python:14047): GStreamer-CRITICAL **: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(python:14047): GStreamer-CRITICAL **: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(python:14047): GStreamer-CRITICAL **: gst_mini_object_ref: assertion 'mini_object != NULL' failed
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in cvCaptureFromCAM_GStreamer, file /home/nvidia/OpenCV3.4/opencv-3.4.0/modules/videoio/src/cap_gstreamer.cpp, line 890
VIDEOIO(cvCreateCapture_GStreamer (CV_CAP_GSTREAMER_FILE, filename)): raised OpenCV exception:

/home/nvidia/OpenCV3.4/opencv-3.4.0/modules/videoio/src/cap_gstreamer.cpp:890: error: (-2) GStreamer: unable to start pipeline
 in function cvCaptureFromCAM_GStreamer

OpenCV Error: Assertion failed (dims <= 2 && step[0] > 0) in locateROI, file /home/nvidia/OpenCV3.4/opencv-3.4.0/modules/core/src/matrix.cpp, line 991
Traceback (most recent call last):
  File "./test_kmeans.py", line 39, in <module>
    blurred = cv2.GaussianBlur(frame_normalized, (7, 7), 0)
cv2.error: /home/nvidia/OpenCV3.4/opencv-3.4.0/modules/core/src/matrix.cpp:991: error: (-215) dims <= 2 && step[0] > 0 in function locateROI

I don’t get why the error is about the “locateROI” function I don’t even use… I checked all the different packages I had to install for the camera and I have all of them…

11

Seems the gstreamer pipeline failed to start.
What shows ?

v4l2-ctl --list-formats-ext -d /dev/video0

You may try to set one of the listed modes either with v4l2-ctl (you may also install qv4l2 for checking available modes, adjust gain, …) if using videoCapture(0), either as specifying this into caps after v4l2src in a gstreamer pipeline.

The locateROI error is probably there because no frame was read, but you tried to process anyway a void frame. You may test the capture open and reads results or at least dimension of read frame in order to prevent this.

12

Here is the output.

ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2
		Size: Discrete 640x480
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1152x768
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x960
			Interval: Discrete 0.017s (58.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.022s (45.000 fps)
		Size: Discrete 1920x1280
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 2048x1536
			Interval: Discrete 0.034s (29.000 fps)
		Size: Discrete 2304x1296
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 2304x1536
			Interval: Discrete 0.042s (24.000 fps)

I also applied this command:

v4l2-ctl --set-fmt-video=width=1280,height=960,pixelformat=UYVY

and then the same with 640x480 to come back to normal but the error occurred already before and still after so it seems it hasn’t something to do with it.

I am still trying to figure out how I can set the default mode

13

Sorry if I’ve been confusing. V4L interface may be able to provide BGR for your camera, so do not worry for conversion for now.
I assume you want to get 1280x960. If you are using a recent version of opencv, these should work:

import cv2
print(cv2.__version__)
#print(cv2.getBuildInformation())

# 1st config: use gstreamer to use V4L interface (v4l2src) and provide BGR to opencv
cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw, format=BGR, width=1280, height=960 ! appsink')

# 2nd config: use V4L interface from opencv, and set capture properties for setting resolution.  
#cap = cv2.VideoCapture(0)
#cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
#cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 960)

# Check camera is opened
if (cap.isOpened() == False):
	print('Error: failed to open camera')
	quit()

# Create the window once now, otherwise imshow would recreate one for each frame
cv2.namedWindow('MyWindow')

# Display loop
while(True):
	ret, frame = cap.read();
	if (ret == False):
		print('Error: failed to read frame')
		quit()
	cv2.imshow('MyWindow', frame)
	if cv2.waitKey(1) & 0xFF == ord('q'):
		break

If none of these work, then probably your camera driver can’t provide V4L BGR pixel format. Then you may try to perform the conversion with gstreamer using one of these pipelines:

# 3rd config: use gstreamer to use V4L interface (v4l2src) to get frame in YUYV format and convert into BGR for opencv with videoconvert
cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw, format=YUYV, width=1280, height=960 ! videoconvert ! video/x-raw, format=BGR ! appsink')

# 4th config: use gstreamer to use V4L interface (v4l2src) to get frame in YUYV format and convert into BGRx with nvvidconv and finally into BGR with videoconvert
cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw, format=YUYV, width=1280, height=960 ! nvvidconv ! video/x-raw(memory:NVMM), format=BGRx ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink')

If you rather want to capture in YUV, you would have to disable BGR default conversion. Not sure it has an advantage to do so, but you would then do something like:

import cv2
print(cv2.__version__)
#print(cv2.getBuildInformation())

# 5th config: use gstreamer to use V4L interface (v4l2src) and provide UYVY to opencv
cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw, format=UYVY, width=1280, height=960 ! appsink')

# 6th config: use V4L interface from opencv, and set capture properties for setting resolution.  
#cap = cv2.VideoCapture(0)
#cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
#cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 960)

# Disable BGR default conversion
cap.set(cv2.CAP_PROP_CONVERT_RGB, False)

# Check camera is opened
if (cap.isOpened() == False):
	print('Error: failed to open camera')
	quit()

# Create the window once now, otherwise imshow would recreate one for each frame
cv2.namedWindow('MyWindow')

# Display loop
while(True):
	ret, frameUYVY = cap.read();
	if (ret == False):
		print('Error: failed to read frame')
		quit()
	frameBGR = cv2.cvtColor(frameUYVY, cv2.COLOR_YUV2BGR_UYVY)
	cv2.imshow('MyWindow', frameBGR)
	if cv2.waitKey(1) & 0xFF == ord('q'):
		break