How can I capture images from V4L2 camera with gstreamer?

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
,
1

Hello,

We use Orin and Camera as shown below:

The camera document said I can capture jpg image with the gst-launch command as shown below:

gst-launch-1.0 v4l2src device=/dev/video0 numbuffers=1 ! “video/x-raw, format=(string)UYVY,
width=(int)3840, height=(int)2160” ! jpegenc ! filesink
location=filename.jpg

And, the question I’ve got is

  • How can I capture RGB or BGR images with nvidia’s gstreamer?
  • With these RGB/BGR images we will make cv::Mat and apply many opencv operations.

I know I need to use appsink, callback, nvvidconv and opencv VideoCapture class.

I’d like to find some easy and short example code for this.
Thank you very much for your kindness in advance…
Thank you.

F.Y.I.

ubuntu:~$ v4l2-ctl -d /dev/video0 --all
Driver Info:
Driver name : tegra-video
Card type : vi-output, ar0230 31-0044
Bus info : platform:tegra-capture-vi:3
Driver version : 5.10.104
Capabilities : 0x84200001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Media Driver Info:
Driver name : tegra-camrtc-ca
Model : NVIDIA Tegra Video Input Device
Serial :
Bus info :
Media version : 5.10.104
Hardware revision: 0x00000003 (3)
Driver version : 5.10.104
Interface Info:
ID : 0x03000017
Type : V4L Video
Entity Info:
ID : 0x00000015 (21)
Name : vi-output, ar0230 31-0044
Function : V4L2 I/O
Pad 0x01000016 : 0: Sink
Link 0x0200001b: from remote pad 0x100000c of entity ‘13e40000.host1x:nvcsi@15a00000-’: Data, Enabled
Priority: 2
Video input : 0 (Camera 3: no power)
Format Video Capture:
Width/Height : 1920/1080
Pixel Format : ‘UYVY’ (UYVY 4:2:2)
Field : None
Bytes per Line : 3840
Size Image : 4147200
Colorspace : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Limited Range)
Flags :
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0

3

Looks like your need YUV to RGB/BGR convert.

4

Nop.
I need to learn how to capture RGB or BGR format video frames via gstreamer pipeline.
source : v4l2src
sink : appsink
plugin(?) : nvvidconv
I think I need VideoCapture class in opencv.
The url you pointed only uses opencv. This is not what I need…

5

Format conversion with opencv may not be the best solution, you may get better performance with gstreamer.

For using v4l2src, you may try to run a gstreamer pipeline to display for checking such as:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,width=3840,height=2160,framerate=30/1 ! nvvidconv ! autovideosink

Maybe for your case this might also work;

gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM),format=UYVY,width=3840,height=2160,framerate=30/1' ! nvvidconv ! autovideosink

Then the conversion to BGR for opencv may be done such as:

# Using CPU-based videoconvert
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,width=3840,height=2160,framerate=30/1 ! videoconvert ! video/x-raw,format=BGR ! fakesink -v

# Using HW VIC-based nvvidconv for conversion into BGRx and CPU-based for removing 4th byte into BGR format:
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,width=3840,height=2160,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, format=BGR ! fakesink -v

# Same with nvv4l2camerasrc would be better as saving first copy from system memory into NVMM
gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM),format=UYVY,width=3840,height=2160,framerate=30/1' ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, format=BGR ! fakesink -v

Choose the one performing best and just replace fakesink with appsink drop=1 and remove single quotes into gstreamer pipeline for opencv appsink such as:

cv::VideoCapture cap("nvv4l2camerasrc device=/dev/video0 ! video/x-raw(memory:NVMM),format=UYVY, width=3840,height=2160,framerate=30/1 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1", cv::CAP_GSTREAMER);
// Then check for cap.isOpened()

Searching this forum you would find many samples about how to read and display the frames with opencv.

1 Like
6

Great!
Thank you very much!
I will carefully check your sample commands and code snippet.
^^
While checking out now… this explanation is perfect! Thank you!

Minor sort of things I had to change F.Y.I. :

%s/video-raw/video\/x-raw/g
%s/3840/1920/g
%s/2160/1080/g

The camera I use provide FHD.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.