How to convert YV12 to RGBA(NVMM) with gstreamer

s.n.3 · May 21, 2021, 9:14am

I’m investigating the v4l2 video capture with the gstreamer.
I can get the image with the following pipeline.

DISPLAY=:0.0 gst-launch-1.0 v4l2src device=/dev/video0 -e ! ‘video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080’ ! xvimagesink sync=false

But I’d like to process its image data as NVMM RGBA format data on appsink.
So I tried the following pipeline.

DISPLAY=:0.0 gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080’ ! videoconvert ! ‘video/x-raw, format=(string)I420, width=1920, height=1080’ ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=RGBA, widht=1920, height=1080’ ! nvoverlaysink sync=false -v

But it failed as follows.
What should I do ?

DISPLAY=:0.0 gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080’ ! videoconvert ! ‘video/x-raw, format=(string)I420, width=1920, height=1080’ ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=RGBA, widht=1920, height=1080’ ! nvoverlaysink sync=false -v
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, widht=(int)1920, framerate=(fraction)60000/1001, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, widht=(int)1920, framerate=(fraction)60000/1001, colorimetry=(string)bt709, interlace-mode=(string)progressive
Setting pipeline to PLAYING …
New clock: GstSystemClock
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, widht=(int)1920, framerate=(fraction)60000/1001, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, widht=(int)1920, framerate=(fraction)60000/1001, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, widht=(int)1920, framerate=(fraction)60000/1001, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, widht=(int)1920, framerate=(fraction)60000/1001, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, widht=(int)1920, framerate=(fraction)60000/1001, colorimetry=(string)bt709, interlace-mode=(string)progressive
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.660372288
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

DaneLLL · May 22, 2021, 12:23am

Hi,
Looks like frame rate is not set so launching the camera fails. Please get information of the camera:

$ v4l2-ctl --list-formats-ext

And set exact width,height,format,framerate.

s.n.3 · May 24, 2021, 6:23am

I check the supported formats and add the framerate for the caps.
But it’s not fixed as follows.

$ ./v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

    [0]: 'YV12' (Planar YVU 4:2:0)
            Size: Discrete 1920x1080
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
            Size: Discrete 3840x2160
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
            Size: Discrete 3840x1080
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
            Size: Discrete 5760x1080
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
            Size: Discrete 7680x1080
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
    [1]: 'YUYV' (YUYV 4:2:2)
            Size: Discrete 1920x1080
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
            Size: Discrete 3840x2160
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
            Size: Discrete 3840x1080
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
            Size: Discrete 5760x1080
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)
            Size: Discrete 7680x1080
                    Interval: Discrete 0.017s (59.940 fps)
                    Interval: Discrete 0.033s (29.970 fps)
                    Interval: Discrete 0.067s (14.985 fps)
                    Interval: Discrete 0.267s (3.746 fps)

$
$ DISPLAY=:0.0 gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, framerate=(fraction)60000/1001’ ! videoconvert ! ‘video/x-raw, format=(string)I420, width=1920, height=1080, framerate=(fraction)60000/1001’ ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=RGBA, widht=1920, height=1080, framerate=(fraction)60000/1001’ ! nvoverlaysink sync=false -v
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, framerate=(fraction)60000/1001, widht=(int)1920, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, framerate=(fraction)60000/1001, widht=(int)1920, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60000/1001, widht=(int)1920, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60000/1001, widht=(int)1920, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60000/1001, widht=(int)1920, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, framerate=(fraction)60000/1001, widht=(int)1920, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1080, framerate=(fraction)60000/1001, widht=(int)1920, colorimetry=(string)bt709, interlace-mode=(string)progressive
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.605938176
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

Honey_Patouceul · May 24, 2021, 4:23pm

It may be easier with YUYV 4:2:2:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=1920, height=1080,framerate=60000/1001 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvoverlaysink

s.n.3 · May 27, 2021, 12:44am

Thank you for the information.
I confirmed to be able to capture the image by using the YUYV 4:2:2 format.
But the performance of our capture device is not good in case of using the YUYV 4:2:2 format.
So I’d like to use the YV12 format.
It would be helpful if somebody could get some advice for the YV12 format.

Honey_Patouceul · May 27, 2021, 8:20pm

May you extend on this ? Is it a performance issue ? For quality it should be better in most cases, isn’t it ?

s.n.3 · May 31, 2021, 8:30am

The priority of the frame rate is highest in my project.
So I’d like to use the YV12 format and NVMM for the post processing in the appsink.

DaneLLL · June 9, 2021, 8:32am

Hi,
Please check if you can run

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YV12,width=1920, height=1080,framerate=60000/1001 ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v