Gst-launch-1.0 decode jpeg image when using nvjpegdec or jpegdec, but does not work with decodebin

javiermatos · May 29, 2020, 7:53pm

According to documentation offered by NVIDIA regarding nvjpegdec, we can show an image calling the following command:

gst-launch-1.0 filesrc location=<filename.jpg> ! nvjpegdec ! imagefreeze ! xvimagesink -e

Also, the same command will work if replacing nvjpegdec for the generic jpegdec.

What is the problem? The problem is that if I have a PNG file, I need to replace the decoder and use pngdec. This is not convenient, so because of that there exist decodebin, that will choose the correct decoder based on image type. In my computer, decodebin works as expected (choose jpegdec for jpeg or pngdec for png). In my jetson nano I can use pngdec, jpegdec and nvjpegdec, but if I try to use decodebin as a generic replacement, it will fail.

$ startx /usr/bin/gst-launch-1.0 -vvv filesrc location=/home/ubuntu/resources/1920x1080/red.jpg ! decodebin ! imagefreeze ! xvimagesink


X.Org X Server 1.19.6
Release Date: 2017-12-20
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-168-generic aarch64 Ubuntu
Current Operating System: Linux jetson-nano 4.9.140-tegra #1 SMP PREEMPT Wed Apr 8 18:10:49 PDT 2020 aarch64
Kernel command line: tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 tegra_fbmem=0x800000@0x92cb4000 is_hdmi_initialised=1  root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0    root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 quiet
Build Date: 14 November 2019  06:20:13PM
xorg-server 2:1.19.6-1ubuntu4.4 (For technical support please see http://www.ubuntu.com/support) 
Current version of pixman: 0.34.0
	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Fri May 29 21:52:04 2020
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = image/jpeg, width=(int)1920, height=(int)1080, sof-marker=(int)2
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstNvJpegDec:nvjpegdec0.GstPad:sink: caps = image/jpeg, width=(int)1920, height=(int)1080, sof-marker=(int)2
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstNvJpegDec:nvjpegdec0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind: Internal data stream error.
Additional debug info:
gsttypefindelement.c(1236): gst_type_find_element_loop (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind:
streaming stopped, reason error (-5)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...
xinit: connection to X server lost

waiting for X server to shut down (II) Server terminated successfully (0). Closing log file.

Couldn't get a file descriptor referring to the console

I think this is because there is an error in nvjpegdec regarding its type find funcionality which will make decodebin to fail.

Honey_Patouceul · May 29, 2020, 8:18pm

Not sure why, but it seems with decodebin it tries output in NVMM memory that imagefreeze cannot use.

You may try to specifiy caps in CPU memory after decodebin:

gst-launch-1.0 filesrc location=<filename.jpg> ! decodebin ! video/x-raw ! imagefreeze ! xvimagesink -e

javiermatos · May 29, 2020, 9:24pm

Thank you for replying my question. Unfortunately specifying capabilities as you shown in the command does not make a difference. I am getting an extra line after the src pad capabilities of the nvjpegdec:

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = image/jpeg, width=(int)1920, height=(int)1080, sof-marker=(int)2
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstNvJpegDec:nvjpegdec0.GstPad:sink: caps = image/jpeg, width=(int)1920, height=(int)1080, sof-marker=(int)2
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstNvJpegDec:nvjpegdec0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1: caps = video/x-raw
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind: Internal data stream error.
Additional debug info:
gsttypefindelement.c(1236): gst_type_find_element_loop (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind:
streaming stopped, reason error (-5)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

DaneLLL · June 1, 2020, 2:16am

Hi,
Please run

$ export GST_DEBUG=*FACTORY*:4

To get which plugin is picked for decoding.

javiermatos · June 6, 2020, 11:37am

Hi Dane,

Thank you so much for sharing that command to show that information regarding created elements. As I can see, decodebin is choosing nvjpegdec. The problem seems to be related with the typefind element according to the output:

ubuntu@jetsonnano:~$ startx /usr/bin/gst-launch-1.0 -v filesrc location=/home/ubuntu/player/resources/1920x1080/red.jpg ! decodebin ! imagefreeze ! videoconvert ! xvimagesink


X.Org X Server 1.19.6
Release Date: 2017-12-20
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-168-generic aarch64 Ubuntu
Current Operating System: Linux jetsonnano 4.9.140-tegra #1 SMP PREEMPT Wed Apr 8 18:10:49 PDT 2020 aarch64
Kernel command line: tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 tegra_fbmem=0x800000@0x92cb4000 is_hdmi_initialised=1  root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0    root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 quiet
Build Date: 14 November 2019  06:20:13PM
xorg-server 2:1.19.6-1ubuntu4.4 (For technical support please see http://www.ubuntu.com/support) 
Current version of pixman: 0.34.0
	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Sat Jun  6 13:34:10 2020
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
0:00:00.049733784 13674   0x5585cab900 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "filesrc"
0:00:00.053114814 13674   0x5585cab900 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "decodebin"
0:00:00.053447631 13674   0x5585cab900 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "typefind" named "typefind"
0:00:00.054380872 13674   0x5585cab900 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "imagefreeze"
0:00:00.055010931 13674   0x5585cab900 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "videoconvert"
0:00:00.057379502 13674   0x5585cab900 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "xvimagesink"
0:00:00.057862268 13674   0x5585cab900 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "pipeline"
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = image/jpeg, width=(int)1920, height=(int)1080, sof-marker=(int)2
0:00:00.140504943 13674   0x5585cc28f0 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "nvjpegdec"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstNvJpegDec:nvjpegdec0.GstPad:sink: caps = image/jpeg, width=(int)1920, height=(int)1080, sof-marker=(int)2
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstNvJpegDec:nvjpegdec0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind: Internal data stream error.
Additional debug info:
gsttypefindelement.c(1236): gst_type_find_element_loop (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind:
streaming stopped, reason error (-5)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...
xinit: connection to X server lost

waiting for X server to shut down (II) Server terminated successfully (0). Closing log file.

Couldn't get a file descriptor referring to the console

Honey_Patouceul · June 6, 2020, 1:31pm

Indeed, decodebin chooses nvjpegdec even if the output is required to be in cpu memory.
nvjpegdec is able to do that, but it seems there is something wrong with decodebin and in this case it ends up with NVMM memory instead.

So you have two very different paths for jpg (requiring a nvvidconv after decodebin) or png, as pngdec only outputs RGB formats for color, but imagefreeze doesn’t supports this as input, so you would need a converter such as videconvert.
I think it may be easier to handle both cases by generating a different pipeline for each case.

javiermatos · June 6, 2020, 2:11pm

Hi,

Yeah, I was trying to do a super portable image slider using C + GStreamer, that’s why I was using decodebin (which is part of playbin, that is the element that I prefer to use). But given the different hardware devices that I am using (mainly Raspberry Pi 4 and Jetson Nano), it would be better to create a custom version for each device.

Thank you so much!

DaneLLL · June 8, 2020, 1:11am

Hi,
By defuat, nvjpegdec is with higher rank than jpegdec:

  if (!gst_element_register (plugin, "nvjpegdec", GST_RANK_PRIMARY + 15,
          GST_TYPE_JPEG_DEC))
    return FALSE;

You may download the source of gst-jpeg and set lower the rank to let *decodebin * pick jpegdec. Source code is in

https://developer.nvidia.com/embedded/L4T/r32_Release_v4.2/Sources/T210/public_sources.tbz2

javiermatos · June 10, 2020, 3:07pm

Hi,
I can change default ranking from C by doing the following:

static void disable_plugin_feature(gchar *name) {
    GstRegistry* reg = gst_registry_get();
    GstPluginFeature* plugin_feature = gst_registry_lookup_feature(reg, name);
    if (plugin_feature != NULL) {
        gst_plugin_feature_set_rank (plugin_feature, GST_RANK_NONE);
        gst_object_unref (plugin_feature);
    }
}

And then just use that function to disable nvjpegdec → disable_plugin_feature("nvjpegdec");

My use case is just to show an image for a while (between 30 and 60 seconds), so using nvjpegdec or jpegdec will not make a difference.

Thank you so much!