The GStreamer 1.0 plugins that support NVMM, provided by NVIDIA, have very wonky coupling.
First of all, I have a video file that is mjpeg in mkv. This is what FFmpeg 3.3.1 says about my file:
ubuntu@tegra-ubuntu:~$ ffmpeg -i output2.mkv
ffmpeg version 3.3.1 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 4.8 (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3)
configuration: --prefix=/home/ubuntu/ffmpeg_build --extra-cflags=-I/home/ubuntu/ffmpeg_build/include --extra-ldflags=-L/home/ubuntu/ffmpeg_build/lib --bindir=/home/ubuntu/bin --enable-gpl --enable-libx264 --enable-avresample --enable-nonfree --enable-shared
libavutil 55. 58.100 / 55. 58.100
libavcodec 57. 89.100 / 57. 89.100
libavformat 57. 71.100 / 57. 71.100
libavdevice 57. 6.100 / 57. 6.100
libavfilter 6. 82.100 / 6. 82.100
libavresample 3. 5. 0 / 3. 5. 0
libswscale 4. 6.100 / 4. 6.100
libswresample 2. 7.100 / 2. 7.100
libpostproc 54. 5.100 / 54. 5.100
Input #0, matroska,webm, from 'output2.mkv':
Metadata:
ENCODER : Lavf56.15.102
Duration: 00:00:46.17, start: 0.000000, bitrate: 152355 kb/s
Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown), 1920x1080, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
At least one output file must be specified
As you can see, it’s mjpeg, in yuvj422p pixel format, bt407bg colourspace, 1920x1080, 30 fps.
I’m going to list 5 examples that involve nvjpegdec and nvvidconv:
Case 1: filesrc location=output2.mkv ! nvjpegdec ! ‘video/x-raw(memory:NVMM)’ ! nvhdmioverlaysink
ubuntu@tegra-ubuntu:~$ gst-launch-1.0 filesrc location=output2.mkv ! nvjpegdec ! nvhdmioverlaysink -v
Setting pipeline to PAUSED ...
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and MjstreamingPipeline is PREROLLING ...
/GstPipeline:pipeline0/GstNvJpegDec:nvjpegdec0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstNvHDMIOverlaySink-nvhdmioverlaysink:nvhdmioverlaysink-nvhdmioverlaysink0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:04.133629995
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Note that nvhdmioverlaysink accepts video/x-raw(memory:NVMM), format=(string)I420 here.
Result: video displayed via HDMI overlay on LCD screen. However, chroma is incorrectly expanded in the vertical direction (doubled in height), while luma remains correct.
Case 2: filesrc location=output2.mkv ! nvjpegdec ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=I420’ ! nvhdmioverlaysink
ubuntu@tegra-ubuntu:~$ gst-launch-1.0 filesrc location=output2.mkv ! nvjpegdec ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! nvhdmioverlaysink -v
Setting pipeline to PAUSED ...
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and MjstreamingPipeline is PREROLLING ...
/GstPipeline:pipeline0/GstNvJpegDec:nvjpegdec0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)I420
/GstPipeline:pipeline0/GstNvHDMIOverlaySink-nvhdmioverlaysink:nvhdmioverlaysink-nvhdmioverlaysink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)I420
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:03.995583364
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Note that both the src and sink pads of nvvidconv are negotiated as video/x-raw(memory:NVMM), format=(string)I420, so the conversion should a no-op.
Result: the overlay is completely green.
Case 3: filesrc location=output2.mkv ! nvjpegdec ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=UYVY’ ! nvhdmioverlaysink
ubuntu@tegra-ubuntu:~$ gst-launch-1.0 filesrc location=output2.mkv ! nvjpegdec ! nvvidconv ! 'video/x-raw(memory:NVMM), format=UYVY' ! nvhdmioverlaysink -v
Setting pipeline to PAUSED ...
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and MjstreamingPipeline is PREROLLING ...
/GstPipeline:pipeline0/GstNvJpegDec:nvjpegdec0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)UYVY
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)UYVY
/GstPipeline:pipeline0/GstNvHDMIOverlaySink-nvhdmioverlaysink:nvhdmioverlaysink-nvhdmioverlaysink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)UYVY
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)UYVY
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:10.847353605
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Note that nvhdmioverlaysink claims to accept video/x-raw(memory:NVMM), format=(string)UYVY here.
Result: the overlay is completely black.
Case 4: filesrc location=output2.mkv ! nvjpegdec ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=GRAY8’ ! nvhdmioverlaysink
ubuntu@tegra-ubuntu:~$ gst-launch-1.0 filesrc location=output2.mkv ! nvjpegdec ! nvvidconv ! 'video/x-raw(memory:NVMM), format=GRAY8' ! nvhdmioverlaysink -v
Setting pipeline to PAUSED ...
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and MjstreamingPipeline is PREROLLING ...
/GstPipeline:pipeline0/GstNvJpegDec:nvjpegdec0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)GRAY8
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)GRAY8
/GstPipeline:pipeline0/GstNvHDMIOverlaySink-nvhdmioverlaysink:nvhdmioverlaysink-nvhdmioverlaysink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)GRAY8
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)GRAY8
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:03.511661359
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Note that between nvvidconv and nvhdmioverlaysink, the caps are negotiated as video/x-raw(memory:NVMM), format=(string)GRAY8, so I expect the resulting rendered video to be in grayscale.
Result: video rendered in correct colours, not in grayscale, via HDMI overlay on LCD screen (WHAT!?)
Case 5: filesrc location=output2.mkv ! nvjpegdec ! ‘video/x-raw(memory:NVMM)’ ! nvvidconv ! xvimagesink
ubuntu@tegra-ubuntu:~$ gst-launch-1.0 filesrc location=output2.mkv ! nvjpegdec ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! xvimagesink -v
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstNvJpegDec:nvjpegdec0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)UYVY
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)0/1, format=(string)UYVY
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)I420, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:0:0, framerate=(fraction)0/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Output window was closed
Additional debug info:
xvimagesink.c(551): gst_xvimagesink_handle_xevents (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0
Execution ended after 0:00:08.186343916
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Note that I now try to convert from I420 in NVMM to UYVY in regular RAM, so that I can display the video in standard sinks like xvimagesink.
Result: video rendered in a window in correct colours, but the whole image is slightly stretched vertically, with the bottom cropped.