nvdewarp odd behavior

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi, I just started playing around with the 360 fisheye dewarp plugin, nvdewarper. I haven’t been able to find much documentation or online resources around the plugin.

First off, it appears that using the plugin inverts the colorspace (RGB-BGR). Testing on the sample_cam6.mp4 360 parking garage video included in the Deepstream docker image, the red fire extinguisher turns blue. These are the gst commands with and without dewarping:

gst-launch-1.0 -v filesrc location=/data/sample_cam6.mp4 ! decodebin ! nvvidconv ! nvdewarper config-file=/data/dewarper.txt ! nvvidconv ! nveglglessink

gst-launch-1.0 -v filesrc location=/data/sample_cam6.mp4 ! decodebin ! nvvidconv ! nveglglessink

Second, I want to apply the dewarping to a single image collected. While this command works to display the image:

gst-launch-1.0 -v filesrc location=/data/fisheye_4000.png ! decodebin ! imagefreeze ! nvvidconv ! glimagesink

no matter what I try, I can’t get it to work with the nvdewarp included:

gst-launch-1.0 -v filesrc location=/data/fisheye_4000.png ! decodebin ! imagefreeze ! nvvidconv ! nvdewarper config-file=/data/dewarper.txt ! nvvidconv ! nveglglessink
 #DOES NOT WORK

Nor any attempt to just convert and save the single image after dewarping works.

Thanks in advance!

2

What’s the output error of this pipeline?

gst-launch-1.0 -v filesrc location=/data/fisheye_4000.png ! decodebin ! imagefreeze ! nvvidconv ! nvdewarper config-file=/data/dewarper.txt ! nvvidconv ! nveglglessink
3

First of all, I sorted out the RGB/BGR color conversion issue. By adding “video/x-raw(memory:NVMM), format=RGBA” it forced RGBA instead of BGRx which was the caps for nvvidconv by default. Working pipeline with good colors:

gst-launch-1.0 -v filesrc location=/data/sample_cam6.mkv ! decodebin ! nvvidconv ! "video/x-raw(memory:NVMM), format=RGBA" ! nvdewarper config-file=/data/dewarper.txt ! nveglglessink

Below is the -v output for the pipeline you mentioned:

gst-launch-1.0 -v filesrc location=/data/source_images/fisheye_3072_100.jpg ! decodebin ! imagefreeze ! nvvidconv ! nvdewarper config-file=/data/dewarper.txt ! nvvidconv ! nveglglessink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = "image/jpeg\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ sof-marker\=\(int\)0"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstJpegDec:jpegdec0.GstPad:sink: caps = "image/jpeg\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ sof-marker\=\(int\)0"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstJpegDec:jpegdec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstImageFreeze:imagefreeze0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)25/1\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)BGRx\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/Gstnvdewarper:nvdewarper0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ framerate\=\(fraction\)25/1\,\ width\=\(int\)1280\,\ height\=\(int\)3840"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv1.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ framerate\=\(fraction\)25/1\,\ width\=\(int\)1280\,\ height\=\(int\)3840"
/GstPipeline:pipeline0/GstEglGlesSink:eglglessink0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ framerate\=\(fraction\)25/1\,\ width\=\(int\)1280\,\ height\=\(int\)3840"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv1.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ framerate\=\(fraction\)25/1\,\ width\=\(int\)1280\,\ height\=\(int\)3840"
/GstPipeline:pipeline0/Gstnvdewarper:nvdewarper0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)BGRx\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)25/1\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0"
/GstPipeline:pipeline0/GstImageFreeze:imagefreeze0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstDecodePad:src_0.GstProxyPad:proxypad1: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)0/1"
nvvidconv0: NOT SUPPROTED CONVERSION ... Use videoconvert ... EXITING... 
ERROR: from element /GstPipeline:pipeline0/GstImageFreeze:imagefreeze0: Internal data stream error.
Additional debug info:
gstimagefreeze.c(851): gst_image_freeze_src_loop (): /GstPipeline:pipeline0/GstImageFreeze:imagefreeze0:
stream stopped, reason not-supported
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
/GstPipeline:pipeline0/GstEglGlesSink:eglglessink0.GstPad:sink: caps = "NULL"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv1.GstPad:src: caps = "NULL"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv1.GstPad:sink: caps = "NULL"
/GstPipeline:pipeline0/Gstnvdewarper:nvdewarper0.GstPad:src: caps = "NULL"
/GstPipeline:pipeline0/Gstnvdewarper:nvdewarper0.GstPad:sink: caps = "NULL"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv0.GstPad:src: caps = "NULL"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv0.GstPad:sink: caps = "NULL"
/GstPipeline:pipeline0/GstImageFreeze:imagefreeze0.GstPad:src: caps = "NULL"
/GstPipeline:pipeline0/GstImageFreeze:imagefreeze0.GstPad:sink: caps = "NULL"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstDecodePad:src_0.GstProxyPad:proxypad1: caps = "NULL"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstDecodePad:src_0: caps = "NULL"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstJpegDec:jpegdec0.GstPad:src: caps = "NULL"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstJpegDec:jpegdec0.GstPad:sink: caps = "NULL"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = "NULL"
Freeing pipeline ...

What I am trying right now is pinpoint exactly what it is that is that is throwing the error when trying to run the dewarper plugin on a single image. This pipeline below without nvdewarper works and shows the output before fakesink as exactly the same as the working video pipeline:

gst-launch-1.0 -v filesrc location=/data/source_images/fisheye_3072_100.jpg ! decodebin ! imagefreeze ! "video/x-raw, format=(string)I420, framerate=(fraction)20/1" ! nvvidconv ! "video/x-raw(memory:NVMM)" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA" ! fakesink

/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"

When now trying to add the dewarp plugin, I get this:

gst-launch-1.0 -v filesrc location=/data/source_images/fisheye_3072_100.jpg ! decodebin ! imagefreeze ! "video/x-raw, format=(string)I420, framerate=(fraction)20/1" ! nvvidconv ! "video/x-raw(memory:NVMM)" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA" ! nvdewarper config-file=/data/dewarper.txt ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = "image/jpeg\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ sof-marker\=\(int\)0"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstJpegDec:jpegdec0.GstPad:sink: caps = "image/jpeg\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ sof-marker\=\(int\)0"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstJpegDec:jpegdec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstImageFreeze:imagefreeze0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv1.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/Gstnvdewarper:nvdewarper0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ framerate\=\(fraction\)20/1\,\ width\=\(int\)1280\,\ height\=\(int\)3840"
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ framerate\=\(fraction\)20/1\,\ width\=\(int\)1280\,\ height\=\(int\)3840"
/GstPipeline:pipeline0/Gstnvdewarper:nvdewarper0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv1.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/Gstnvvidconv:nvvidconv0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)20/1"
/GstPipeline:pipeline0/GstImageFreeze:imagefreeze0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstDecodePad:src_0.GstProxyPad:proxypad1: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)3072\,\ height\=\(int\)3072\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)1:4:0:0\,\ framerate\=\(fraction\)0/1"
**
ERROR:gstnvdewarper.cpp:1406:GstFlowReturn gst_nvdewarper_transform(GstBaseTransform*, GstBuffer*, GstBuffer*): assertion failed: (frameinfo != NULL)
Aborted (core dumped)

Any help is appriciated.

Also very interested in more documentation and information regarding the dewarp plugin if possible, especially if it is being developed for the jetson platform and included in the jetson deepstream sdk. Thanks!

4

It appears that using the plugin inverts the colorspace (RGB-BGR).