[Jetson Orin] Fisheye dewarp aliasing issue

[OZD]

Hi NVIDIA Community,

I’m currently using NVDEWARPER on a Jetson Orin to create panoramic images from fisheye rtsp input (uniview camera), but I’m encountering significant image quality issues and aliasing compared to the results I achieve with ffmpeg. Below are the details of my setup and the commands I’m using.

ffmpeg Command (from image):

ffmpeg -i input.jpg -vf v360=fisheye:e:ih_fov=180:iv_fov=180:pitch=90 -y output.png

The ffmpeg command yields high-quality dewarped images with minimal aliasing. However, when I attempt a similar process with GStreamer and NVDEWARPER, the results are subpar.

GStreamer Command:

export GST_PLUGIN_PATH=/opt/nvidia/deepstream/deepstream-6.2/lib
gst-launch-1.0 rtspsrc location='rtsp://xxx' ! \
    rtph265depay ! \
    h265parse ! \
    nvv4l2decoder ! \
    nvvideoconvert ! "video/x-raw(memory:NVMM),width=3000,height=3000" ! \
    videorate ! "video/x-raw(memory:NVMM),framerate=5/1" ! \
    nvdewarper config-file=/home/node/storage/fish/1.txt ! \
    m.sink_0 nvstreammux name=m width=6000 height=3000 batch-size=1 num-surfaces-per-frame=1 ! \
    nvmultistreamtiler ! \
    nvvideoconvert ! "video/x-raw(memory:NVMM),width=6000,height=3000" ! \
    nvv4l2h265enc ! \
    h265parse ! \
    mp4mux ! \
    filesink location=/dev/shm/output.mp4 -e

Configuration File (1.txt):

# 1=PushBroom,              2=VertRadCyl,               3=Perspective_Perspective,
# 4=FISH_PERSPECTIVE,       5=FISH_FISH,                6=FISH_CYL,
# 7=FISH_EQUIRECT,          8=FISH_PANINI,              9=PERSPECTIVE_EQUIRECT,
# 10=PERSPECTIVE_PANINI,    11=EQUIRECT_CYLINDER,       12=EQUIRECT_EQUIRECT,
# 13=EQUIRECT_FISHEYE,      14=EQUIRECT_PANINI,         15=EQUIRECT_PERSPECTIVE,
# 16=EQUIRECT_PUSHBROOM,    17=EQUIRECT_STEREOGRAPHIC,  18=EQUIRECT_VERTCYLINDER

[property]
output-width=6000
output-height=3000
num-batch-buffers=1

[surface0]
projection-type=7
surface-index=0
width=6000
height=3000
top-angle=90
bottom-angle=-90
pitch=90
yaw=0
roll=0
src-fov=180

Issue:
The dewarped images produced by GStreamer with NVDEWARPER exhibit poor image quality and noticeable aliasing, unlike the smooth results from ffmpeg.

Example Images:
See attached image from gst-nvdewarper vs ffmpeg dewarp
dewarp_images.zip (11.8 MB)

Questions:

1. How can I improve the image quality and reduce aliasing in NVDEWARPER?
2. Are there specific parameters or configurations I should adjust to match the quality achieved by ffmpeg?

Any suggestions or insights would be greatly appreciated!

Thank you,
Oz

[Fiona.Chen]

What is the original resolution of the video from the rtsp stream?

[OZD]

The original rtsp stream is fisheye with resolution of 3000x3000.

[Fiona.Chen]

Can you provide the original image or video?

[OZD]

Attached the original video.

[Fiona.Chen]

They are two different algorithms, why do you compare them?

I’ve tried the two methods with our sample video /opt/nvidia/deepstream/deepstream/samples/streams/sample_office.mp4:

1. The ffmpeg dewarper
   ffmpeg -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_office.mp4 -vf v360=fisheye:e:ih_fov=180:iv_fov=180:pitch=90 -y ffmpeg_output.mp4
2. The DeepStream dewarper:
   gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_office.mp4 ! qtdemux name=demux demux.video_0 ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=./config_dewarper.txt ! m.sink_0 nvstreammux name=m width=1728 height=1080 batch-size=1 num-surfaces-per-frame=1 ! nvmultistreamtiler columns=1 rows=1 width=1728 height=1080 ! nvvideoconvert ! nvv4l2h264enc bitrate=8000000 ! h264parse ! mux.video_0 qtmux name=mux ! filesink location=dewarp.mp4
   config_dewarper.txt (201 Bytes)

[Fiona.Chen]

The ffmpeg output video and DeepStream output video

[Fiona.Chen]

I don’t see any video quality issue with the test result.

[OZD]

It is working well now with the dewarper command you sent, I missed the width and height arguments to nvmultistreamtiler.
Thanks!

[OZD]

Hi,
I would like to be able to change the following parameters dynamically: top-angle, bottom-angle, pitch, yaw. Modifying the config file at runtime doesn’t give the desired result, because it is parsed at the beginning.
How can I achieve that?

[Fiona.Chen]

The nvdewarper plugin is open source. You can customize as you like.

/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdewarper

[OZD]

Hi,
I am trying to set two surfaces into separate tiles, here is my example:

export GST_PLUGIN_PATH=/opt/nvidia/deepstream/deepstream-6.2/lib
gst-launch-1.0 rtspsrc location='rtsp://username:password@ipaddress:port/rtsp' ! \
        rtph265depay ! \
        h265parse ! \
        nvv4l2decoder ! \
        nvvideoconvert ! "video/x-raw(memory:NVMM),width=3000,height=3000" ! \
        nvdewarper config-file=/home/node/storage/fish/tile.cfg ! \
        m.sink_0 nvstreammux name=m width=1920 height=1080 batch-size=2 num-surfaces-per-frame=2 !\
        nvmultistreamtiler columns=2 rows=1 width=1920 height=1080 ! \
        nvvideoconvert ! \
        nvv4l2h265enc bitrate=8000000 ! \
        h265parse ! \
        mux.video_0 qtmux name=mux ! \
        filesink location=/dev/shm/output.mp4 -e

Here is my config:

[property]
output-width=1920
output-height=1080
num-batch-buffers=2

[surface0]
projection-type=7
surface-index=0
width=1920
height=1080
top-angle=22.5
bottom-angle=-22.5
pitch=15.0
yaw=15.0
roll=0
src-fov=180

[surface1]
projection-type=7
surface-index=1
width=1920
height=1080
top-angle=22.5
bottom-angle=-22.5
pitch=25.0
yaw=25.0
roll=0
src-fov=180

This is the image i am getting:

How can I put a surface per tile?
Thanks

[Fiona.Chen]

Please raise new topic for new issue/question