Demux surfaces from nvdewarper

[razvan.ilascu]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU - RTX2060 with Docker
• DeepStream Version
5.1
• JetPack Version (valid for Jetson only)
n/a
• TensorRT Version
7.2.3-1+cuda11.1
• NVIDIA GPU Driver Version (valid for GPU only)
455.32.00
• Issue Type( questions, new requirements, bugs)
question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

---

I would like to create a pipeline using nvdewarper:

gst-launch-1.0 --gst-debug=3 \
filesrc location=/data/FISHEYE_cut.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/data/config_dewarper.txt nvbuf-memory-type=3 ! mux.sink_0 \
nvstreammux name=mux batch-size=2 num-surfaces-per-frame=2 width=1920 height=1080 ! nvstreamdemux name=demux \
demux.src_0 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=/data/output_0.mp4 \
demux.src_1 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=/data/output_1.mp4

But the pipeline fails to run with core dump:

[…]
Redistribute latency…
0:00:00.673620131 642 0x559fe907fc50 WARN v4l2bufferpool gstv4l2bufferpool.c:1066:gst_v4l2_buffer_pool_start:nvv4l2h264enc1:pool:src Uncertain or not enough buffers, enabling copy threshold
Redistribute latency…
0:00:00.682295752 642 0x559fe907fc50 WARN v4l2bufferpool gstv4l2bufferpool.c:1066:gst_v4l2_buffer_pool_start:nvv4l2h264enc0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:00.682759482 642 0x559fe907fc50 WARN v4l2videodec gstv4l2videodec.c:1673:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency
0:00:00.682789497 642 0x559fe907fc50 WARN v4l2bufferpool gstv4l2bufferpool.c:1066:gst_v4l2_buffer_pool_start:nvv4l2decoder0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:00.707336912 642 0x7efe744e8a30 WARN v4l2bufferpool gstv4l2bufferpool.c:1513:gst_v4l2_buffer_pool_dqbuf:nvv4l2decoder0:pool:src Driver should never set v4l2_buffer.field to ANY
0:00:00.814979219 642 0x7efe740230a0 ERROR nvvideoconvert gstnvvideoconvert.c:3387:gst_nvvideoconvert_transform: buffer transform failed
0:00:00.815424763 642 0x7efe740230a0 ERROR nvvideoconvert gstnvvideoconvert.c:3387:gst_nvvideoconvert_transform: buffer transform failed
0:00:00.816408151 642 0x7efe740230a0 ERROR nvvideoconvert gstnvvideoconvert.c:3387:gst_nvvideoconvert_transform: buffer transform failed
0:00:00.816441894 642 0x559fe907fc50 WARN qtdemux qtdemux.c:6073:gst_qtdemux_loop: error: Internal data stream error.
0:00:00.816482802 642 0x559fe907fc50 WARN qtdemux qtdemux.c:6073:gst_qtdemux_loop: error: streaming stopped, reason error (-5)
0:00:00.816955769 642 0x7efe740230a0 FIXME basesink gstbasesink.c:3145:gst_base_sink_default_event: stream-start event without group-id. Consider implementing group-id handling in the upstream elements
ERROR: from element /GstPipeline:pipeline0/GstQTDemux:qtdemux0: Internal data stream error.
Additional debug info:
qtdemux.c(6073): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstQTDemux:qtdemux0:
streaming stopped, reason error (-5)
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to NULL …
free(): invalid next size (fast)
Aborted (core dumped)

When using a single dewarped surface num-surfaces-per-frame=1 the output is properly generated.

gst-launch-1.0 --gst-debug=3 \
filesrc location=/data/FISHEYE_cut.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/data/config_dewarper1.txt nvbuf-memory-type=3 ! mux.sink_0 \
nvstreammux name=mux batch-size=2 num-surfaces-per-frame=1 width=1920 height=1080 ! nvstreamdemux name=demux \
demux.src_0 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=/data/output_single.mp4

The file /data/config_dewarper.txt has the following content:

[property]
output-width=4096
output-height=1300
num-batch-buffers=2

[surface0]
projection-type=1
surface-index=0
width=1024
height=512
top-angle=15
bottom-angle=-60
pitch=90
yaw=0
roll=-110
focal-length=512
src-fov=180

[surface1]
projection-type=1
surface-index=1
width=1024
height=512
top-angle=15
bottom-angle=-60
pitch=90
yaw=0
roll=-110
focal-length=512
src-fov=180

and /data/config_dewarper1.txt has one of the surfaces removed and num-batch-buffers=1

Any clues on how to make this work? Thanks!

[Fiona.Chen]

The nvstreamdemux does not support “num-surfaces-per-frame” larger than 1 case currently.

You can use nvmultistreamtiler to combine the 2 surfaces together before encoding.

gst-launch-1.0 -e filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=config_dewarper_perspective_2.txt nvbuf-memory-type=3 ! m.sink_0 nvstreammux name=m width=3680 height=2428 batch-size=2 num-surfaces-per-frame=2 ! nvmultistreamtiler width=2480 height=1920 columns=2 rows=1 ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12’ ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=/data/dewarp.mp4

[razvan.ilascu]

Thank you for the response! I tried the pipeline you proposed, and the quality is good using 2 cameras. However, when I increased the number of surfaces to 4, the output video quality decreased significantly, even though I doubled the height in nvmultistreamtiler. I attached an image showing the decrease in quality (hoping the image isn’t too compressed by the file upload).
Is there a way to keep a high output quality when using 4 surfaces per source, or more than 1 source?

[Fiona.Chen]

There are many places which will impact video quality. You need to check your original camera resolution, surface resolution, nvstreammux resolution to set proper parameters to avoid scaling as much as possible.

And you can also set better encoding parameters of nvv4l2h264enc such as “bitrate” to improve encoding quality.

It depends on your case. There is no common settings.

[razvan.ilascu]

Thank you! Setting the bitrate for nvv4l2h264enc to 32,000,000 increased the output quality the most, according to some trial and error experiments.

Returning to the initial question, is there any plan to support multiple surfaces for nvstreamdemux in a future deepstream release?

[Fiona.Chen]

We have add the feature in our roadmap. It may be implemented in the future.