Deepstream pipeline stuck when trying to encode nvmultistreamtiler output

sanatmharolkar · May 11, 2021, 7:48am

Setup:

Xavier AGX
Jetpack Version 4.5
Deepstream Version 5.0.1

Hi,

My project requires multiple camera sources which need to be composited, encoded and then saved/streamed. For this, I realised I can use nvstreammux and nvmultistreamtiler. To test the pipeline, I tried the following:

gst-launch-1.0 nvstreammux name=nvmux batch-size=3 width=1920 height=1080 live-source=true \
nvv4l2camerasrc device=/dev/video0 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvmux.sink_0 \
nvv4l2camerasrc device=/dev/video1 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvmux.sink_1 \
nvv4l2camerasrc device=/dev/video2 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvmux.sink_2 \
nvmux.src ! nvmultistreamtiler columns=3 rows=1 width=5760 height=1080 ! nvvideoconvert ! nvv4l2h265enc maxperf-enable=true ! h265parse ! qtmux ! filesink location=test.mp4 -e

This does not generate any playable streams. The pipeline gets stuck with no error messages and when I try to end it using Ctrl+C., the following message is shown and the pipeline remains stuck until I kill the process.

^Chandling interrupt.
Interrupt: Stopping pipeline ...
EOS on shutdown enabled -- Forcing EOS on the pipeline

I tried the same idea with just one camera using the following pipeline:

gst-launch-1.0 nvstreammux name=nvmux batch-size=1 width=1920 height=1080 live-source=true \
nvv4l2camerasrc device=/dev/video0 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvmux.sink_0 \
nvmux.src ! nvmultistreamtiler columns=1 rows=1 width=1920 height=1080 ! nvvideoconvert ! nvv4l2h265enc maxperf-enable=true ! h265parse ! qtmux ! filesink location=test.mp4 async=false -e

But this also gives me the same problem. No error message, the pipeline is just stuck.

If I try to avoid the nvstreammux and nvmultistreamtiler completely, and test out a simple pipeline for saving one video, it works. This is the pipeline that works:

gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvv4l2h265enc maxperf-enable=true ! h265parse ! qtmux ! filesink location=test.mp4 -e

I am not able to figure out why the nvmultistreamtiler output cannot be encoded and saved. I would appreciate any help.

Fiona.Chen · May 11, 2021, 8:54am

gst-launch-1.0 tool can not send EOS when you kill the app by “Ctrl+C”. Please either just save raw hevc stream file (it can be broken in any place) or write a gstreamer application to end the pipeline with sending EOS( you can refer to deepstream-test5 sample).

sanatmharolkar · May 11, 2021, 9:20am

I tried as you suggested. Even though the app exits cleanly, the stream is still stuck without error when it is set to PLAYING.

Normally, in the pipeline without nvstreammux and nvmultistreamtiler, when I set GST_DEBUG=4, the following output message is constantly printed to the terminal when the stream is PLAYING:

FIXME h265parse gsth265parse.c:1798:gst_h265_parse_parse_frame:<h265parser> Implement timestamp/duration interpolation based on SEI message

This does not happen when I use nvstreammux and nvmultistreamtiler. The output is just stuck and when I exit the app, the file has no playable streams.

Fiona.Chen · May 11, 2021, 9:32am

Please refer to deepstream-test5 sample

sanatmharolkar · May 19, 2021, 8:09am

Hi,

I’m sorry, but that did not help with my problem. I tried this simple pipeline to better explain my issue:

gst-launch-1.0 nvstreammux name=nvmux batch-size=3 width=1920 height=1080 live-source=true \
nvv4l2camerasrc device=/dev/video2 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvmux.sink_0 \
nvv4l2camerasrc device=/dev/video0 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvmux.sink_1 \
nvv4l2camerasrc device=/dev/video1 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvmux.sink_2 \
nvmux.src ! nvmultistreamtiler columns=3 rows=1 width=5760 height=1080 ! nvvideoconvert ! nvv4l2h265enc maxperf-enable=true bufapi-version=true ! h265parse ! queue ! h265parse ! nvv4l2decoder ! nv3dsink window-height=360 window-width=1920 -e

I see only the first frame on the nv3dsink display, and then the whole pipeline is stuck. There are no warning or error messages even when I do GST_DEBUG=4. The pipeline is getting stuck at the encoder stage.

I would appreciate it if you could help me debug this problem.

Fiona.Chen · May 19, 2021, 8:13am

What kind of camera do you use? CSI or USB?

sanatmharolkar · May 19, 2021, 8:18am

CSI. I am using the e-Con Systems NileCam30 GMSL cameras.

If I create the same pipeline for just one camera, without the nvstreammux and nvmultistreamtiler, the pipeline works properly:

gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvv4l2h265enc maxperf-enable=true bufapi-version=true ! h265parse ! queue ! h265parse ! nvv4l2decoder ! nv3dsink window-height=720 window-width=1280 -e

Fiona.Chen · May 19, 2021, 8:21am

Does the nvstreammux pipeline work with only one camera?

sanatmharolkar · May 19, 2021, 8:24am

No. I tried this:

gst-launch-1.0 nvstreammux name=nvmux batch-size=1 width=1920 height=1080 live-source=true nvv4l2camerasrc device=/dev/video0 bufapi-version=true ! "video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1" ! nvmux.sink_0 nvmux.src ! nvmultistreamtiler columns=1 rows=1 width=1920 height=1080 ! nvvideoconvert ! nvv4l2h265enc maxperf-enable=true bufapi-version=true ! h265parse ! queue ! h265parse ! nvv4l2decoder ! nv3dsink window-height=720 window-width=1280 -e

Just added the nvstreammux with batch-size=1 and nvmultistreamtiler with columns=1 and rows=1, and I get the same problem. The pipeline is stuck after the first frame.

Fiona.Chen · May 19, 2021, 8:31am

Why do you encode and decode the video before display?

It is better to convert the format to “video/x-raw(memory:NVMM), format=I420” before encoding. And please add queue before encoder and after encoder.

sanatmharolkar · May 19, 2021, 8:42am

My main goal is to encode and stream it over RTP. That is what I tried first, but did not receive any RTP packets on my receiver side (I checked using Wireshark). So then I tried to see if I can at least view the stream locally first to confirm that the pipeline is stuck at the encoding side.

Okay so I was using nvvideoconvert before the encoder, but I tried specifying I420 format and adding queues as you suggested, and that seems to have solved the issue. The stream is now playing.

Thanks for your help!

Fiona.Chen · May 19, 2021, 8:47am

If you have read the deepstream-test5 code carefully, you will find the solution is already there.