Stalling with nvvideoconvert, nvstreammux, nvdsosd and nvstreamdemux

[Krokosh]

I am trying to use the nvdsosd plugin to annotate live video, both to display live using nvmultistreamtiler and to record video to disc.

I have found that I can manage up to three channels on the Xavier NX but that if I increase this to four, the pipeline fails to start. I am using a four-channel capture card based on the TP2855 to capture up to four inputs at 1080p. The device nodes are numbered 2-5 because of two additional capture devices that are not currently in use. I have found that the following command will give me three channels:

gst-launch-1.0 nvv4l2camerasrc device=/dev/video2 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_0 nvv4l2camerasrc device=/dev/video3 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_1 nvv4l2camerasrc device=/dev/video4 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_2 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvdsosd display-clock=1 ! tee name=t t.src_0 ! nvmultistreamtiler ! queue ! autovideosink sync=0 t.src_1 ! queue ! nvstreamdemux name=d d.src_0 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan0.ts d.src_1 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan1.ts d.src_2 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan2.ts

However, if I use the following command, I see a single frame from one channel on the live preview and all of the files created are zero-length:

gst-launch-1.0 nvv4l2camerasrc device=/dev/video2 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_0 nvv4l2camerasrc device=/dev/video3 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_1 nvv4l2camerasrc device=/dev/video4 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_2 nvv4l2camerasrc device=/dev/video5 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_3 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvdsosd display-clock=1 ! tee name=t t.src_0 ! nvmultistreamtiler ! queue ! autovideosink sync=0 t.src_1 ! queue ! nvstreamdemux name=d d.src_0 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan0.ts d.src_1 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan1.ts d.src_2 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan2.ts d.src_3 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan3.ts

However, if I remove the recording section, I get live video from all 4 channels:

gst-launch-1.0 nvv4l2camerasrc device=/dev/video2 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_0 nvv4l2camerasrc device=/dev/video3 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_1 nvv4l2camerasrc device=/dev/video4 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_2 nvv4l2camerasrc device=/dev/video5 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_3 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvdsosd display-clock=1 ! tee name=t t.src_0 ! nvmultistreamtiler ! queue ! autovideosink sync=0

If I remove the live preview, the recordings are still zero length:

gst-launch-1.0 nvv4l2camerasrc device=/dev/video2 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_0 nvv4l2camerasrc device=/dev/video3 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_1 nvv4l2camerasrc device=/dev/video4 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_2 nvv4l2camerasrc device=/dev/video5 bufapi-version=1 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! nvvideoconvert ! m.sink_3 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvdsosd display-clock=1 ! tee name=t t.src_0 ! queue ! nvstreamdemux name=d d.src_0 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan0.ts d.src_1 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan1.ts d.src_2 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan2.ts d.src_3 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan3.ts

I am using deepstream 6.0 on JetPack 4.6.1.

[yuweiw]

Could you try to use the filesrc instead of nvv4l2camerasrc to exclude the issue of nvv4l2camerasrc?

[Krokosh]

I can repeat the problem with videotestsrc. If I run:

gst-launch-1.0 videotestsrc ! ‘video/x-raw,width=1920,height=1080’ ! nvvideoconvert ! m.sink_0 videotestsrc ! ‘video/x-raw,width=1920,height=1080’ ! nvvideoconvert ! m.sink_1 videotestsrc ! ‘video/x-raw,width=1920,height=1080’ ! nvvideoconvert ! m.sink_2 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvdsosd display-clock=1 ! tee name=t t.src_0 ! nvmultistreamtiler ! queue ! autovideosink sync=0 t.src_1 ! queue ! nvstreamdemux name=d d.src_0 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan0.ts d.src_1 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan1.ts d.src_2 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan2.ts

I can see three sets of colour bars with moving noise noise in the bottom right hand corner of each picture and the recordings increase in size. However, if I run:

gst-launch-1.0 videotestsrc ! ‘video/x-raw,width=1920,height=1080’ ! nvvideoconvert ! m.sink_0 videotestsrc ! ‘video/x-raw,width=1920,height=1080’ ! nvvideoconvert ! m.sink_1 videotestsrc ! ‘video/x-raw,width=1920,height=1080’ ! nvvideoconvert ! m.sink_2 videotestsrc ! ‘video/x-raw,width=1920,height=1080’ ! nvvideoconvert ! m.sink_3 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvdsosd display-clock=1 ! tee name=t t.src_0 ! nvmultistreamtiler ! queue ! autovideosink sync=0 t.src_1 ! queue ! nvstreamdemux name=d d.src_0 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan0.ts d.src_1 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan1.ts d.src_2 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan2.ts d.src_3 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan3.ts

I see one set of colour bars (either in one quadrant of the screen, half of the screen or full-screen; it appears to be random) and the white noise is not moving. Again, all recordings are zero length.

[yuweiw]

I try the pipeline on my platform with DeepStream 6.3. It works well. Since you use the Xaviar, could you update the DeepStream to the latest version?

gst-launch-1.0 videotestsrc ! 'video/x-raw,width=1920,height=1080' ! nvvideoconvert ! m.sink_0 videotestsrc \
! 'video/x-raw,width=1920,height=1080' ! nvvideoconvert ! m.sink_1 videotestsrc ! 'video/x-raw,width=1920,height=1080' \
! nvvideoconvert ! m.sink_2 videotestsrc ! 'video/x-raw,width=1920,height=1080' \
! nvvideoconvert ! m.sink_3 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvdsosd display-clock=1 \
! queue ! nvstreamdemux name=d d.src_0 \
! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan0.ts d.src_1 ! nvvideoconvert ! nvv4l2h264enc \
! mpegtsmux ! filesink location=chan1.ts d.src_2 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux \
! filesink location=chan2.ts d.src_3 ! nvvideoconvert ! nvv4l2h264enc ! mpegtsmux ! filesink location=chan3.ts

[Krokosh]

Thank you for checking; unfortunately, I can’t update the JetPack version above 4.6 as our carrier board and the capture card do not yet support 5.x. Is there a way to use DeepStream 6.3 with JetPack 4.6?

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

No. There is a strict correlation between the DeepStream version and the JetPack version. https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Quickstart.html#id5