The output video file appears fast forward when the pipeline is under heavy load

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Jetson Orin NX 16G
• DeepStream Version
DS 7.0
• JetPack Version (valid for Jetson only)
JetPack 6.0 GA (L4T 36.3)
• Issue Type( questions, new requirements, bugs)
questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

An example that works as expected:

gst-launch-1.0 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_0 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_1 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_2 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_3 \
nvstreammux name=streammux batch-size=4 sync-inputs=1 max-latency=10000000000000000000 ! nvinfer name=nvinfer config-file-path="/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt" batch-size=4 interval=0 unique-id=1 ! nvdslogger ! nvvideoconvert ! capsfilter caps="video/x-raw(memory:NVMM), format=(string)RGBA" ! nvmultistreamtiler gpu-id=0 nvbuf-memory-type=0 rows=2 columns=2 width=1920 height=1080 qos=0 ! nvdsosd process-mode=0 display-clock=1 display-text=1 display-bbox=1 display-mask=1 ! nvvideoconvert ! capsfilter caps="video/x-raw(memory:NVMM), format=(string)I420" ! nvv4l2h264enc insert-sps-pps=true iframeinterval=5 control-rate=1 preset-level=1 maxperf-enable=false bitrate=4000000 ! h264parse config-interval=5 ! hlssink2 playlist-root="hls" location="out/ch%05d.ts" playlist-location="out/playlist.m3u8" playlist-length=12 target-duration=10 max-files=10

An example of the question:

gst-launch-1.0 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_0 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_1 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_2 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_3 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_4 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_5 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_6 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_7 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_8 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_9 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_10 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_11 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_12 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_13 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_14 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_15 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_16 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_17 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_18 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_19 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_20 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_21 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_22 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_23 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_24 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_25 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_26 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_27 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_28 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_29 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_30 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_31 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_32 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_33 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_34 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_35 \
nvstreammux name=streammux batch-size=36 sync-inputs=1 max-latency=10000000000000000000 ! nvinfer name=nvinfer config-file-path="/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt" batch-size=36 interval=0 unique-id=1 ! nvdslogger ! nvvideoconvert ! capsfilter caps="video/x-raw(memory:NVMM), format=(string)RGBA" ! nvmultistreamtiler gpu-id=0 nvbuf-memory-type=0 rows=6 columns=6 width=1920 height=1080 qos=0 ! nvdsosd process-mode=0 display-clock=1 display-text=1 display-bbox=1 display-mask=1 ! nvvideoconvert ! capsfilter caps="video/x-raw(memory:NVMM), format=(string)I420" ! nvv4l2h264enc insert-sps-pps=true iframeinterval=5 control-rate=1 preset-level=1 maxperf-enable=false bitrate=4000000 ! h264parse config-interval=5 ! hlssink2 playlist-root="hls" location="out/ch%05d.ts" playlist-location="out/playlist.m3u8" playlist-length=12 target-duration=10 max-files=10

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

We are considering a DeepStream application that can infer any number of multiple videos, arrange them in tiles, and then output the results in HLS format so that users can check them. Let’s say the frame rate of the input video is 30 FPS. For example, if we input 4 videos and infer them, the resulting video is displayed as expected.

**PERF : FPS_0 (30.00) FPS_1 (30.00)   FPS_2 (29.99)   FPS_3 (29.99)

As an extreme example, if we input 36 videos and perform batch inference, the hardware processing resources will naturally not be able to keep up and all the videos cannot be processed at 30 FPS (that is totally no problem). In my environment, it was about 10 FPS.

**PERF : FPS_0 (10.20)  FPS_1 (10.20)   FPS_2 (10.20)   FPS_3 (10.20)   FPS_4 (10.20)   FPS_5 (10.20)   FPS_6 (10.20)   FPS_7 (10.20)   FPS_8 (10.20)   FPS_9 (10.20)      FPS_10 (10.20)  FPS_11 (10.20)  FPS_12 (10.20)  FPS_13 (10.20)  FPS_14 (10.20)  FPS_15 (10.20)  FPS_16 (10.20)  FPS_17 (10.20)  FPS_18 (10.20)  FPS_19 (10.20)     FPS_20 (10.20)  FPS_21 (10.20)  FPS_22 (10.20)  FPS_23 (10.20)  FPS_24 (10.20)  FPS_25 (10.20)  FPS_26 (10.20)  FPS_27 (10.20)  FPS_28 (10.20)  FPS_29 (10.20)     FPS_30 (10.20)  FPS_31 (10.20)  FPS_32 (10.20)  FPS_33 (10.20)  FPS_34 (10.20)  FPS_35 (10.20)

The problem is, when I downloaded the output video and checked it, the video looked like it was fast-forwarded. (Note how the clock in the upper left counts up.)

I think this is because the video output at about 10FPS is output as a file at a fixed frame rate of 30FPS, making it 3 times faster. Instead, is there a good way to adjust the output video to the correct speed based on the current actual FPS value measured during pipeline execution? We were able to change the playback speed to a predetermined value using videorate etc. However, since we don’t know what the actual FPS will be until I start the pipeline, we would like to dynamically adjust the playback speed during pipeline playback rather than deciding on a fixed value. If you could share a good method, I would appreciate it. Attached are the commands used to run the above two example pipelines.

Thank you very much in advance.

2 Likes
3

The Jetson Orin NX 16G does not support 36 1080p@30fps H264 videos decoding. Jetson Modules, Support, Ecosystem, and Lineup | NVIDIA Developer

4

@Fiona.Chen From the question above I don’t think the number of inputs really matter here. We had the same issue with even 6 inputs. It really depends on the AI model performance.
The question is not about the hardware video encoding / decoding support capabilities but pipeline behavior in case there is a frame rate (~10 FPS) drop (after nvinfer due to the AI model processing time) below the input (nvurisrcbin elements) frame rate (~30 FPS).
In this case the output cannot not be processed at the frame rate as the inputs (which is fine in this use case) but is trying to “catch up” to keep some higher frame rate (fast forwarding effect explained here and also buffering issues for what I could check).

5

Please set “sync” property of hlssink2 as “false” in your pipeline. The video frames are hard to reach to sink in time, many frames will be dropped by sink.

6

@Fiona.Chen There is no such “sync” property for the hlssink2 Gst plugin as for what we can check with gst-inspect-1.0

7

“sync” property is for the base class GstBaseSink. GstBaseSink (gstreamer.freedesktop.org). You can set it.

8

We’ve tried to set the "sync "value in hlssink2, but we couldn’t. It is because the hlssink2 element is not based on the base class GstBaseSink.
https://gstreamer.freedesktop.org/documentation/hls/hlssink2.html?gi-language=c#hierarchy

(cf. FileSink is based on GstBaseSink
filesink)

We could also find an example from NVIDIA to output the result to mp4 file.
We tried to use the same method in our pipeline (filesink, sync=0) and observed the same behavior.
I will attach the code to reproduce this.

gst-launch-1.0 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_0 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_1 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_2 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_3 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_4 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_5 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_6 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_7 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_8 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_9 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_10 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_11 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_12 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_13 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_14 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_15 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_16 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_17 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_18 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_19 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_20 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_21 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_22 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_23 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_24 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_25 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_26 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_27 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_28 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_29 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_30 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_31 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_32 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_33 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_34 \
nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_35 \
nvstreammux name=streammux batch-size=36 sync-inputs=1 max-latency=10000000000000000000 ! nvinfer name=nvinfer config-file-path="/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt" batch-size=36 interval=0 unique-id=1 ! nvdslogger ! nvvideoconvert ! capsfilter caps="video/x-raw(memory:NVMM), format=(string)RGBA" ! nvmultistreamtiler gpu-id=0 nvbuf-memory-type=0 rows=6 columns=6 width=1920 height=1080 qos=0 ! nvdsosd process-mode=0 display-clock=1 display-text=1 display-bbox=1 display-mask=1 ! queue ! nvvideoconvert ! capsfilter caps="video/x-raw, format=I420" ! avenc_mpeg4 bitrate=2000000 ! mpeg4videoparse ! qtmux ! filesink location=out.mp4 sync=0 async=0

Also please find the result here.

9

I tried with my Orin NX 16G board, the pipeline crash when the input source numner reach to 32. The pipeline is out of the hardware’s capability.

10

Adapting the gst-launch pipeline structure above for testing with only 6 inputs, to stay within the device hardware capabilities (also using Orin NX 16Go in my case), we can confirm the same issue occurs.

11

It is not reproduced in our board.

gst-launch-1.0 nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_0 nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_1 nvstreammux name=streammux batch-size=7 sync-inputs=1 max-latency=10000000000000000000 ! nvinfer name=nvinfer config-file-path="/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt" batch-size=30 interval=0 unique-id=1 ! nvdslogger ! nvvideoconvert ! capsfilter caps="video/x-raw(memory:NVMM), format=(string)RGBA" ! nvmultistreamtiler gpu-id=0 nvbuf-memory-type=0 rows=3 columns=3 width=1920 height=1080 qos=0 ! nvdsosd process-mode=0 display-clock=1 display-text=1 display-bbox=1 display-mask=1 ! queue ! nvvideoconvert ! capsfilter caps="video/x-raw, format=I420" ! avenc_mpeg4 bitrate=2000000 ! mpeg4videoparse ! qtmux ! filesink location=out.mp4 nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_2 nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_3 nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_4 nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_5 nvurisrcbin uri="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" gpu-id=0 cudadec-memtype=0 ! nvvideoconvert ! streammux.sink_6
12

@Fiona.Chen Could you document what is the FPS rate you get after the nvinfer server element on your test here ?
If it is similar to your inputs frame rate you won’t be able to reproduce this issue.
The problem occurs when the inference processing takes time and therefore make the FPS drop at a level that is significantly different from the input frame rate (as mentioned here).

13

I’m using the same pipeline you post here. For local video files sources, DeepStream will not change the GstBuffer timestamps. So the output video are just the same rate as the input sources.

14

OK so I understand there is currently no way using the existing DeepStream elements to re-sync the frames passed to the HLS element if there is any frame rate dropping after the nvinfer processing.
If the presentation timestamp (PTS) is too much delayed it will be required to readjust the buffer clock to create the HLS chunks on time preventing too much buffering delay issues.

15

Are you talking about the live streams input case? If so, the nvstreammux should be configured correctly. DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

16

Have you tried to setup your board’s max power and clock as Performance — DeepStream documentation?

17

The box is already at MAX-N performance. So nothing we can do here.
Regarding your previous question, as shown in the instructions shared, the pipeline is using files for this test.
Currently one remediation (that we need to confirm) would be to forcibly re-align the PTS for the HLS output sync. But it requires some custom probe or new Gst element development.