Gstreamer pipeline stop

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): Jetson
• DeepStream Version: DS7.1
• JetPack Version (valid for Jetson only): Jetpack 6.1
• TensorRT Version: 10.3.0
• Issue Type( questions, new requirements, bugs): bugs

I ran the following GStreamer pipeline based on the deepstream-test5-app application, but the pipeline stopped running after 7 minites.

How can I get the following pipeline to work?

Only the ps command and the tegrastats command were running in the background.

GStreamer logs are as follows.

Pipeline
The pipeline uses the new nvstreammux.

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! \
    tee ! \
    nvstreamdemux name=demux \
    demux.src_0 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_1 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_2 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_3 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_4 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_5 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_6 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_7 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_8 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_9 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_10 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_11 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_12 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_13 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_14 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_15 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0

Docker container

How to reproduce

Here is gst.sh.txt (5.3 KB).

xhost +
sudo docker run -it --runtime=nvidia -v ${PWD}:${PWD} -w ${PWD} --device=/dev/video0:/dev/video0 -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=${DISPLAY} nvcr.io/nvidia/deepstream:7.1-samples-multiarch
bash ./gst.sh
3
  1. what do you mean by “stop”? did all output videos freeze?
  2. to rule out the source 's issue, will the following cmd pipelne hang?
export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! fakesink
  1. if the cmd in step1 run well, to rule out the issue of inference and tracker, will the following cmd run hang?
export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! fakesink
4
  1. what do you mean by “stop”? did all output videos freeze?

“stop” means hang.
Yes. All output videos froze.

  1. to rule out the source 's issue, will the following cmd pipelne hang?

No, the pipeline did not hang.
The cmd ran well.
GStreamer log is as follows.
gstreamer.log (864.4 KB)

  1. if the cmd in step1 run well, to rule out the issue of inference and tracker, will the following cmd run hang?

No, the pipeline did not hang.
The cmd ran well.
GStreamer log is as follows.
gstreamer.log (992.2 KB)

5
  1. what is the Jetson device model? please refer to this faq to set Jetson clocks high for a better performance. From the result in your last comment, using “… tracker->fakesink” still works well, what do you need to demux each source? why not user tiler to show the output video in one window?
  2. to check if the issue is related to demux, will the following cmd include one demux hang?
export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! \
    nvstreamdemux name=demux \
    demux.src_0 ! nv3dsink sync=0 enable-last-sample=0

3 . If the cmd in the step2 works well, if you continue to add more demux src, will the cmd hang? Thanks!

6
  1. what is the Jetson device model? please refer to this faq to set Jetson clocks high for a better performance.

After following the FAQ and executing the command below, I ran the command again, but the issue still occurred.
The FAQ method does not resolve this problem.

sudo nvpmodel -m 0 --for MAX perf and power
sudo jetson_clocks

From the result in your last comment, using “… tracker->fakesink” still works well, what do you need to demux each source? why not user tiler to show the output video in one window?

After inference, I need to perform specific processing for each source, so nvstreamdemux is required.

  1. to check if the issue is related to demux, will the following cmd include one demux hang?

No, the pipeline did not hang.
The cmd ran well.
GStreamer log is as follows.
gstreamer.log (992.1 KB)

3 . If the cmd in the step2 works well, if you continue to add more demux src, will the cmd hang? Thanks!

I added more demux src and ran the following pipeline.
However, immediately after running the pipeline, the timer remained at 0, and the screen output froze right after execution.
It appears the pipeline has stopped due to a different issue from the one I initially posted.
Even if you increase the number of sources to three or four, it stops in the same way.

GStreamer log is as follows.
gstreamer.log (33.4 KB)

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! \
    nvstreamdemux name=demux \
    demux.src_0 ! nv3dsink sync=0 enable-last-sample=0 \
    demux.src_1 ! nv3dsink sync=0 enable-last-sample=0
7

Based on the pipeline in your last comment, please add async=0 for each sink, as written in the doc. Testing with two demux src, the cmd run well on Orin with DS7.1. Here is my test log-1128.txt (13.5 KB).

8

3 . If the cmd in the step2 works well, if you continue to add more demux src, will the cmd hang? Thanks!

I added 16 demux src and ran the pipeline,
but the pipeline did not hang.
gstreamer.log (2.2 MB)

The difference from the pipeline in the initial post is that it lacks queue and tee, and sync=0 is specified.
Does this mean we need to add sync=0?

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! \
    nvstreamdemux name=demux \
    demux.src_0 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_1 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_2 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_3 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_4 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_5 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_6 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_7 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_8 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_9 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_10 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_11 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_12 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_13 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_14 ! nv3dsink sync=0 enable-last-sample=0 async=0 \
    demux.src_15 ! nv3dsink sync=0 enable-last-sample=0 async=0
9
  1. Based on the pipeline in your last comment, using “sync=1”, the cmd still works well.Here is loglog-1205.txt (14.6 KB). can you get the same result?
  2. can the original hang issue be reproduced every time? if so, could you provide the detailed log 1.log after running the following cmd. you can compress the log wth zip.

export NVDSINFER_LOG_LEVEL=3 && export USE_NEW_NVSTREAMMUX=yes && gst-launch-1.0 --gst-debug-level=3 --gst-debug=nvstreammux:6,nvinfer:6,nvstreamdemux:6 v4l2src… >1.log 2>1.log

  1. to rule out the source’s issue, can this hang issue be reproduced with test soruce? Below is a videotestsrc sample.
    gst-launch-1.0 -v videotestsrc ! video/x-raw,format=RGBA,width=1280,height=720,framerate=5/1 ! fakesink
10

Based on the pipeline in your last comment, using “sync=1”, the cmd still works well.Here is loglog-1205.txt (14.6 KB). can you get the same result?

When I ran the pipeline with fakesink and sync=1, I got the same results as you.
However, when I ran the pipeline with nv3dsink and sync=1, it hung.

fakesink and sync=1
GStreamer log is as follows.
gstreamer.log (2.3 MB)

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! \
    nvstreamdemux name=demux \
    demux.src_0 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_1 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_2 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_3 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_4 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_5 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_6 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_7 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_8 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_9 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_10 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_11 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_12 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_13 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_14 ! fakesink sync=1 enable-last-sample=0 async=0 \
    demux.src_15 ! fakesink sync=1 enable-last-sample=0 async=0

nv3dsink and sync=1
GStreamer log is as follows.
gstreamer.log (3.3 MB)

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! \
    nvstreamdemux name=demux \
    demux.src_0 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_1 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_2 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_3 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_4 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_5 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_6 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_7 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_8 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_9 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_10 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_11 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_12 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_13 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_14 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_15 ! nv3dsink sync=1 enable-last-sample=0 async=0

can the original hang issue be reproduced every time? if so, could you provide the detailed log 1.log after running the following cmd. you can compress the log wth zip.

Yes. The hang issue reproduces every time.
I executed the command and retrieved the log.
The following is the log.
1.log.zip (22.1 MB)

to rule out the source’s issue, can this hang issue be reproduced with test soruce? Below is a videotestsrc sample.

I used videotestsrc, and the hang issue can be reproduced.
GStreamer log is as follows.
gstreamer.log (168.8 KB)

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 -v \
    videotestsrc ! "video/x-raw, format=RGBA, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    queue ! nvvideoconvert ! nvinfer config-file-path="/opt/nvidia/deepstream/deepstream-7.1/samples/configs/deepstream-app/config_infer_primary.txt" model-engine-file="/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b30_gpu0_int8.engine" ! \
    nvtracker tracker-width=1280 tracker-height=720 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ! \
    tee ! \
    nvstreamdemux name=demux \
    demux.src_0 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_1 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_2 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_3 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_4 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_5 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_6 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_7 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_8 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_9 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_10 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_11 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_12 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_13 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_14 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0 \
    demux.src_15 ! queue ! tee ! queue ! nv3dsink async=0 enable-last-sample=0
11
  1. Testing with the last pipeline in your last comment, the issue can’t be reproduced on my side. the app run well more than 2 hours. Here is the log log-1210.txt (126.5 KB).
  2. To narrow down this issue, based on the last pipeline in your last comment, if removing nvtracker, can the issue be reproduced? if continue to removing nvinfer, can the issue be reproduced?
  3. if the issue persists after using a simplified cmd, could you share the 2.log after running the following cmd using the simplified pipeline? Thanks! you can end the cmd after the timestamp does not change.
export NVDSINFER_LOG_LEVEL=3 && export USE_NEW_NVSTREAMMUX=yes && gst-launch-1.0 --gst-debug-level=3 --gst-debug=nvstreammux:6,nvinfer:6,nvstreamdemux:6,nvvideoconvert:6    videotestsrc … >2.log 2>2.log
12
  1. To narrow down this issue, based on the last pipeline in your last comment, if removing nvtracker, can the issue be reproduced? if continue to removing nvinfer, can the issue be reproduced?

I removed both nvtracker and nvinfer, but the issue still reproduces.
GStreamer log is as follows.
gstreamer.log (706.3 KB)

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    nvstreamdemux name=demux \
    demux.src_0 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_1 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_2 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_3 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_4 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_5 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_6 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_7 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_8 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_9 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_10 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_11 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_12 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_13 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_14 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_15 ! nv3dsink sync=1 enable-last-sample=0 async=0
  1. if the issue persists after using a simplified cmd, could you share the 2.log after running the following cmd using the simplified pipeline? Thanks! you can end the cmd after the timestamp does not change.

The logs were retrieved using the following command.
Since the file size is large, it has been split using the following command.

split -n l/4 2.log 2.log_

2.log_aa.zip (92.3 MB)
2.log_ab.zip (87.8 MB)
2.log_ac.zip (87.8 MB)
2.log_ad.zip (87.8 MB)

export NVDSINFER_LOG_LEVEL=3 && export USE_NEW_NVSTREAMMUX=yes && \
gst-launch-1.0 --gst-debug-level=3 --gst-debug=nvstreammux:6,nvinfer:6,nvstreamdemux:6,nvvideoconvert:6 \
    v4l2src device=/dev/video0 ! "video/x-raw, width=1280, height=720" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    nvstreamdemux name=demux \
    demux.src_0 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_1 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_2 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_3 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_4 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_5 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_6 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_7 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_8 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_9 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_10 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_11 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_12 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_13 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_14 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_15 ! nv3dsink sync=1 enable-last-sample=0 async=0 >2.log 2>2.log
13

handling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:33:20.112665344

  1. Thanks for the sharing! After checking the log 2.log, as the log above shown, the app run for more than 33 minutes. did the app run well(the timestamp printing was still changing or the output video still refreshed) when it was ended? seems the log does include any abnormal printing.
  2. for now, will only " with nv3dsink and sync=1" hang? if so, could you share the 3.log after running the following cmd? Thanks! you can end the cmd after the timestamp does not change or the output video does not refresh. If the log is too large, please only provide the first and last files after splitting.
export NVDSINFER_LOG_LEVEL=3 && export USE_NEW_NVSTREAMMUX=yes && \
gst-launch-1.0 --gst-debug-level=3 --gst-debug=nvstreammux:6,nvstreamdemux:6,nvvideoconvert:6,nv3dsink:6 \
    videotestsrc ! "video/x-raw, format=RGBA, width=1280, height=720, framerate=5/1"  ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), format=NV12, width=320, height=180" ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    nvstreamdemux name=demux \
    demux.src_0 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_1 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_2 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_3 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_4 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_5 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_6 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_7 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_8 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_9 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_10 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_11 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_12 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_13 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_14 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_15 ! nv3dsink sync=1 enable-last-sample=0 async=0 >3.log 2>3.log
  1. you can simplify
    nvvideoconvert src-crop=“0:0:320:180” copy-hw=2 ! “video/x-raw(memory:NVMM), format=NV12, width=320, height=180” ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080” to
    nvvideoconvert src-crop=“0:0:320:180” copy-hw=2 ! “video/x-raw(memory:NVMM), width=1920, height=1080”
  2. additionnally, what is the device model?
14
  1. Thanks for the sharing! After checking the log 2.log, as the log above shown, the app run for more than 33 minutes. did the app run well(the timestamp printing was still changing or the output video still refreshed) when it was ended? seems the log does include any abnormal printing.

I was unable to verify the timestamp because I was outputting the log to a file.
I confirmed that the video output had stopped.

  1. for now, will only " with nv3dsink and sync=1" hang? if so, could you share the 3.log after running the following cmd? Thanks! you can end the cmd after the timestamp does not change or the output video does not refresh. If the log is too large, please only provide the first and last files after splitting.

Since the file size is large, it has been split using the following command.

split -n l/5 3.log 3.log_

First file: 3.log_aa.zip (85.0 MB)
Last file: 3.log_ae.zip (81.5 MB)

  1. you can simplify
    nvvideoconvert src-crop=“0:0:320:180” copy-hw=2 ! “video/x-raw(memory:NVMM), format=NV12, width=320, height=180” ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080” to
    nvvideoconvert src-crop=“0:0:320:180” copy-hw=2 ! “video/x-raw(memory:NVMM), width=1920, height=1080”

The following error occurs.

/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:4786: => VIC Configuration failed image scale factor exceeds 16, use GPU for Transformation
0:00:00.353653728   105 0xffff50000b70 ERROR         nvvideoconvert gstnvvideoconvert.c:4255:gst_nvvideoconvert_transform: buffer transform failed
ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason error (-5)

command

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 \
    videotestsrc ! "video/x-raw, format=RGBA, width=1280, height=720, framerate=5/1" ! tee name=t \
    t.src_0 ! nvvideoconvert src-crop="0:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_0 \
    t.src_1 ! nvvideoconvert src-crop="320:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_1 \
    t.src_2 ! nvvideoconvert src-crop="640:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_2 \
    t.src_3 ! nvvideoconvert src-crop="960:0:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_3 \
    t.src_4 ! nvvideoconvert src-crop="0:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_4 \
    t.src_5 ! nvvideoconvert src-crop="320:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_5 \
    t.src_6 ! nvvideoconvert src-crop="640:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_6 \
    t.src_7 ! nvvideoconvert src-crop="960:180:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_7 \
    t.src_8 ! nvvideoconvert src-crop="0:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_8 \
    t.src_9 ! nvvideoconvert src-crop="320:360:320:18" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_9 \
    t.src_10 ! nvvideoconvert src-crop="640:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_10 \
    t.src_11 ! nvvideoconvert src-crop="960:360:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_11 \
    t.src_12 ! nvvideoconvert src-crop="0:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_12 \
    t.src_13 ! nvvideoconvert src-crop="320:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_13 \
    t.src_14 ! nvvideoconvert src-crop="640:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_14 \
    t.src_15 ! nvvideoconvert src-crop="960:540:320:180" copy-hw=2 ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! mux.sink_15 \
    nvstreammux name=mux batch-size=16 ! \
    nvstreamdemux name=demux \
    demux.src_0 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_1 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_2 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_3 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_4 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_5 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_6 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_7 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_8 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_9 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_10 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_11 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_12 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_13 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_14 ! nv3dsink sync=1 enable-last-sample=0 async=0 \
    demux.src_15 ! nv3dsink sync=1 enable-last-sample=0 async=0
  1. additionnally, what is the device model?

I use Jetson Orin Nano Developer Kit.

15

from the log, there are many “There may be a timestamping problem, or this computer is too slow.” printing, which means many buffers are dropped before rendering when sync =1. Hence the video outputs seem be stopped. please set sync to 0, as written in the solution5.