Recombining nvstreamdemux output with nvstreammux fails

• Hardware Platform (Jetson / GPU): GPU
• DeepStream Version: 5.0
• TensorRT Version 7.1.3.4
• NVIDIA GPU Driver Version (valid for GPU only) 440.33.01
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ?

I want to run inference on four rectangular tiles/crops of the image using the src-crop option of nvvideoconvert. The goal is to improve detection of small objects. This part works. The idea is to recombine those images into the full frame using nvmultistreamtiler. This also works:

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! tee name=t \
t. ! queue ! nvvideoconvert src-crop=0:0:1064:640 ! m.sink_0 \
t. ! queue ! nvvideoconvert src-crop=854:0:1920:640 ! m.sink_1 \
t. ! queue ! nvvideoconvert src-crop=0:440:1064:1080 ! m.sink_2 \
t. ! queue ! nvvideoconvert src-crop=854:440:1920:1080 ! m.sink_3 \
nvstreammux name=m batch-size=4 width=1064 height=640 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt batch-size=4 unique-id=1 ! nvmultistreamtiler ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so ! nvvideoconvert ! nvdsosd ! nveglglessink

However, I intentionally defined overlapping boxes as crops to avoid missing detections at tile boundaries, so I need to crop the tiles before combining them. This doesn’t work anymore:

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! tee name=t \
t. ! queue ! nvvideoconvert src-crop=0:0:1064:640 ! m.sink_0 \
t. ! queue ! nvvideoconvert src-crop=854:0:1920:640 ! m.sink_1 \
t. ! queue ! nvvideoconvert src-crop=0:440:1064:1080 ! m.sink_2 \
t. ! queue ! nvvideoconvert src-crop=854:440:1920:1080 ! m.sink_3 \
nvstreammux name=m batch-size=4 width=1064 height=640 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt batch-size=4 unique-id=1 ! nvstreamdemux name=de2 \
de2.src_0 ! nvvideoconvert src-crop=0:0:960:540 ! m4.sink_0 \
de2.src_1 ! nvvideoconvert src-crop=100:0:1064:540 ! m4.sink_1 \
de2.src_2 ! nvvideoconvert src-crop=0:100:960:640 ! m4.sink_2 \
de2.src_3 ! nvvideoconvert src-crop=100:100:1064:640 ! m4.sink_3 \
nvstreammux width=800 height=640 batch-size=4 name=m4 batched-push-timeout=40000000 ! nvmultistreamtiler rows=2 columns=2 width=800 height=640 ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so ! nvvideoconvert ! nvdsosd ! nveglglessink 

I get the following error:

Setting pipeline to PAUSED ...
0:00:00.185908491 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.185927621 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MJPG
0:00:00.185932910 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.185937178 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MJPG
0:00:00.185950123 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.185954094 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MPG4
0:00:00.185957438 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.185962571 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MPG4
0:00:00.185973475 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.185978283 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat H265
0:00:00.185982432 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.185987040 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat H265
0:00:00.185996918 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.186001619 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat H264
0:00:00.186005590 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.186009932 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat H264
0:00:00.186244257 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:src> Unable to try format: Unknown error -1
0:00:00.186250992 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:src> Could not probe minimum capture size for pixelformat NM12
0:00:00.186255394 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:src> Unable to try format: Unknown error -1
0:00:00.186259831 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:src> Could not probe maximum capture size for pixelformat NM12
0:00:00.186266885 13067 0x5639c68e0000 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x5639c650b9e0 Failed to determine interlace mode
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1523 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine open error
0:00:00.407730994 13067 0x5639c68e0000 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1690> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine failed
0:00:00.407770090 13067 0x5639c68e0000 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1797> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine failed, try rebuild
0:00:00.407778187 13067 0x5639c68e0000 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Reading Calibration Cache for calibrator: EntropyCalibration2
INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales.
INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache.
INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors.
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1495 Serialize engine failed because of file path: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine opened error
0:00:05.166323770 13067 0x5639c68e0000 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1743> [UID = 1]: failed to serialize cude engine to file: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x368x640       
1   OUTPUT kFLOAT conv2d_bbox     16x23x40        
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40         

0:00:05.168320520 13067 0x5639c68e0000 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<nvinfer0> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt sucessfully
0:00:05.168559626 13067 0x5639c68e0000 WARN                 basesrc gstbasesrc.c:3583:gst_base_src_start_complete:<filesrc0> pad not activated yet
Pipeline is PREROLLING ...
0:00:05.168694812 13067 0x5639c68e0000 WARN               structure gststructure.c:1832:priv_gst_structure_append_to_gstring: No value transform to serialize field 'display' of type 'GstEGLDisplay'
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
0:00:05.168782020 13067 0x5639d4ced800 WARN                 qtdemux qtdemux.c:3031:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:05.168842639 13067 0x5639d4ced800 WARN                 qtdemux qtdemux.c:3031:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2
0:00:05.273614489 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273663921 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273731416 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273750500 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273808044 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273828758 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273880842 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273897550 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273945971 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.273963512 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.274017370 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.274034080 13067 0x5639d4ced800 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:05.444912175 13067 0x5639d4ced800 WARN            v4l2videodec gstv4l2videodec.c:1614:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency
0:00:05.444933768 13067 0x5639d4ced800 WARN          v4l2bufferpool gstv4l2bufferpool.c:1057:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:05.445379582 13067 0x5639c76b9e80 WARN          v4l2bufferpool gstv4l2bufferpool.c:1503:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY
KLT Tracker Init
Caught SIGSEGV
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Spinning.  Please run 'gdb gst-launch-1.0 13067' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

What am I doing wrong? Or is there a better way of doing this?

Thanks.

Seems it crashes in nvmultistreamtiler, we will investigate it.

Did you have a chance to look into this?

This is a bug. We will update when the fix is available.