• Hardware Platform (Jetson / GPU): GPU
• DeepStream Version: 5.0
• TensorRT Version 7.1.3.4
• NVIDIA GPU Driver Version (valid for GPU only) 440.33.01
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ?
I want to run inference on four rectangular tiles/crops of the image using the src-crop option of nvvideoconvert. The goal is to improve detection of small objects. This part works. The idea is to recombine those images into the full frame using nvmultistreamtiler. This also works:
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! tee name=t \
t. ! queue ! nvvideoconvert src-crop=0:0:1064:640 ! m.sink_0 \
t. ! queue ! nvvideoconvert src-crop=854:0:1920:640 ! m.sink_1 \
t. ! queue ! nvvideoconvert src-crop=0:440:1064:1080 ! m.sink_2 \
t. ! queue ! nvvideoconvert src-crop=854:440:1920:1080 ! m.sink_3 \
nvstreammux name=m batch-size=4 width=1064 height=640 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt batch-size=4 unique-id=1 ! nvmultistreamtiler ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so ! nvvideoconvert ! nvdsosd ! nveglglessink
However, I intentionally defined overlapping boxes as crops to avoid missing detections at tile boundaries, so I need to crop the tiles before combining them. This doesn’t work anymore:
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! tee name=t \
t. ! queue ! nvvideoconvert src-crop=0:0:1064:640 ! m.sink_0 \
t. ! queue ! nvvideoconvert src-crop=854:0:1920:640 ! m.sink_1 \
t. ! queue ! nvvideoconvert src-crop=0:440:1064:1080 ! m.sink_2 \
t. ! queue ! nvvideoconvert src-crop=854:440:1920:1080 ! m.sink_3 \
nvstreammux name=m batch-size=4 width=1064 height=640 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt batch-size=4 unique-id=1 ! nvstreamdemux name=de2 \
de2.src_0 ! nvvideoconvert src-crop=0:0:960:540 ! m4.sink_0 \
de2.src_1 ! nvvideoconvert src-crop=100:0:1064:540 ! m4.sink_1 \
de2.src_2 ! nvvideoconvert src-crop=0:100:960:640 ! m4.sink_2 \
de2.src_3 ! nvvideoconvert src-crop=100:100:1064:640 ! m4.sink_3 \
nvstreammux width=800 height=640 batch-size=4 name=m4 batched-push-timeout=40000000 ! nvmultistreamtiler rows=2 columns=2 width=800 height=640 ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so ! nvvideoconvert ! nvdsosd ! nveglglessink
I get the following error:
Setting pipeline to PAUSED ... 0:00:00.185908491 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1 0:00:00.185927621 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MJPG 0:00:00.185932910 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1 0:00:00.185937178 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MJPG 0:00:00.185950123 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1 0:00:00.185954094 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MPG4 0:00:00.185957438 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1 0:00:00.185962571 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MPG4 0:00:00.185973475 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1 0:00:00.185978283 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat H265 0:00:00.185982432 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1 0:00:00.185987040 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat H265 0:00:00.185996918 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1 0:00:00.186001619 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat H264 0:00:00.186005590 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1 0:00:00.186009932 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat H264 0:00:00.186244257 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:src> Unable to try format: Unknown error -1 0:00:00.186250992 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:src> Could not probe minimum capture size for pixelformat NM12 0:00:00.186255394 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:src> Unable to try format: Unknown error -1 0:00:00.186259831 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:src> Could not probe maximum capture size for pixelformat NM12 0:00:00.186266885 13067 0x5639c68e0000 WARN v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x5639c650b9e0 Failed to determine interlace mode gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so gstnvtracker: Optional NvMOT_RemoveStreams not implemented gstnvtracker: Batch processing is OFF gstnvtracker: Past frame output is OFF ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1523 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine open error 0:00:00.407730994 13067 0x5639c68e0000 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1690> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine failed 0:00:00.407770090 13067 0x5639c68e0000 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1797> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine failed, try rebuild 0:00:00.407778187 13067 0x5639c68e0000 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Reading Calibration Cache for calibrator: EntropyCalibration2 INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales. INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache. INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors. ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1495 Serialize engine failed because of file path: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine opened error 0:00:05.166323770 13067 0x5639c68e0000 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1743> [UID = 1]: failed to serialize cude engine to file: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3 0 INPUT kFLOAT input_1 3x368x640 1 OUTPUT kFLOAT conv2d_bbox 16x23x40 2 OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40 0:00:05.168320520 13067 0x5639c68e0000 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<nvinfer0> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt sucessfully 0:00:05.168559626 13067 0x5639c68e0000 WARN basesrc gstbasesrc.c:3583:gst_base_src_start_complete:<filesrc0> pad not activated yet Pipeline is PREROLLING ... 0:00:05.168694812 13067 0x5639c68e0000 WARN structure gststructure.c:1832:priv_gst_structure_append_to_gstring: No value transform to serialize field 'display' of type 'GstEGLDisplay' Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL; 0:00:05.168782020 13067 0x5639d4ced800 WARN qtdemux qtdemux.c:3031:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1 0:00:05.168842639 13067 0x5639d4ced800 WARN qtdemux qtdemux.c:3031:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2 0:00:05.273614489 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273663921 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273731416 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273750500 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273808044 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273828758 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273880842 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273897550 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273945971 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.273963512 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.274017370 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.274034080 13067 0x5639d4ced800 ERROR v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git 0:00:05.444912175 13067 0x5639d4ced800 WARN v4l2videodec gstv4l2videodec.c:1614:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency 0:00:05.444933768 13067 0x5639d4ced800 WARN v4l2bufferpool gstv4l2bufferpool.c:1057:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold 0:00:05.445379582 13067 0x5639c76b9e80 WARN v4l2bufferpool gstv4l2bufferpool.c:1503:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY KLT Tracker Init Caught SIGSEGV Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Spinning. Please run 'gdb gst-launch-1.0 13067' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
What am I doing wrong? Or is there a better way of doing this?
Thanks.