DS5.0 sample deepstream-imagedata-multistream got exited unexpectedly

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2

Built an object detection model and replaced the model in sample deepstream-imagedata-multistream, it ran into the issue on some videos.

With that specific video, the sample with original model is working well.

With some other videos, the sample with my model is working well.

But the combination of that video and my model isn’t working. The video is 5-minute long, 1920 x 1080, fps: 15.0, frames: 4501. It appears that the callback function tiler_sink_pad_buffer_probe is not invoked for all frames.
tiler_sink_pad.add_probe(Gst.PadProbeType.BUFFER, tiler_sink_pad_buffer_probe, 0)

normal:

frame number = 4498
frame number = 4499
frame number = 4500
End-of-stream
Exiting app

INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: Reading Calibration Cache for calibrator: EntropyCalibration2
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: Detected 1 inputs and 2 output network tensors.
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [FullDims Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x368x640 min: 1x3x368x640 opt: 1x3x368x640 Max: 1x3x368x640
1 OUTPUT kFLOAT conv2d_bbox 16x23x40 min: 0 opt: 0 Max: 0
2 OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40 min: 0 opt: 0 Max: 0

bad:

frame number = 2881
frame number = 2882
frame number = 2883WARNING: …/nvdsinfer/nvdsinfer_func_utils.cpp:34 [TRT]: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x416x416
1 OUTPUT kFLOAT yolo_17 24x13x13
2 OUTPUT kFLOAT yolo_24 24x26x26

Any idea what could be the root cause why the callback function cannot be invoked for all frames.

3

Can you provide the whole error log?

4

Here is the full log. In the callback function tiler_sink_pad_buffer_probe, I only printed the frame number for each frame. I tested a 5-minute video Push_Pen_Front-20200630150417.300-300.mp4, it ran into problem after frame 2891. I also tested on multiple small videos from this one, like [300sec, 480sec], [480s, 600sec], they were fine, but [300sec, 600sec] was bad, very weird.

0:00:00.414721141 ^[[336m33467^[[00m 0x2df66a0 ^[[36mINFO ^[[00m ^[[00m nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger:^[[00m NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1591> [UID = 1]: Trying to create engine from model files
Loading pre-trained weights…
Loading weights of yolov3-tiny complete!
Total Number of weights read : 8680864
Loading pre-trained weights…
Loading weights of yolov3-tiny complete!
Total Number of weights read : 8680864
Building Yolo network…
layer inp_size out_size weightPtr
(0) conv-bn-leaky 3 x 416 x 416 16 x 416 x 416 496
(1) maxpool 16 x 416 x 416 16 x 208 x 208 496
(2) conv-bn-leaky 16 x 208 x 208 32 x 208 x 208 5232
(3) maxpool 32 x 208 x 208 32 x 104 x 104 5232
(4) conv-bn-leaky 32 x 104 x 104 64 x 104 x 104 23920
(5) maxpool 64 x 104 x 104 64 x 52 x 52 23920
(6) conv-bn-leaky 64 x 52 x 52 128 x 52 x 52 98160
(7) maxpool 128 x 52 x 52 128 x 26 x 26 98160
(8) conv-bn-leaky 128 x 26 x 26 256 x 26 x 26 394096
(9) maxpool 256 x 26 x 26 256 x 13 x 13 394096
(10) conv-bn-leaky 256 x 13 x 13 512 x 13 x 13 1575792
(11) maxpool 512 x 13 x 13 512 x 13 x 13 1575792
(12) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 6298480
(13) conv-bn-leaky 1024 x 13 x 13 256 x 13 x 13 6561648
(14) conv-bn-leaky 256 x 13 x 13 512 x 13 x 13 7743344
(15) conv-linear 512 x 13 x 13 24 x 13 x 13 7755656
(16) yolo 24 x 13 x 13 24 x 13 x 13 7755656
(17) route - 256 x 13 x 13 7755656
(18) conv-bn-leaky 256 x 13 x 13 128 x 13 x 13 7788936
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: mm1_19: broadcasting input0 to make tensors conform, dims(input0)=[1,26,13][NONE] dims(input1)=[128,13,13][NONE].
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: mm2_19: broadcasting input1 to make tensors conform, dims(input0)=[128,26,13][NONE] dims(input1)=[1,13,26][NONE].
(19) upsample 128 x 13 x 13 128 x 26 x 26 -
(20) route - 384 x 26 x 26 7788936
(21) conv-bn-leaky 384 x 26 x 26 256 x 26 x 26 8674696
(22) conv-linear 256 x 26 x 26 24 x 26 x 26 8680864
(23) yolo 24 x 26 x 26 24 x 26 x 26 8680864
Output yolo blob names :
yolo_17
yolo_24
Total number of yolo layers: 49
Building yolo network complete!
Building the TensorRT Engine…
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: mm1_19: broadcasting input0 to make tensors conform, dims(input0)=[1,26,13][NONE] dims(input1)=[128,13,13][NONE].
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: mm2_19: broadcasting input1 to make tensors conform, dims(input0)=[128,26,13][NONE] dims(input1)=[1,13,26][NONE].
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:37 [TRT]: Detected 1 inputs and 2 output network tensors.
Building complete!
0:00:20.389677474 ^[[336m33467^[[00m 0x2df66a0 ^[[36mINFO ^[[00m ^[[00m nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger:^[[00m NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1624> [UID = 1]: serialize cuda engine to file: /root/ds-dp-5.0/deepstream_python_v0.9/apps/deepstream-imagedata-multistream/model_b1_gpu0_fp32.engine successfully
0:00:20.395163109 ^[[336m33467^[[00m 0x2df66a0 ^[[36mINFO ^[[00m ^[[00m nvinfer gstnvinfer_impl.cpp:311:notifyLoadModelStatus:^[[00m [UID 1]: Load new model:config_infer_primary_yoloV3_tiny.txt sucessfully
Creating Pipeline

Creating streamux

Creating source_bin 0

Creating source bin
source-bin-00
Creating Pgie

Creating nvvidconv1

Creating filter1

Creating tiler

Creating nvvidconv

Creating nvosd

Creating EGLSink

Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
1 : file:///root/data/input_video/Push_Pen_Front-20200630150417.300-300.mp4
Starting pipeline

Decodebin child added: source

Decodebin child added: decodebin0

Decodebin child added: qtdemux0

Decodebin child added: multiqueue0

Decodebin child added: h264parse0

Decodebin child added: capsfilter0

Decodebin child added: nvv4l2decoder0

In cb_newpad

frame number = 0
frame number = 1
…
rame number = 2888
frame number = 2889
frame number = 2890
frame number = 2891WARNING: …/nvdsinfer/nvdsinfer_func_utils.cpp:34 [TRT]: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x416x416
1 OUTPUT kFLOAT yolo_17 24x13x13
2 OUTPUT kFLOAT yolo_24 24x26x26

5

By the way, with the same video, it can be processed by DS4.0 code in C.

Does this indicate DS5.0 python code has some problem performing object detection?

6

Figured out below parameters impacted the result. With 1920x1080, those videos failed to be processed, but with 1280x720, they were able to be processed successfully.

TILED_OUTPUT_WIDTH=1920
TILED_OUTPUT_HEIGHT=1080

Any idea why this matters?