some RTSP sources would blocking after many hours

ClancyLian · November 5, 2019, 2:02am

Hi,

we have created a project like ‘deepstream-app’ to run multi sources with RTSP protocols.

The pipeline is

rtspsrc -> rtph264depay -> queue -> decodebin -> queue -> nvstreammux -> queue -> nvinfer -> queue -> nvtracker -> queue -> fakesink

there are 11 RTSP sources to go to nvstreammux with bin "rtspsrc → rtph264depay → queue → decodebin → queue → " but after hours later, some source would be blocking(stalled or hang?).

To locate the problem, we delete the nvinfer and nvtracker. So the pipeline now is :

rtspsrc -> rtph264depay -> queue -> decodebin -> queue -> nvstreammux -> fakesink

But it also occur the same error( we use probe on nvstreammux src pad to see if source frames come out).

And we used GST_DEBUG=3 to see the log:

2:31:25.565533726    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.557571332 < 2:30:29.892039841)
2:31:25.603441913    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:31.509671334 < 2:30:29.045737381)
2:31:25.603630068    13     0x471f3b70 WARN       codecparsers_h264 gsth264parser.c:2086:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 1
2:31:25.605793579    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.597591789 < 2:30:29.892039841)
2:31:25.646160819    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.637612083 < 2:30:29.892039841)
2:31:25.684071777    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:31.629669523 < 2:30:29.285714313)
2:31:25.684185774    13     0x471f3b70 WARN       codecparsers_h264 gsth264parser.c:2086:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 17
2:31:25.686401405    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.677632214 < 2:30:29.892039841)
2:31:25.726629367    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.717652184 < 2:30:29.892039841)
2:31:25.767012435    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.757613858 < 2:30:29.892039841)
2:31:25.805038586    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:31.749631112 < 2:30:29.525692331)
2:31:25.807192165    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.797575839 < 2:30:29.892039841)
2:31:25.845341921    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:31.829606014 < 2:30:29.525692331)
2:31:25.847502410    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.837538124 < 2:30:29.892039841)
2:31:25.887773928    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.877500711 < 2:30:29.892039841)
2:31:25.928131638    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.917463597 < 2:30:29.892039841)
2:31:25.948880894    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:31.949569115 < 2:30:29.645681731)
2:31:25.968403943    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.957426780 < 2:30:29.892039841)
2:31:26.008747822    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:32.997390258 < 2:30:29.892039841)
2:31:26.048942612    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:33.037354028 < 2:30:29.892039841)
2:31:26.078562203    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:32.069533094 < 2:30:29.845664622)
2:31:26.089224581    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:33.077318088 < 2:30:29.892039841)
2:31:26.241127521    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:33.197211982 < 2:30:29.892039841)
2:31:26.329216173    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:33.317108403 < 2:30:31.171847576)
2:31:26.436666756    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:33.437007290 < 2:30:31.291833204)
2:31:26.545949792    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:33.556908585 < 2:30:31.411819174)
2:31:26.666110796    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:33.676812229 < 2:30:31.531805478)
2:31:26.859506666    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:33.876656349 < 2:30:31.731783374)
2:31:27.064555863    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:34.076506369 < 2:30:31.931762138)
2:31:27.091773305    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:33.069264404 < 2:30:30.925948723)
2:31:27.094771141    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:33.109254738 < 2:30:30.925948723)
2:31:27.096383907    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:34.116477089 < 2:30:31.931762138)
2:31:27.136721883    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:27:34.156448043 < 2:30:31.931762138)
2:31:27.145633232    13     0x471f3b70 WARN       codecparsers_h264 gsth264parser.c:2086:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 8
2:31:27.145917793    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:27:33.149245150 < 2:30:30.925948723)

2:31:56.673461810    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:28:03.672771328 < 2:30:53.890573865)
2:31:56.684334216    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:28:02.589105771 < 2:30:58.683997106)
2:31:56.713773641    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:28:03.712769364 < 2:30:53.890573865)
2:31:56.724539769    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:28:02.629083116 < 2:30:58.683997106)
2:31:56.754175911    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:28:03.752767416 < 2:30:53.890573865)
2:31:56.764922303    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:28:02.669060643 < 2:30:58.683997106)
2:31:56.789446582    13 0x7fa98c002a80 WARN       codecparsers_h264 gsth264parser.c:674:slice_parse_ref_pic_list_modification_1: error parsing "Reference picture list 0 modification"
2:31:56.789513544    13 0x7fa98c002a80 WARN       codecparsers_h264 gsth264parser.c:2219:gst_h264_parser_parse_slice_hdr: error parsing "Slice header"
2:31:56.794460520    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:28:03.792765484 < 2:30:53.890573865)
2:31:56.805257791    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:28:02.709038349 < 2:30:58.683997106)
2:31:56.834807456    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (2:28:03.832763567 < 2:30:53.890573865)
2:31:56.845542237    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (2:28:02.749016234 < 2:30:58.683997106)

2:32:24.063974834    13 0x7faab80042d0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder3> decreasing timestamp (2:28:28.224694146 < 2:31:27.181320266)
2:32:24.066883834    13 0x7fa96c002de0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder11> decreasing timestamp (2:28:28.570888210 < 2:31:14.248715154)
2:32:24.083976125    13 0x7faab00029e0 WARN       codecparsers_h264 gsth264parser.c:1544:gst_h264_parse_sps_data: value greater than max. value: 13, max 12
2:32:24.084055293    13 0x7faab00029e0 WARN       codecparsers_h264 gsth264parser.c:1854:gst_h264_parse_subset_sps: error parsing "Subset sequence parameter set"
2:32:24.084079068    13 0x7faab00029e0 WARN               h264parse gsth264parse.c:739:gst_h264_parse_process_nal:<h264parse6> failed to parse SPS:
2:32:24.107579103    13 0x7faa3400a320 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder7> decreasing timestamp (2:28:28.739651268 < 2:31:27.818643636)
2:32:24.134587865    13 0x7faa5c0090a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder6> decreasing timestamp (2:28:29.864007614 < 2:31:28.981996415)

videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder3> decreasing timestamp (2:28:42.144230853 < 2:31:28.061325162)
2:32:37.609437964    13 0x7fa96c002de0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder11> decreasing timestamp (2:28:42.251231746 < 2:31:41.008452525)
2:32:37.646682536    13 0x7faab80042d0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder3> decreasing timestamp (2:28:42.184230486 < 2:31:28.061325162)
2:32:37.658587063    13 0x7fa98c003000 WARN         rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:01.497769009 too big, reset skew
2:32:37.667488692    13 0x7faa3400a320 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder7> decreasing timestamp (2:28:42.379311074 < 2:31:41.538890389)
2:32:37.668906911    13     0x471f3ad0 WARN         rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:01.507238852 too big, reset skew
2:32:37.686709470    13 0x7faab80042d0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder3> decreasing timestamp (2:28:42.224230122 < 2:31:28.061325162)
2:32:37.689663353    13 0x7fa86c0099e0 WARN         rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:01.498377072 too big, reset skew
2:32:37.727325725    13 0x7faab80042d0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder3> decreasing timestamp (2:28:42.264229761 < 2:31:28.061325162)
2:32:37.729712826    13 0x7fa938009400 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder13> decreasing timestamp (2:28:42.152583770 < 2:31:40.151066111)
2:32:37.766537039    13 0x7faa3400a320 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder7> decreasing timestamp (2:28:42.459309150 < 2:31:41.578892649)
2:32:37.767961555    13 0x7faab80042d0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder3> decreasing timestamp (2:28:42.304229402 < 2:31:28.061325162)
2:32:37.778600492    13 0x7fa938009400 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder13> decreasing timestamp (2:28:42.192582958 < 2:31:40.151066111)
2:32:37.807973300    13 0x7faab80042d0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder3> decreasing timestamp (2:28:42.344229046 < 2:31:28.061325162)

videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder16> decreasing timestamp (2:28:45.850805880 < 2:31:44.351975573)
2:32:39.738068474    13 0x7fa8ec00a4a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder16> decreasing timestamp (2:28:45.890829006 < 2:31:44.351975573)
2:32:39.739497215    13 0x7fa938009400 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder13> decreasing timestamp (2:28:44.152508641 < 2:31:43.071041214)
2:32:39.769788366    13 0x7fa998009630 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder10> decreasing timestamp (2:28:45.814860677 < 2:31:44.184977847)
2:32:39.770113135    13 0x7fa8ec00a4a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder16> decreasing timestamp (2:28:45.930852909 < 2:31:44.351975573)
2:32:39.770184582    13 0x7fa9440046d0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder12> decreasing timestamp (2:28:45.511764922 < 2:31:44.109925159)
2:32:39.791467280    13 0x7fa98c002990 WARN       codecparsers_h264 gsth264parser.c:1194:gst_h264_parser_parse_sei_message: Bit non equal to zero.
2:32:39.791521791    13 0x7fa98c002990 WARN       codecparsers_h264 gsth264parser.c:1194:gst_h264_parser_parse_sei_message: Bit non equal to zero.
2:32:39.791625080    13 0x7fa98c002990 WARN       codecparsers_h264 gsth264parser.c:1188:gst_h264_parser_parse_sei_message: Bit non equal to one.
2:32:39.791779087    13 0x7fa98c002990 WARN       codecparsers_h264 gsth264parser.c:1194:gst_h264_parser_parse_sei_message: Bit non equal to zero.
2:32:39.791794894    13 0x7fa98c002990 WARN       codecparsers_h264 gsth264parser.c:1194:gst_h264_parser_parse_sei_message: Bit non equal to zero.
2:32:39.791809749    13 0x7fa98c002990 WARN       codecparsers_h264 gsth264parser.c:1194:gst_h264_parser_parse_sei_message: Bit non equal to zero.
2:32:39.791910251    13 0x7fa98c002990 WARN       codecparsers_h264 gsth264parser.c:1188:gst_h264_parser_parse_sei_message: Bit non equal to one.
2:32:39.810556296    13 0x7fa8ec00a4a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder16> decreasing timestamp (2:28:45.970876621 < 2:31:44.351975573)
2:32:39.810667007    13 0x7fa998009630 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder10> decreasing timestamp (2:28:45.854858478 < 2:31:44.184977847)
2:32:39.810945668    13 0x7fa9440046d0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder12> decreasing timestamp (2:28:45.551763711 < 2:31:44.109925159)
2:32:39.839648958    13 0x7fa938009400 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder13> decreasing timestamp (2:28:44.192506524 < 2:31:43.071041214)

3:22:57.451747074    13 0x7fa7fc008770 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder20> decreasing timestamp (3:18:04.313210594 < 3:22:00.463006693)
3:22:57.492165703    13 0x7fa7fc008770 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder20> decreasing timestamp (3:18:04.353207507 < 3:22:00.463006693)
3:22:57.532560945    13 0x7fa7fc008770 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder20> decreasing timestamp (3:18:04.393204445 < 3:22:00.463006693)
3:22:57.538934973    13 0x7fa86c009ad0 WARN       codecparsers_h264 gsth264parser.c:1548:gst_h264_parse_sps_data: value greater than max. value: 124, max 2
3:22:57.539000061    13 0x7fa86c009ad0 WARN       codecparsers_h264 gsth264parser.c:1854:gst_h264_parse_subset_sps: error parsing "Subset sequence parameter set"
3:22:57.539032310    13 0x7fa86c009ad0 WARN               h264parse gsth264parse.c:739:gst_h264_parse_process_nal:<h264parse20> failed to parse SPS:
3:22:57.664868287    13 0x7fa86c009ad0 WARN       codecparsers_h264 gsth264parser.c:1544:gst_h264_parse_sps_data: value greater than max. value: 48, max 12
3:22:57.664903007    13 0x7fa86c009ad0 WARN       codecparsers_h264 gsth264parser.c:1854:gst_h264_parse_subset_sps: error parsing "Subset sequence parameter set"
3:22:57.664923444    13 0x7fa86c009ad0 WARN               h264parse gsth264parse.c:739:gst_h264_parse_process_nal:<h264parse20> failed to parse SPS:
3:22:57.694211456    13 0x7fa7fc008770 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder20> decreasing timestamp (3:18:04.553201077 < 3:22:02.543736667)
3:22:57.787793211    13 0x7fa7fc008770 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder20> decreasing timestamp (3:18:04.673198621 < 3:22:02.663770276)

5:10:35.548988086    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (5:06:42.191441783 < 5:09:40.470299151)
5:10:35.598744500    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (5:06:42.231443106 < 5:09:40.470299151)
5:10:35.613972824    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (5:06:42.271444419 < 5:09:40.470299151)
5:10:35.654371525    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (5:06:42.311445721 < 5:09:40.470299151)
5:10:35.693248075    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (5:06:41.185382268 < 5:09:39.423615788)
5:10:35.733655382    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (5:06:41.225381268 < 5:09:39.423615788)
5:10:35.749193921    13 0x7fa98c002b70 WARN       codecparsers_h264 gsth264parser.c:1898:gst_h264_parse_pps: couldn't find associated sequence parameter set with id: 2
5:10:35.749243306    13 0x7fa98c002b70 WARN               h264parse gsth264parse.c:769:gst_h264_parse_process_nal:<h264parse18> failed to parse PPS:
5:10:35.774075677    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (5:06:41.265380276 < 5:09:39.423615788)
5:10:36.056669942    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (5:06:41.585372620 < 5:09:39.823774568)
5:10:36.056871047    13     0x471f3b70 WARN       codecparsers_h264 gsth264parser.c:2086:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 6
5:10:36.137448264    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (5:06:41.785372882 < 5:09:39.863789756)
5:10:36.298910311    13 0x7fa848004450 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder19> decreasing timestamp (5:06:41.985375912 < 5:09:40.103878365)
5:10:36.319464352    13 0x7fa86c009000 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder18> decreasing timestamp (5:06:42.471450827 < 5:09:40.470299151)

17:34:12.680268212    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:07.906903306 < 17:33:07.988073416)
17:34:12.720915110    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:07.946902110 < 17:33:07.988073416)
17:34:12.761173084    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:07.986900923 < 17:33:07.988073416)
17:34:12.801228577    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.026899746 < 17:33:07.988073416)
17:34:12.841887827    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.066898578 < 17:33:07.988073416)
17:34:12.881739702    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.106897419 < 17:33:07.988073416)
17:34:12.922126395    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.146896270 < 17:33:07.988073416)
17:34:12.962560765    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.186895130 < 17:33:07.988073416)
17:34:13.002550195    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.226893999 < 17:33:07.988073416)
17:34:13.042830609    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.266892877 < 17:33:07.988073416)
17:34:13.083218513    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.306891764 < 17:33:07.988073416)
17:34:13.123656049    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.346890660 < 17:33:07.988073416)
17:34:13.163418151    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.386889565 < 17:33:07.988073416)
17:34:13.203559366    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.426888478 < 17:33:07.988073416)
17:34:13.243507188    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.466887400 < 17:33:07.988073416)
17:34:13.285152498    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.506886331 < 17:33:07.988073416)
17:34:13.325095854    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.546885270 < 17:33:07.988073416)
17:34:13.365407323    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.586884218 < 17:33:07.988073416)
17:34:13.405268697    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.626883174 < 17:33:07.988073416)
17:34:13.445873750    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.666882139 < 17:33:07.988073416)
17:34:13.486117966    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.706881112 < 17:33:07.988073416)
17:34:13.525991710    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.746880093 < 17:33:07.988073416)
17:34:13.566741619    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.786879082 < 17:33:07.988073416)
17:34:13.607024404    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.826878079 < 17:33:07.988073416)
17:34:13.647087753    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.866877084 < 17:33:07.988073416)
17:34:13.687464996    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.946875118 < 17:33:07.988073416)
17:34:13.727543230    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:08.986874147 < 17:33:07.988073416)
17:34:13.767627624    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.026873184 < 17:33:07.988073416)
17:34:13.807991844    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.066872228 < 17:33:07.988073416)
17:34:13.848477189    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.106871280 < 17:33:07.988073416)
17:34:14.896258941    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.546861340 < 17:33:09.987530757)
17:34:14.936678019    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.586860479 < 17:33:09.987530757)
17:34:14.976925727    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.626859625 < 17:33:09.987530757)
17:34:15.016815646    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.666858778 < 17:33:09.987530757)
17:34:15.057067949    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.706857938 < 17:33:09.987530757)
17:34:15.097724163    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.746857104 < 17:33:09.987530757)
17:34:15.137915563    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.786856277 < 17:33:09.987530757)
17:34:15.178251397    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.826855457 < 17:33:09.987530757)
17:34:15.217481199    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.866854643 < 17:33:09.987530757)
17:34:15.257996553    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.906854145 < 17:33:09.987530757)
17:34:15.299022974    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.946853651 < 17:33:09.987530757)
17:34:15.339218098    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:09.986853161 < 17:33:09.987530757)
17:34:15.379650397    13 0x7fab50004590 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder1> decreasing timestamp (17:31:10.026852917 < 17:33:09.987530757)

many warning from nvv4l2decoder plugin with ‘decreasing timestamp’ and some from gsth264parser and rtpjitterbuffer

and there is not any message from pipeline bus callback. So we don’t know why.

Thanks.

DaneLLL · November 5, 2019, 7:48am

Hi,
Please configure sync=0 in fakesink and check if it helps. Maybe timestamps of the sources disorder after hours.

ClancyLian · November 5, 2019, 7:52am

Hi,

we have set fakesink attribute sync=0 but not help.

Thanks.

DaneLLL · November 5, 2019, 9:04am

Hi,
We suggest you share a config file so that we can run ‘deepstream-app -c CONFIG_FILE’ to reproduce the issue.
Also please share the brand and model ID of your IP cameras.

ClancyLian · November 8, 2019, 3:22am

Hi,

we have write demo to reproduce this issue in attachment.

we create a rtspserver to provide rtsp source.
we write a demo to run check it. can use GST_DEBUG=3 to run it.

after about 12 hours, maybe under 12 hours or longer. it would happen, some source would no frame from probe in streammux src pad.

Thanks.
rtsp-server.zip (5.58 KB)
demo.zip (5.35 KB)

ClancyLian · November 12, 2019, 1:03am

Hi,

I have checked it again, the demo above would not happen block. This is because the number sources is the same as max batch in nvstreammux property. But when I set batch size is 26 and number source is 11, then it will occur the blocking.

g_object_set (G_OBJECT (streammux), "width", MUXER_OUTPUT_WIDTH, "height",
      MUXER_OUTPUT_HEIGHT, "batch-size", 26,
      "batched-push-timeout", 40000, "live-source", true,
                "enable-padding", true, "nvbuf-memory-type", 0,
                "gpu-id", 0, NULL);

our project may dynamic add cameras, so we always set the batch size in nvstreammux more than the source. And would cause this bug.

Thanks.

DaneLLL · November 12, 2019, 6:59am

Hi,
We have Dynamic stream addition/removal app added on GitHub. Here is the link for the app:

Please try and check if it helps your usecase. Thanks.

ClancyLian · November 12, 2019, 8:04am

Hi, DaneLLL

In my demo, I don’t use dynamic stream addition/removal, I just set streammux’s property “batch-size” =26, and my source number is 11, when many hours later, some sources would blocking.

this will also happen in https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/tree/master/runtime_source_add_delete because I have seen the demo set streammux’s “batch-size” = 30, and the source is less than number sources.

Thanks.

ClancyLian · November 12, 2019, 8:05am

Hi,

you can just try in #5 attachment, and #6. and my theme is “some RTSP sources would blocking after many hours” but not “dynamic stream addition/removal”

Thanks.

DaneLLL · November 21, 2019, 8:18am

Hi,
Please try to add gst_event_new_flush_stop(FALSE) as suggested in the following post.
https://devtalk.nvidia.com/default/topic/1064141/deepstream-sdk/adding-and-removing-streams-during-runtime/post/5400986/#5400986