Pipelines fails to push into python appsink after introduction of tensorrt/nvinfer

[awennersteen]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) T4
• DeepStream Version 5.0.1
• TensorRT Version 7.2.1
• NVIDIA GPU Driver Version (valid for GPU only) 450.80.02 and 440.33 and 440.64
• CONTAINER: Modified from the deepstream 5.0.1 20.09 base (also tried devel)

We’ve got a Python pipeline ending in a gstreamer appsink where we do various things, including extracting the raw frames and metadata from previous nvinfer elements.

The (simplified but still with the problem present) pipeline basically goes:
RTSPsrc → decode on and convert to RGBA on gpu → nvstreammux → NvInfer (YoloV4) → appsink

After recently integrating YoloV4, using the steps provided here: GitHub - NVIDIA-AI-IOT/yolov4_deepstream, the pipeline only runs for a few seconds to a few minutes with RTSP on the T4, after which it freezes/stops pusing frames into the appsink. The logs below both end when this freeze happens, at which point no more logs are seen at this log level.
The pipeline stays alive, and no errors or EOS signal is posted, which can be seen by the fact that it continues to do things in the rtsp src level, and keeps allocating new memory.

However, it runs fine with video files and with RTSP on the RTX2080TI. The pipeline works fine without tensorrt, and has worked fine for months (without tensorrt models) until now.

We’ve tried to debug with debug levels

GST_DEBUG=3,*nv*:7,rtph265depay:1,*parse*:1,nvvideoconvert:1,*app*:7,basesink:7,basetransform:7,GST_BUFFER:5.GST_MEMORY:5

without any luck, and also tried to include GST_SCHEDULING:5 but it doesn’t show anything obviously wrong, although we don’t know what to look for either.

NvStreamMux settings:
‘batch-size’: 1,
‘height’: 1920,
‘width’: 1080,
‘gpu-id’: 0,
‘batched-push-timeout’: 40000
‘live-source’: 1,
‘attach-sys-ts’: 0

appsink has sync 0 ( also tried sync 1)

Some example logs (logs marked root are from inside Python appsink), the second one also logs basetransform at debug level.

Logs 1

2020-11-17 13:29:29,511 root         DEBUG    Gstreamer PTS: 0:00:09.806430
2020-11-17 13:29:29,512 root         DEBUG    Num frames in nvstreammux:1
0:00:34.905192055   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:3577:gst_base_sink_chain_unlocked:<appsink0> object unref after render 0x7f011c00aea0
0:00:34.905243078   134      0x5f8b8a0 DEBUG             GST_BUFFER gstbuffer.c:2384:gst_buffer_foreach_meta: remove metadata 0x7f00b002aaa8 (NvDsMeta)
0:00:34.905383845   134      0x5f8b8a0 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool1> release_buffer
0:00:34.905395891   134      0x5f8b8a0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.905425216   134      0x5f8b8a0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.905476576   134 0x7f011c04bed0 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f00b002b068 (NvDsMeta) of size 72
0:00:34.905500202   134 0x7f011c04bed0 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:00:34.905613358   134 0x7f00c401bad0 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:00:34.905620129   134 0x7f011c04bed0 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:00:34.905639621   134 0x7f011c04bed0 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:00:34.905658869   134 0x7f011c04bed0 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:194:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Frame NTP calculated. mode: System Time. source 0: PTS:0:00:09.828447027(9828447027) NTP: 2020-11-17T13:29:29.576Z(1605619769576998000)
0:00:34.905688875   134 0x7f011c04bed0 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f011c00aea0, batch size 1, PTS 0:00:10.142090057
0:00:34.905698617   134 0x7f011c04bed0 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:10.142090057
0:00:34.905622953   134 0x7f00c401bad0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.905842413   134 0x7f00c401bad0 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:00:34.905860726   134 0x7f00c401bad0 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f00d0054248 of API type GstVideoMetaAPI
0:00:34.905924170   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.906029690   134 0x7f00c401bad0 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f00d0053480 from source 0
0:00:34.906041896   134 0x7f00c401bad0 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:00:34.909038425   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.911722825   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.919208670   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.923010283   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.925728675   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.928239201   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.928359466   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:3421:gst_base_sink_chain_unlocked:<appsink0> got times start: 0:00:09.890736151, end: 99:99:99.999999999
0:00:34.928407879   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:2006:gst_base_sink_get_sync_times:<appsink0> got times start: 0:00:09.890736151, stop: 99:99:99.999999999, do_sync 1
0:00:34.928422950   134      0x5f8b8a0 LOG                 basesink gstbasesink.c:2551:gst_base_sink_do_sync:<appsink0> avg frame diff 0:00:00.086672094
0:00:34.928430738   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:2591:gst_base_sink_do_sync:<appsink0> reset rc_time to time 0:00:09.890736151
0:00:34.928442076   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:2603:gst_base_sink_do_sync:<appsink0> possibly waiting for clock to reach 0:00:09.890736151, adjusted 0:00:09.890736151
0:00:34.928451342   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:2223:gst_base_sink_wait_clock:<appsink0> sync disabled
0:00:34.928459786   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:2610:gst_base_sink_do_sync:<appsink0> clock returned 4, jitter  0:00:00.000000000
0:00:34.928467077   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:3538:gst_base_sink_chain_unlocked:<appsink0> rendering object 0x7f011c00ac80
0:00:34.928474630   134      0x5f8b8a0 DEBUG                appsink gstappsink.c:876:gst_app_sink_render_common:<appsink0> pushing render buffer/list 0x7f011c00ac80 on queue (0)
0:00:34.928507890   134      0x5f8b8a0 DEBUG                appsink gstappsink.c:1630:gst_app_sink_try_pull_sample:<appsink0> trying to grab a buffer
0:00:34.928518358   134      0x5f8b8a0 DEBUG                appsink gstappsink.c:821:dequeue_buffer:<appsink0> dequeued buffer/list 0x7f011c00ac80
0:00:34.928522854   134      0x5f8b8a0 DEBUG                appsink gstappsink.c:1654:gst_app_sink_try_pull_sample:<appsink0> we have a buffer 0x7f011c00ac80
2020-11-17 13:29:29,599 root         DEBUG    Gstreamer PTS: 0:00:09.890736
2020-11-17 13:29:29,600 root         DEBUG    Num frames in nvstreammux:1
0:00:34.992279179   134      0x5f8b8a0 DEBUG               basesink gstbasesink.c:3577:gst_base_sink_chain_unlocked:<appsink0> object unref after render 0x7f011c00ac80
0:00:34.992297367   134      0x5f8b8a0 DEBUG             GST_BUFFER gstbuffer.c:2384:gst_buffer_foreach_meta: remove metadata 0x7f00b002ab08 (NvDsMeta)
0:00:34.993298463   134      0x5f8b8a0 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool1> release_buffer
0:00:34.993318623   134      0x5f8b8a0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.993351408   134      0x5f8b8a0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.993481195   134 0x7f011c04bed0 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f00b0010408 (NvDsMeta) of size 72
0:00:34.993504280   134 0x7f011c04bed0 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:00:34.993566504   134 0x7f00c401bad0 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:00:34.993575002   134 0x7f00c401bad0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.993613941   134 0x7f011c04bed0 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:00:34.993622874   134 0x7f011c04bed0 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:00:34.993641190   134 0x7f011c04bed0 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:194:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Frame NTP calculated. mode: System Time. source 0: PTS:0:00:10.003087672(10003087672) NTP: 2020-11-17T13:29:29.664Z(1605619769664978000)
0:00:34.993646294   134 0x7f00d002f540 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.993687919   134 0x7f011c04bed0 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f011c00ac80, batch size 1, PTS 0:00:10.230088342
0:00:34.993696835   134 0x7f011c04bed0 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:10.230088342
0:00:34.994007577   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.994139172   134 0x7f00d002f540 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.994190064   134 0x7f00c401bad0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder0> decreasing timestamp (0:00:09.739451815 < 0:00:10.003087672)
0:00:34.994256773   134 0x7f00c401bad0 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:00:34.994289980   134 0x7f00c401bad0 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f00d0054368 of API type GstVideoMetaAPI
0:00:34.994441821   134 0x7f00c401bad0 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f00d007ac70 from source 0
0:00:34.994454434   134 0x7f00c401bad0 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:00:34.996290609   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:34.998845744   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.001389916   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.009001561   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.013341757   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.015931374   134 0x7f00d004bb50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1

Logs 2

2020-11-17 14:09:28,307 root         DEBUG    Gstreamer PTS: 0:02:32.989348
2020-11-17 14:09:28,308 root         DEBUG    Num frames in nvstreammux:1
0:02:54.210179132   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f909808b030 of size 64621, PTS 0:02:33.174224351 and offset 68734593
0:02:54.210207359   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:02:54.210216091   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f909808b030, out 0x7f909808b030
0:02:54.210221513   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:02:54.210289604   134 0x7f909802e050 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.214096246   134 0x7f907d65f990 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder0> decreasing timestamp (0:02:32.836681248 < 0:02:33.007550565)
0:02:54.214134704   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f9098080c50 of size 64, PTS 0:02:33.007550565 and offset NONE
0:02:54.214142140   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:02:54.214147272   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:02:54.214154435   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:02:54.214160780   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:02:54.214170631   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:02:54.214177045   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f9098080c50, out 0x7f90980804e0
0:02:54.214181672   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:02:54.214192716   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f909807d938 of API type GstVideoMetaAPI
0:02:54.214423910   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f90980804e0 of size 64, PTS 0:02:33.007550565 and offset NONE
0:02:54.214438705   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:02:54.214445309   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f90980804e0, out 0x7f90980804e0
0:02:54.214449665   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:02:54.214461991   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f90980804e0 of size 64, PTS 0:02:33.007550565 and offset NONE
0:02:54.214467021   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:02:54.214472274   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:02:54.214478386   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f90980804e0, out 0x7f90980804e0
0:02:54.214482454   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:02:54.214494499   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f90980804e0 of size 64, PTS 0:02:33.007550565 and offset 1835
0:02:54.214499402   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:02:54.214504379   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f90980804e0, out 0x7f90980804e0
0:02:54.214508742   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:02:54.214540432   134 0x7f907d65f990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f90980804e0 from source 0
0:02:54.214549313   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:02:54.214816644   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2323:gst_base_transform_chain:<videorate0> we got return custom-success
0:02:54.214833300   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:02:54.214841039   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.214849352   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2332:gst_base_transform_chain:<videorate0> dropped a buffer, marking DISCONT
0:02:54.214876535   134 0x7f907c04b590 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:02:54.214891955   134 0x7f907c04b590 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:02:54.214915753   134 0x7f907c04b590 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:194:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Frame NTP calculated. mode: System Time. source 0: PTS:0:02:33.007550565(153007550565) NTP: 2020-11-17T14:09:28.359Z(1605622168359587000)
0:02:54.214949977   134 0x7f907c04b590 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f9098051260, batch size 1, PTS 0:02:33.075564809
0:02:54.214963079   134 0x7f907c04b590 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:02:33.075564809
0:02:54.215166442   134 0x7f907c04b590 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f909400c988 (NvDsMeta) of size 72
0:02:54.215292036   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.215301113   134 0x7f907c04b590 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:02:54.234790079   134      0x651bca0 DEBUG               basesink gstbasesink.c:3577:gst_base_sink_chain_unlocked:<appsink0> object unref after render 0x7f9098051040
0:02:54.234810099   134      0x651bca0 DEBUG             GST_BUFFER gstbuffer.c:2384:gst_buffer_foreach_meta: remove metadata 0x7f9070041e48 (NvDsMeta)
0:02:54.234886304   134      0x651bca0 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool1> release_buffer
0:02:54.234894948   134      0x651bca0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.234902628   134      0x651bca0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.237653115   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.238893696   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.241537840   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.243655244   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.245704441   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.247654834   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.249628554   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.289557267   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f909808a350 of size 24290, PTS 0:02:32.984200626 and offset 68799214
0:02:54.289572353   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:02:54.289596152   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f909808a350, out 0x7f909808a350
0:02:54.289600271   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:02:54.289655915   134 0x7f909802e050 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.289947485   134 0x7f907d65f990 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder0> decreasing timestamp (0:02:32.984200626 < 0:02:33.007550565)
0:02:54.289975448   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f907c0bf830 of size 64, PTS 0:02:33.007550565 and offset NONE
0:02:54.289989172   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:02:54.289994804   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:02:54.290001818   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:02:54.290013460   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:02:54.290023209   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:02:54.290030653   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f907c0bf830, out 0x7f90980803d0
0:02:54.290034155   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:02:54.290043309   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f9098047448 of API type GstVideoMetaAPI
0:02:54.290162398   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f90980803d0 of size 64, PTS 0:02:33.007550565 and offset NONE
0:02:54.290171261   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:02:54.290175929   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f90980803d0, out 0x7f90980803d0
0:02:54.290180878   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:02:54.290191198   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f90980803d0 of size 64, PTS 0:02:33.007550565 and offset NONE
0:02:54.290200413   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:02:54.290206445   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:02:54.290212900   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f90980803d0, out 0x7f90980803d0
0:02:54.290217862   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:02:54.290231222   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f90980803d0 of size 64, PTS 0:02:33.007550565 and offset 1836
0:02:54.290236986   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:02:54.290243757   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f90980803d0, out 0x7f90980803d0
0:02:54.290253096   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:02:54.290267790   134 0x7f907d65f990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f90980803d0 from source 0
0:02:54.290278778   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:02:54.290392338   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2323:gst_base_transform_chain:<videorate0> we got return custom-success
0:02:54.290423316   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:02:54.290433398   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.290441213   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2332:gst_base_transform_chain:<videorate0> dropped a buffer, marking DISCONT
0:02:54.290402310   134 0x7f907c04b590 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:02:54.290483973   134 0x7f907c04b590 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:02:54.290516470   134 0x7f907c04b590 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:194:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Frame NTP calculated. mode: System Time. source 0: PTS:0:02:33.007550565(153007550565) NTP: 2020-11-17T14:09:28.435Z(1605622168435184000)
0:02:54.290554033   134 0x7f907c04b590 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x5befcc0, batch size 1, PTS 0:02:33.151168431
0:02:54.290566109   134 0x7f907c04b590 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:02:33.151168431
0:02:54.290757596   134 0x7f907c04b590 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f90700424c8 (NvDsMeta) of size 72
0:02:54.290857295   134 0x7f907c04b590 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:02:54.290916740   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.360386077   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f90980785f0 of size 18888, PTS 0:02:33.220278728 and offset 68823504
0:02:54.360403157   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:02:54.360408874   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f90980785f0, out 0x7f90980785f0
0:02:54.360412561   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:02:54.360469097   134 0x7f909802e050 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.360803898   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f9098080d60 of size 64, PTS 0:02:33.174224351 and offset NONE
0:02:54.360821754   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:02:54.360827174   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:02:54.360834544   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:02:54.360841414   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:02:54.360849569   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:02:54.360856455   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f9098080d60, out 0x7f9098078e70
0:02:54.360861431   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:02:54.360872745   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f909807da58 of API type GstVideoMetaAPI
0:02:54.360986095   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f9098078e70 of size 64, PTS 0:02:33.174224351 and offset NONE
0:02:54.360993918   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:02:54.360998451   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f9098078e70, out 0x7f9098078e70
0:02:54.361001804   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:02:54.361010115   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f9098078e70 of size 64, PTS 0:02:33.174224351 and offset NONE
0:02:54.361024076   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:02:54.361029114   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:02:54.361034415   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f9098078e70, out 0x7f9098078e70
0:02:54.361038561   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:02:54.361050005   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f9098078e70 of size 64, PTS 0:02:33.174224351 and offset 1837
0:02:54.361055101   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:02:54.361061129   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f9098078e70, out 0x7f9098078e70
0:02:54.361065588   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:02:54.361076064   134 0x7f907d65f990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f9098078e70 from source 0
0:02:54.361081512   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:02:54.361189863   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2323:gst_base_transform_chain:<videorate0> we got return custom-success
0:02:54.361202945   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:02:54.361210507   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.361213295   134 0x7f907c04b590 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:02:54.361217590   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2332:gst_base_transform_chain:<videorate0> dropped a buffer, marking DISCONT
0:02:54.361237246   134 0x7f907c04b590 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:02:54.361260985   134 0x7f907c04b590 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:194:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Frame NTP calculated. mode: System Time. source 0: PTS:0:02:33.174224351(153174224351) NTP: 2020-11-17T14:09:28.505Z(1605622168505935000)
0:02:54.361297807   134 0x7f907c04b590 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f907c0bf1d0, batch size 1, PTS 0:02:33.221913790
0:02:54.361310092   134 0x7f907c04b590 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:02:33.221913790
0:02:54.361474773   134 0x7f907c04b590 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f909400c808 (NvDsMeta) of size 72
0:02:54.361491941   134 0x7f907c04b590 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:02:54.361559535   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.441383771   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f9098072a10 of size 12172, PTS 0:02:33.379774659 and offset 68842392
0:02:54.441402994   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:02:54.441410276   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f9098072a10, out 0x7f9098072a10
0:02:54.441414943   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:02:54.441474422   134 0x7f909802e050 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.441756143   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f9098080b40 of size 64, PTS 0:02:33.220278728 and offset NONE
0:02:54.441770241   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:02:54.441776263   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:02:54.441783032   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:02:54.441790069   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:02:54.441798729   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:02:54.441805741   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f9098080b40, out 0x7f9098080e70
0:02:54.441810417   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:02:54.441821343   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f907c006128 of API type GstVideoMetaAPI
0:02:54.441932878   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f9098080e70 of size 64, PTS 0:02:33.220278728 and offset NONE
0:02:54.441941099   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:02:54.441945886   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f9098080e70, out 0x7f9098080e70
0:02:54.441950069   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:02:54.441959190   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f9098080e70 of size 64, PTS 0:02:33.220278728 and offset NONE
0:02:54.441964373   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:02:54.441969159   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:02:54.441974876   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f9098080e70, out 0x7f9098080e70
0:02:54.441979125   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:02:54.441990536   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f9098080e70 of size 64, PTS 0:02:33.220278728 and offset 1838
0:02:54.441999147   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:02:54.442004610   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f9098080e70, out 0x7f9098080e70
0:02:54.442008719   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:02:54.442021584   134 0x7f907d65f990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f9098080e70 from source 0
0:02:54.442028529   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:02:54.442137082   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2323:gst_base_transform_chain:<videorate0> we got return custom-success
0:02:54.442146687   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:02:54.442152821   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.442165439   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2332:gst_base_transform_chain:<videorate0> dropped a buffer, marking DISCONT
0:02:54.442145733   134 0x7f907c04b590 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:02:54.442197935   134 0x7f907c04b590 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:02:54.442213803   134 0x7f907c04b590 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:194:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Frame NTP calculated. mode: System Time. source 0: PTS:0:02:33.220278728(153220278728) NTP: 2020-11-17T14:09:28.586Z(1605622168586891000)
0:02:54.442244256   134 0x7f907c04b590 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f9098051040, batch size 1, PTS 0:02:33.302859863
0:02:54.442252320   134 0x7f907c04b590 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:02:33.302859863
0:02:54.442490087   134 0x7f907c0c18f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.528090244   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f9098075350 of size 153014, PTS 0:02:33.748669984 and offset 68854564
0:02:54.528107677   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:02:54.528114302   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f9098075350, out 0x7f9098075350
0:02:54.528117940   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:02:54.528188158   134 0x7f909802e050 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:02:54.528599377   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f9098080c50 of size 64, PTS 0:02:33.379774659 and offset NONE
0:02:54.528612869   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:02:54.528618451   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:02:54.528625821   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:02:54.528636793   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:02:54.528645570   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:02:54.528656713   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f9098080c50, out 0x7f90980804e0
0:02:54.528661913   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:02:54.528671724   134 0x7f907d65f990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f909807d938 of API type GstVideoMetaAPI
0:02:54.528777896   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f90980804e0 of size 64, PTS 0:02:33.379774659 and offset NONE
0:02:54.528786609   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:02:54.528792271   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f90980804e0, out 0x7f90980804e0
0:02:54.528796360   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:02:54.528804625   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f90980804e0 of size 64, PTS 0:02:33.379774659 and offset NONE
0:02:54.528812111   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:02:54.528817164   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:02:54.528822396   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f90980804e0, out 0x7f90980804e0
0:02:54.528826871   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:02:54.528838018   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f90980804e0 of size 64, PTS 0:02:33.379774659 and offset 1839
0:02:54.528845545   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:02:54.528851368   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f90980804e0, out 0x7f90980804e0
0:02:54.528860405   134 0x7f907d65f990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:02:54.528874032   134 0x7f907d65f990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f90980804e0 from source 0
0:02:54.528883661   134 0x7f907d65f990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:02:54.605517014   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f90980727f0 of size 25743, PTS 0:02:33.574044439 and offset 69007578
0:02:54.605529436   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:02:54.605535207   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f90980727f0, out 0x7f90980727f0
0:02:54.605538530   134 0x7f909802e050 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform

[Fiona.Chen]

What does “the pipeline only runs for a few seconds to a few minutes with RTSP on the T4” mean? There is no GstBuffer received in appsink or the pipeline stopped without any log print out?

[awennersteen]

Hi Fiona,

I tried to edit more for clarity. But I’ll try to expand as well:

The pieline runs as normal for some time, doing tensorrt inference and pushing it into the appsink.
At some point, let’s call it T, the appsink stops receiving buffers and similarly the logs at the level shown above stops as well.

After T the logs keep on going for other parts of the pipeline though. the rtspsrc element appears to keep on going, and the logs from GST_MEMORY show that memory is continuously allocated but not freed.

It appears that at T, the NvDsMeta is being written to the buffer, but then the pipeline stops progressing shortly afterwards and the basesink element never engages.

[Fiona.Chen]

How long will T be? This problem can only be reproduced by YoloV4 model? What will happen if you replace appsink with fakesink?

[awennersteen]

T has been between 10 seconds and 3 minutes for me.
Yes, just the yolov4 into an appsink is sufficient. The appsink doesn’t have to do anything.

[Fiona.Chen]

What will happen if you replace appsink with fakesink?

[awennersteen]

Also fails. Since it never goes into the basesink I suppose any sink will fail

0:00:34.951189004   411 0x7f4754030c50 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:00:34.951209539   411 0x7f4754030c50 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f4754068460, out 0x7f4754068460
0:00:34.951224267   411 0x7f4754030c50 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:00:35.284641692   411 0x7f47400c9d00 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.287155867   411 0x7f47400c9d00 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.289515464   411 0x7f47400c9d00 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.291688352   411 0x7f47400c9d00 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.293673341   411      0x59b38a0 DEBUG               basesink gstbasesink.c:3421:gst_base_sink_chain_unlocked:<fake_sink> got times start: 0:00:15.108831056, end: 99:99:99.999999999
0:00:35.293694657   411      0x59b38a0 DEBUG               basesink gstbasesink.c:2006:gst_base_sink_get_sync_times:<fake_sink> got times start: 0:00:15.108831056, stop: 99:99:99.999999999, do_sync 1
0:00:35.293705161   411      0x59b38a0 LOG                 basesink gstbasesink.c:2551:gst_base_sink_do_sync:<fake_sink> avg frame diff 0:00:00.086183079
0:00:35.293714002   411      0x59b38a0 DEBUG               basesink gstbasesink.c:2591:gst_base_sink_do_sync:<fake_sink> reset rc_time to time 0:00:15.108831056
0:00:35.293723372   411      0x59b38a0 DEBUG               basesink gstbasesink.c:2603:gst_base_sink_do_sync:<fake_sink> possibly waiting for clock to reach 0:00:15.108831056, adjusted 0:00:15.108831056
0:00:35.293730055   411      0x59b38a0 DEBUG               basesink gstbasesink.c:2223:gst_base_sink_wait_clock:<fake_sink> sync disabled
0:00:35.293739859   411      0x59b38a0 DEBUG               basesink gstbasesink.c:2610:gst_base_sink_do_sync:<fake_sink> clock returned 4, jitter  0:00:00.000000000
0:00:35.293746923   411      0x59b38a0 DEBUG               basesink gstbasesink.c:3538:gst_base_sink_chain_unlocked:<fake_sink> rendering object 0x7f4754067e60
0:00:35.293752749   411      0x59b38a0 DEBUG               basesink gstbasesink.c:994:gst_base_sink_set_last_buffer_unlocked:<fake_sink> setting last buffer to 0x7f4754067e60
0:00:35.293760502   411      0x59b38a0 DEBUG             GST_BUFFER gstbuffer.c:2384:gst_buffer_foreach_meta: remove metadata 0x7f4714008868 (NvDsMeta)
0:00:35.293784675   411      0x59b38a0 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool1> release_buffer
0:00:35.293791000   411      0x59b38a0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.293805712   411      0x59b38a0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.293818817   411      0x59b38a0 DEBUG               basesink gstbasesink.c:3577:gst_base_sink_chain_unlocked:<fake_sink> object unref after render 0x7f4754067e60
0:00:35.293862011   411 0x7f4740052d90 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f47280146a8 (NvDsMeta) of size 72
0:00:35.293911108   411 0x7f4740052d90 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:00:35.293985488   411 0x7f47400c9d00 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.294227342   411 0x7f4740052d90 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:00:35.294247025   411 0x7f4740052d90 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:00:35.294305871   411 0x7f4740052d90 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:194:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Frame NTP calculated. mode: System Time. source 0: PTS:0:00:15.356672260(15356672260) NTP: 2020-11-18T07:23:38.233Z(1605684218233849000)
0:00:35.294343700   411 0x7f4740052d90 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f4754067c40, batch size 1, PTS 0:00:15.769105986
0:00:35.294378203   411 0x7f4740052d90 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:15.769105986
0:00:35.294231483   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2323:gst_base_transform_chain:<videorate0> we got return custom-success
0:00:35.294493622   411 0x7f4741672990 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:00:35.294517662   411 0x7f4741672990 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.294539314   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2332:gst_base_transform_chain:<videorate0> dropped a buffer, marking DISCONT
0:00:35.294654152   411 0x7f4754030c50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.294836684   411 0x7f4754030c50 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f47540654c0 of size 19481, PTS 0:00:15.505391151 and offset 7638214
0:00:35.294848599   411 0x7f4754030c50 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:00:35.294968196   411 0x7f4754030c50 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f47540654c0, out 0x7f47540654c0
0:00:35.294995405   411 0x7f4754030c50 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:00:35.295131331   411 0x7f47400c9d00 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.295583902   411 0x7f4741672990 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder0> decreasing timestamp (0:00:15.336157732 < 0:00:15.356672260)
0:00:35.295645626   411 0x7f4754030c50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:35.295792915   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f4754067d50 of size 64, PTS 0:00:15.356672260 and offset NONE
0:00:35.295826835   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:00:35.295843633   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:00:35.295861303   411 0x7f4741672990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:00:35.295872935   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:00:35.295892391   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:00:35.295903771   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f4754067d50, out 0x7f4741680010
0:00:35.295961005   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:00:35.295975652   411 0x7f4741672990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f475406c8b8 of API type GstVideoMetaAPI
0:00:35.297988801   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f4741680010 of size 64, PTS 0:00:15.356672260 and offset NONE
0:00:35.298006633   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:00:35.298024013   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f4741680010, out 0x7f4741680010
0:00:35.298033921   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:00:35.298079443   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f4741680010 of size 64, PTS 0:00:15.356672260 and offset NONE
0:00:35.298088344   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:00:35.298218134   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:00:35.298233142   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f4741680010, out 0x7f4741680010
0:00:35.298242062   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:00:35.298257839   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f4741680010 of size 64, PTS 0:00:15.356672260 and offset 183
0:00:35.298278876   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:00:35.298288074   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f4741680010, out 0x7f4741680010
0:00:35.298298494   411 0x7f4741672990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:00:35.298420962   411 0x7f4741672990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f4741680010 from source 0
0:00:35.298436183   411 0x7f4741672990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:00:35.301775672   411 0x7f47400c9d00 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1

[Fiona.Chen]

As you also mentioned RTX2080TI, can the same app with YoloV4 work well on RTX2080TI?

[awennersteen]

Correct, everything works fine on rtx2080ti (driver 450.36.06). Meaning, we’ve had it run for maybe 8 hours straight before shutting it down, as well as running it many times for shorter periods without encountering this issue.
I’ve also not encountered any issues on GTX1660 (driver 450.51.06) (laptop).

[awennersteen]

So I just decided to look at the probes, it appears that the pgie sink pad is being called after the appsink.
If everything was working as expected, the two locations with the “streammux sink pad called” should be followed by a call to the appsink.

logs (starts right after last succesfull appsink, ends when the pipeline “freezes”):
end of python appsink
0:01:11.324433706 541 0x61890a0 DEBUG basesink gstbasesink.c:3577:gst_base_sink_chain_unlocked: object unref after render 0x7f0af0066280
0:01:11.324455621 541 0x61890a0 DEBUG GST_BUFFER gstbuffer.c:2384:gst_buffer_foreach_meta: remove metadata 0x7f0b90015368 (NvDsMeta)
0:01:11.324548532 541 0x61890a0 DEBUG nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer: release_buffer
0:01:11.324561889 541 0x61890a0 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.324576624 541 0x61890a0 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.327049148 541 0x7f0b300c9900 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.330192174 541 0x7f0b300c9900 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.332639666 541 0x7f0b300c9900 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.335021765 541 0x7f0b300c9900 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.337221232 541 0x7f0b300c9900 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.339783479 541 0x7f0b300c9900 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.341873520 541 0x7f0b300c9900 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.384148994 541 0x7f0af0027c50 DEBUG basetransform gstbasetransform.c:1985:default_submit_input_buffer: handling buffer 0x7f0af008c330 of size 24290, PTS 0:00:51.699074997 and offset 25522524
0:01:11.384203202 541 0x7f0af0027c50 DEBUG basetransform gstbasetransform.c:2102:default_generate_output: calling prepare buffer
0:01:11.384217501 541 0x7f0af0027c50 DEBUG basetransform gstbasetransform.c:2109:default_generate_output: using allocated buffer in 0x7f0af008c330, out 0x7f0af008c330
0:01:11.384228282 541 0x7f0af0027c50 DEBUG basetransform gstbasetransform.c:2126:default_generate_output: doing inplace transform
0:01:11.384320280 541 0x7f0af0027c50 DEBUG GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.384693343 541 0x7f0b31671990 WARN videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame: decreasing timestamp (0:00:51.699074997 < 0:00:51.892751394)
0:01:11.384814300 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:1990:default_submit_input_buffer: handling buffer 0x7f0b3000aa00 of size 64, PTS 0:00:51.892751394 and offset NONE
0:01:11.384829767 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2102:default_generate_output: calling prepare buffer
0:01:11.384849180 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:1605:default_prepare_output_buffer: using pool alloc
0:01:11.384909196 541 0x7f0b31671990 DEBUG nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer: acquire_buffer
0:01:11.384923535 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:1745:default_copy_metadata: copying metadata
0:01:11.384959467 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:1712:foreach_metadata: not copying memory specific metadata GstVideoMetaAPI
0:01:11.384997019 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2109:default_generate_output: using allocated buffer in 0x7f0b3000aa00, out 0x7f0af00668e0
0:01:11.385023165 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2129:default_generate_output: doing non-inplace transform
0:01:11.385048128 541 0x7f0b31671990 DEBUG GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don’t copy memory meta 0x7f0af0061838 of API type GstVideoMetaAPI
0:01:11.385204829 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:1990:default_submit_input_buffer: handling buffer 0x7f0af00668e0 of size 64, PTS 0:00:51.892751394 and offset NONE
0:01:11.385217405 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2102:default_generate_output: calling prepare buffer
0:01:11.385230934 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2109:default_generate_output: using allocated buffer in 0x7f0af00668e0, out 0x7f0af00668e0
0:01:11.385246957 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2126:default_generate_output: doing inplace transform
0:01:11.385261704 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:1990:default_submit_input_buffer: handling buffer 0x7f0af00668e0 of size 64, PTS 0:00:51.892751394 and offset NONE
0:01:11.385280681 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2102:default_generate_output: calling prepare buffer
0:01:11.385291940 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:1617:default_prepare_output_buffer: inplace reuse writable input buffer
0:01:11.385316636 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2109:default_generate_output: using allocated buffer in 0x7f0af00668e0, out 0x7f0af00668e0
0:01:11.385338554 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2126:default_generate_output: doing inplace transform
0:01:11.385364532 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:1985:default_submit_input_buffer: handling buffer 0x7f0af00668e0 of size 64, PTS 0:00:51.892751394 and offset 620
0:01:11.385383722 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2102:default_generate_output: calling prepare buffer
0:01:11.385403476 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2109:default_generate_output: using allocated buffer in 0x7f0af00668e0, out 0x7f0af00668e0
0:01:11.385419464 541 0x7f0b31671990 DEBUG basetransform gstbasetransform.c:2126:default_generate_output: doing inplace transform
0:01:11.385439918 541 0x7f0b31671990 LOG nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain: Got buffer 0x7f0af00668e0 from source 0
0:01:11.385467593 541 0x7f0b31671990 DEBUG nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer: acquire_buffer
0:01:11.385619746 541 0x7f0b30053990 TRACE nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers: SETTING CUDA DEVICE = 0 in nvstreammux func=gst_nvstreammux_src_collect_buffers
0:01:11.385642646 541 0x7f0b30053990 DEBUG GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:01:11.385671461 541 0x7f0b30053990 LOG nvstreammux_ntp gstnvstreammux_ntp.cpp:212:gst_nvds_ntp_calculator_get_buffer_ntp: Cannot calculate frame NTP. mode: RTCP SR. source 0: PTS:0:00:51.892751394(51892751394) NTP: 1970-01-01T00:00:00.000Z(0). ntp_time_epoch_ns = 1970-01-01T00:00:00.000Z(0) ntp_frame_timestamp = 0:00:00.000000000(0) System Time: 2020-11-18T08:09:19.414Z
0:01:11.385708005 541 0x7f0b30053990 DEBUG nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop: Pushing buffer 0x7f0b3000a5c0, batch size 1, PTS 0:00:51.716247653
0:01:11.385718414 541 0x7f0b30053990 DEBUG nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop: STREAMMUX OUT BUFFER attached timestamp 0:00:51.716247653

streammux sink pad called

0:01:11.386011305   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2323:gst_base_transform_chain:<videorate0> we got return custom-success
0:01:11.386030882   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:01:11.386059640   541 0x7f0b31671990 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.386071497   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2332:gst_base_transform_chain:<videorate0> dropped a buffer, marking DISCONT
0:01:11.386120739   541 0x7f0b30053990 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f0b30008368 (NvDsMeta) of size 72
0:01:11.386150805   541 0x7f0b30053990 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:01:11.386312137   541 0x7f0b300c9900 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.465263335   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f0af00747e0 of size 18888, PTS 0:00:51.595914949 and offset 25546814
0:01:11.465288849   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:01:11.465302412   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f0af00747e0, out 0x7f0af00747e0
0:01:11.465313529   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:01:11.465380120   541 0x7f0af0027c50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.465745016   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f0af00669f0 of size 64, PTS 0:00:51.972751394 and offset NONE
0:01:11.465771219   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:01:11.465784579   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:01:11.465810803   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:01:11.465836774   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:01:11.466027824   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:01:11.466061338   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f0af00669f0, out 0x7f0af0076d70
0:01:11.466082498   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:01:11.466101757   541 0x7f0b31671990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f0af00846d8 of API type GstVideoMetaAPI
0:01:11.466329925   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f0af0076d70 of size 64, PTS 0:00:51.972751394 and offset NONE
0:01:11.466343436   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:01:11.466354131   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f0af0076d70, out 0x7f0af0076d70
0:01:11.466363724   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:01:11.466379710   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f0af0076d70 of size 64, PTS 0:00:51.972751394 and offset NONE
0:01:11.466388391   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:01:11.466464700   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:01:11.466506140   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f0af0076d70, out 0x7f0af0076d70
0:01:11.466540871   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:01:11.466577054   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f0af0076d70 of size 64, PTS 0:00:51.972751394 and offset 621
0:01:11.466589311   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:01:11.466598647   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f0af0076d70, out 0x7f0af0076d70
0:01:11.466615868   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:01:11.466631700   541 0x7f0b31671990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f0af0076d70 from source 0
0:01:11.466640954   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:01:11.466795946   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2323:gst_base_transform_chain:<videorate0> we got return custom-success
0:01:11.466803383   541 0x7f0b30053990 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:01:11.466806141   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:01:11.466853993   541 0x7f0b31671990 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.466866902   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2332:gst_base_transform_chain:<videorate0> dropped a buffer, marking DISCONT
0:01:11.467009166   541 0x7f0b30053990 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:01:11.467040844   541 0x7f0b30053990 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:212:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Cannot calculate frame NTP. mode: RTCP SR. source 0: PTS:0:00:51.972751394(51972751394) NTP: 1970-01-01T00:00:00.000Z(0). ntp_time_epoch_ns = 1970-01-01T00:00:00.000Z(0) ntp_frame_timestamp = 0:00:00.000000000(0) System Time: 2020-11-18T08:09:19.496Z
0:01:11.467076848   541 0x7f0b30053990 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f0af0066390, batch size 1, PTS 0:00:51.797616931
0:01:11.467086964   541 0x7f0b30053990 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:51.797616931

streammux sink pad called

0:01:11.467350007   541 0x7f0b30053990 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f0ac802d008 (NvDsMeta) of size 72
0:01:11.467366549   541 0x7f0b30053990 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:01:11.467419769   541 0x7f0b300c9900 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.546187207   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f0af009f770 of size 12172, PTS 0:00:51.723432982 and offset 25565702
0:01:11.546208877   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:01:11.546222197   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f0af009f770, out 0x7f0af009f770
0:01:11.546245203   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:01:11.546302268   541 0x7f0af0027c50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.546777014   541 0x7f0b31671990 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder0> decreasing timestamp (0:00:51.723432982 < 0:00:51.972751394)
0:01:11.546802970   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f0b3000ad30 of size 64, PTS 0:00:51.972751394 and offset NONE
0:01:11.546813870   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:01:11.546824149   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:01:11.546833743   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:01:11.546843214   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:01:11.546855480   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:01:11.546865106   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f0b3000ad30, out 0x7f0af0076600
0:01:11.546875623   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:01:11.546891984   541 0x7f0b31671990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f0af0084498 of API type GstVideoMetaAPI
0:01:11.547001602   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f0af0076600 of size 64, PTS 0:00:51.972751394 and offset NONE
0:01:11.547010186   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:01:11.547020917   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f0af0076600, out 0x7f0af0076600
0:01:11.547030002   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:01:11.547041740   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f0af0076600 of size 64, PTS 0:00:51.972751394 and offset NONE
0:01:11.547051429   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:01:11.547062104   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:01:11.547073560   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f0af0076600, out 0x7f0af0076600
0:01:11.547084840   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:01:11.547104970   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f0af0076600 of size 64, PTS 0:00:51.972751394 and offset 622
0:01:11.547115672   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:01:11.547124876   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f0af0076600, out 0x7f0af0076600
0:01:11.547134693   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:01:11.547151511   541 0x7f0b31671990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f0af0076600 from source 0
0:01:11.547163092   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:01:11.547273936   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2323:gst_base_transform_chain:<videorate0> we got return custom-success
0:01:11.547282574   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:492:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:01:11.547291300   541 0x7f0b31671990 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.547307615   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2332:gst_base_transform_chain:<videorate0> dropped a buffer, marking DISCONT
0:01:11.547296601   541 0x7f0b30053990 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers
0:01:11.547364172   541 0x7f0b30053990 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:01:11.547529511   541 0x7f0b30053990 LOG          nvstreammux_ntp gstnvstreammux_ntp.cpp:212:gst_nvds_ntp_calculator_get_buffer_ntp:<Stream-muxer> Cannot calculate frame NTP. mode: RTCP SR. source 0: PTS:0:00:51.972751394(51972751394) NTP: 1970-01-01T00:00:00.000Z(0). ntp_time_epoch_ns = 1970-01-01T00:00:00.000Z(0) ntp_frame_timestamp = 0:00:00.000000000(0) System Time: 2020-11-18T08:09:19.576Z
0:01:11.547575292   541 0x7f0b30053990 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f0af0066280, batch size 1, PTS 0:00:51.878114849
0:01:11.547587346   541 0x7f0b30053990 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:51.878114849

streammux sink pad called

0:01:11.547918084   541 0x7f0b300c9900 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.622478373   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f0af0081600 of size 153014, PTS 0:00:52.060206030 and offset 25577874
0:01:11.622499451   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:01:11.622511198   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f0af0081600, out 0x7f0af0081600
0:01:11.622520345   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform
0:01:11.622595139   541 0x7f0af0027c50 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:11.623178616   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f0af0066b00 of size 64, PTS 0:00:52.060206030 and offset NONE
0:01:11.623195042   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:01:11.623205440   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:01:11.623215809   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:01:11.623227243   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:01:11.623239412   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1712:foreach_metadata:<nvvideoconvert0> not copying memory specific metadata GstVideoMetaAPI
0:01:11.623247973   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f0af0066b00, out 0x7f0af00666c0
0:01:11.623255020   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:01:11.623270252   541 0x7f0b31671990 DEBUG             GST_BUFFER gstbuffer.c:630:gst_buffer_copy_into: don't copy memory meta 0x7f0af00845b8 of API type GstVideoMetaAPI
0:01:11.623395198   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<capsfilter0> handling buffer 0x7f0af00666c0 of size 64, PTS 0:00:52.060206030 and offset NONE
0:01:11.623409010   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter0> calling prepare buffer
0:01:11.623422403   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter0> using allocated buffer in 0x7f0af00666c0, out 0x7f0af00666c0
0:01:11.623432036   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter0> doing inplace transform
0:01:11.623455850   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videorate0> handling buffer 0x7f0af00666c0 of size 64, PTS 0:00:52.060206030 and offset NONE
0:01:11.623467657   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videorate0> calling prepare buffer
0:01:11.623478157   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1617:default_prepare_output_buffer:<videorate0> inplace reuse writable input buffer
0:01:11.623489295   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videorate0> using allocated buffer in 0x7f0af00666c0, out 0x7f0af00666c0
0:01:11.623498214   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<videorate0> doing inplace transform
0:01:11.623519694   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter1> handling buffer 0x7f0af00666c0 of size 64, PTS 0:00:52.060206030 and offset 623
0:01:11.623530447   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter1> calling prepare buffer
0:01:11.623541388   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter1> using allocated buffer in 0x7f0af00666c0, out 0x7f0af00666c0
0:01:11.623549403   541 0x7f0b31671990 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter1> doing inplace transform
0:01:11.623568670   541 0x7f0b31671990 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f0af00666c0 from source 0
0:01:11.623581542   541 0x7f0b31671990 DEBUG               nvdspool gstnvdsbufferpool.c:442:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:01:11.699619833   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:1985:default_submit_input_buffer:<capsfilter2> handling buffer 0x7f0af0092410 of size 25743, PTS 0:00:51.869812132 and offset 25730888
0:01:11.699642493   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<capsfilter2> calling prepare buffer
0:01:11.699653039   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<capsfilter2> using allocated buffer in 0x7f0af0092410, out 0x7f0af0092410
0:01:11.699661405   541 0x7f0af0027c50 DEBUG          basetransform gstbasetransform.c:2126:default_generate_output:<capsfilter2> doing inplace transform

[Fiona.Chen]

@awennersteen Can you help to confirm the GPU driver version and TensorRT version in your T4 and RTX2080TI seperately? Are they just the same?

[awennersteen]

We’ve tried on the following T4 drivers, all with the same result
450.80.02 and 440.33 and 440.64
Although all the logs herein are from the 450.80.02 version

And for RTX2080TI only this driver
450.36.06

In other words, no overlap of drivers. ( I can change the T4 driver version without issues, but the RTX2080ti I can’t easily /quickly change)

[Fiona.Chen]

Can you monitor the GPU status with “nvidia-smi dmon” while runing the app. Is there any change before and after T?

[awennersteen]

Memory is unchanged, but gpu compute and decode goes down 0-1%.
I can’t tell if it happens exactly at T, or slightly later. (recall the logs showing nvinfer/nvstreammux activity slightly longer than appsink, I can’t tell when it fails)

[Fiona.Chen]

@awennersteen

After T the logs keep on going for other parts of the pipeline though. the rtspsrc element appears to keep on going, and the logs from GST_MEMORY show that memory is continuously allocated but not freed.

What is the GPU status when the application is in the above state? GPU loading is always 0 after T, never back?

[awennersteen]

GPU stays at 0.
At most I’ve left it for maybe 20-30 minutes without seeing any change

[Fiona.Chen]

Do you mean GPU loading keep 0 while the rtspsrc is still working?
Can get a complete log with only “GST_DEBUG=nvstreammux:7,v4l2videodec:7”?

[awennersteen]

@Fiona.Chen

Yes, that is what I mean.

Here are the logs, it would appear I’m getting no logs from v4l2videodec

0:00:17.470287803   453      0x5db6e00 DEBUG            nvstreammux gstnvstreammux.c:1177:gst_nvstreammux_request_new_pad:<Stream-muxer> Requesting new sink pad
Unknown or legacy key specified 'is-classifier' for group [property]
0:00:17.497724938   453      0x5db6e00 TRACE            nvstreammux gstnvstreammux.c:2274:gst_nvstreammux_change_state:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux func=gst_nvstreammux_change_state

0:00:18.512460650   453      0x5db6e00 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<ReID> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 2]: deserialized trt engine from :/cja-models/reid/reid_dynamic_1-8x3x64x64.engine
0:00:18.512538388   453      0x5db6e00 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<ReID> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 2]: Use deserialized engine model: /cja-models/reid/reid_dynamic_1-8x3x64x64.engine
0:00:18.513180266   453      0x5db6e00 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<ReID> [UID 2]: Load new model:/cja-models/reid/config_infer_secondary_reid.txt sucessfully
0:00:19.439417610   453      0x5db6e00 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<YoloV4> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/cja-models/tensorrt/deepstream_yolov4/yolov4_bodyparts_1-8_3_608_608_dynamic_fp16.engine
0:00:19.439497910   453      0x5db6e00 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<YoloV4> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /cja-models/tensorrt/deepstream_yolov4/yolov4_bodyparts_1-8_3_608_608_dynamic_fp16.engine
0:00:19.446270407   453      0x5db6e00 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<YoloV4> [UID 1]: Load new model:/cja-models/tensorrt/deepstream_yolov4/config_infer_primary_yoloV4_bodyparts.txt sucessfully
0:00:19.446324962   453      0x5db6e00 TRACE            nvstreammux gstnvstreammux.c:2274:gst_nvstreammux_change_state:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux func=gst_nvstreammux_change_state

0:00:19.453462758   453      0x5f8f230 TRACE            nvstreammux gstnvstreammux.c:2274:gst_nvstreammux_change_state:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux func=gst_nvstreammux_change_state

0:00:19.838833723   453 0x7f7b6c028050 DEBUG            nvstreammux gstnvstreammux.c:975:gst_nvstreammux_sink_event:<Stream-muxer> parse video info from caps video/x-raw(memory:NVMM), width=(int)1080, height=(int)608, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1216/1215, framerate=(fraction)12/1, format=(string)RGBA
0:00:19.839232840   453 0x7f7b6c028050 INFO             nvstreammux gstnvstreammux.c:1091:gst_nvstreammux_sink_event:<Stream-muxer> Conversion enabled for source 0
0:00:19.839520447   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:19.841985143   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:19.844521867   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:19.849159738   453 0x7f7b5965cd90 DEBUG            nvstreammux gstnvstreammux.c:651:gst_nvstreammux_alloc_output_buffers:  Output buffer pool (0x7f7b5965eb40) successfully created with 4 buffers
0:00:19.849324642   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:19.849427902   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:19.849472712   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:1878:gst_nvstreammux_src_collect_buffers:<Stream-muxer> Pad added event sent 0

0:00:19.849618381   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:00.396228840
0:00:19.849627942   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.396228840
0:00:19.850247784   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:19.850287746   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:00.396899494
0:00:19.850300301   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.396899494
0:00:19.850537563   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:19.877467070   453 0x7f7b5965d770 DEBUG            nvstreammux gstnvstreammux.c:921:gst_nvstreammux_src_event:<Stream-muxer> latency 0:00:00.000000000
0:00:19.955381350   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:19.955685985   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:19.955817522   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:00.502422358
0:00:19.955846550   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.502422358
0:00:19.956954526   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.003906673   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:20.004081247   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.004176766   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:00.550786165
0:00:20.004233416   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.550786165
0:00:20.006479237   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.083732370   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:20.083980040   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.084062078   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:00.630668638
0:00:20.084110171   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.630668638
0:00:20.084396192   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.164600787   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:20.164746832   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.164805052   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:00.711402698
0:00:20.164817298   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.711402698
0:00:20.165067058   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.245349084   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:20.245494664   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.245542583   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:00.792153856
0:00:20.245553957   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.792153856
0:00:20.245769299   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.337607463   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:20.337792273   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.337894510   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:00.884502538
0:00:20.337939544   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.884502538
0:00:20.338202259   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.402305148   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:20.402471337   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.402523193   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:00.949131964
0:00:20.402536341   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:00.949131964
0:00:20.402896182   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.482631531   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:20.482783341   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.482837614   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:01.029446310
0:00:20.482847941   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.029446310
0:00:20.483073852   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.563566706   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:20.563713680   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.563768359   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:01.110375938
0:00:20.563781193   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.110375938
0:00:20.564105756   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.646855883   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:20.647011269   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.647166449   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:01.193751992
0:00:20.647184128   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.193751992
0:00:20.647668703   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.726353586   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:20.726695495   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.726760919   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:01.273368109
0:00:20.726774715   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.273368109
0:00:20.727146932   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.807166888   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:20.807335749   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.807420252   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:01.354025218
0:00:20.807440188   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.354025218
0:00:20.807662686   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.878435505   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:20.878609721   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.878657845   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:01.425266660
0:00:20.878667711   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.425266660
0:00:20.879024489   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.963533055   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:20.963688474   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:20.963754179   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:01.510362700
0:00:20.963769481   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.510362700
0:00:20.964017890   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.042761299   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:21.042923620   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.042984218   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:01.589593793
0:00:21.042996510   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.589593793
0:00:21.043335584   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.122997072   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:21.123153968   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.123292911   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:01.669900181
0:00:21.123307751   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.669900181
0:00:21.123580389   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.205064761   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:21.205224804   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.205401716   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:01.752007595
0:00:21.205416004   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.752007595
0:00:21.205693663   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.287865267   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:21.288019501   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.288077016   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:01.834684429
0:00:21.288092985   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.834684429
0:00:21.288399886   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.368568287   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:21.368760685   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.368852932   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:01.915459362
0:00:21.368866229   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.915459362
0:00:21.369206236   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.448950660   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:21.449101661   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.449164300   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:01.995771203
0:00:21.449181131   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:01.995771203
0:00:21.449436450   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.519875553   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:21.520028530   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.520077401   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:02.066685980
0:00:21.520091084   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.066685980
0:00:21.520389755   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.603845394   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:21.603988886   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.604030119   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:02.150640396
0:00:21.604042159   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.150640396
0:00:21.604391290   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.683453144   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:21.683608563   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.683656611   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:02.230265555
0:00:21.683668674   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.230265555
0:00:21.684000055   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.763599328   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:21.763747706   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.763821710   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:02.310403969
0:00:21.763838126   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.310403969
0:00:21.764129691   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.845329826   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:21.845480646   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.845561102   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:02.392143118
0:00:21.845575206   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.392143118
0:00:21.845842284   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.928621936   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:21.928774407   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:21.928966570   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:02.475570128
0:00:21.928990716   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.475570128
0:00:21.929282528   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.007633282   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:22.007790160   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.007849283   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:02.554458084
0:00:22.007864556   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.554458084
0:00:22.008095668   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.078632582   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:22.078782209   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.078842438   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:02.625450202
0:00:22.078857018   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.625450202
0:00:22.079106273   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.159297564   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:22.159449718   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.159495896   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:02.706105372
0:00:22.159509760   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.706105372
0:00:22.159782728   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.245558099   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:22.245699390   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.245816512   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:02.792413301
0:00:22.245835969   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.792413301
0:00:22.246148781   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.322356380   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:22.322513949   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.322562491   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:02.869170661
0:00:22.322575003   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.869170661
0:00:22.322914059   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.403394345   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:22.403557872   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.403608182   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:02.950218664
0:00:22.403619113   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:02.950218664
0:00:22.403878502   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.484495355   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:22.484766644   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.484817274   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:03.031425610
0:00:22.484878504   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.031425610
0:00:22.485124272   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.566657329   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:22.566816869   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.566867992   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:03.113477174
0:00:22.566881008   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.113477174
0:00:22.567099515   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.647026566   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:22.647185990   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.647248232   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:03.193857621
0:00:22.647263652   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.193857621
0:00:22.647505619   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.718076362   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:22.718226135   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.718277266   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:03.264885193
0:00:22.718285661   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.264885193
0:00:22.718551134   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.799240758   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:22.799403111   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.799470880   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:03.346077740
0:00:22.799485634   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.346077740
0:00:22.799738575   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.894554017   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:22.894711415   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.894789384   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:03.441395945
0:00:22.894805827   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.441395945
0:00:22.895059302   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.964480829   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:22.964631521   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:22.964692183   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:03.511299070
0:00:22.964706009   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.511299070
0:00:22.964989260   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.047757981   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:23.047919611   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.048082615   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:03.594640405
0:00:23.048203920   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.594640405
0:00:23.049037796   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.127515533   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:23.127689150   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.127807494   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:03.674412674
0:00:23.127834114   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.674412674
0:00:23.128178192   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.199744097   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:23.207823402   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.207878724   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:03.754487509
0:00:23.207901964   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.754487509
0:00:23.208175053   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.282462858   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:23.283996586   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.284061654   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:03.830670463
0:00:23.284074496   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:03.830670463
0:00:23.285364517   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.773400622   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:23.773667008   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.773828111   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:04.320400920
0:00:23.773911428   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:04.320400920
0:00:23.874936504   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.885034092   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:23.885173226   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.885235409   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:04.431842081
0:00:23.885254652   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:04.431842081
0:00:23.885464839   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:23.996276764   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.996367598   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:23.996415279   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:04.543024321
0:00:23.996451918   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:04.543024321
0:00:23.996778710   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:24.077266393   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.077432970   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.077522819   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:04.624131339
0:00:24.077547430   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:04.624131339
0:00:24.078189340   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:24.160827516   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.161160774   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.161235346   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:04.707844135
0:00:24.161250552   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:04.707844135
0:00:24.176213643   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:24.243976171   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.244329848   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.244596808   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:04.791202344
0:00:24.244626307   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:04.791202344
0:00:24.262208100   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:24.325569991   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.325757226   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.325827802   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:04.872435769
0:00:24.325838819   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:04.872435769
0:00:24.326118966   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:24.407526638   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.407612022   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:04.954220901
0:00:24.407629381   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:04.954220901
0:00:24.429456102   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:24.490517424   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.490579498   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:05.037188722
0:00:24.490595470   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.037188722
0:00:24.490834143   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:24.573387379   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.573548040   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.573612252   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:05.120221180
0:00:24.573628900   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.120221180
0:00:24.593957520   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:24.655095170   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.655310081   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.655363779   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:05.201972066
0:00:24.655379476   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.201972066
0:00:24.656946402   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:24.737429816   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.737643883   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.737717825   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:05.284309008
0:00:24.737735423   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.284309008
0:00:24.737980727   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:24.818805358   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.819219616   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.819369853   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:24.819654701   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:05.366145025
0:00:24.819888285   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.366145025
0:00:24.899285074   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.899550227   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.899618249   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:05.446215751
0:00:24.899634287   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.446215751
0:00:24.900623893   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:24.981000090   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.981173284   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:24.981312796   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:05.527911125
0:00:24.981403830   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.527911125
0:00:24.995585461   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:25.061559647   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.061747220   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.061929195   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:05.608535045
0:00:25.061965863   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.608535045
0:00:25.081763538   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0
0:00:25.144239215   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.144458977   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.144514593   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800be90, batch size 1, PTS 0:00:05.691122300
0:00:25.144530859   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.691122300
0:00:25.144760044   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b780 from source 0
0:00:25.226220228   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.226360320   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.226404277   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bc70, batch size 1, PTS 0:00:05.773012938
0:00:25.226416929   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.773012938
0:00:25.226695605   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b6c09b890 from source 0
0:00:25.309215757   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.309385027   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.310431770   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800ba50, batch size 1, PTS 0:00:05.857037065
0:00:25.310596453   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.857037065
0:00:25.311047182   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b5800bb60 from source 0
0:00:25.391016842   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.391190100   453 0x7f7b58055190 TRACE            nvstreammux gstnvstreammux.c:1572:gst_nvstreammux_src_collect_buffers:<Stream-muxer> SETTING CUDA DEVICE = 0 in nvstreammux  func=gst_nvstreammux_src_collect_buffers

0:00:25.391258727   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2196:gst_nvstreammux_src_push_loop:<Stream-muxer> Pushing buffer 0x7f7b5800bd80, batch size 1, PTS 0:00:05.937848793
0:00:25.391275162   453 0x7f7b58055190 DEBUG            nvstreammux gstnvstreammux.c:2218:gst_nvstreammux_src_push_loop:<Stream-muxer> STREAMMUX OUT BUFFER attached timestamp 0:00:05.937848793
0:00:25.391322048   453 0x7f7b5965cd90 LOG              nvstreammux gstnvstreammux.c:362:gst_nvstreammux_chain:<Stream-muxer> Got buffer 0x7f7b596664b0 from source 0

[Fiona.Chen]

The log is from 0:00:17.470287803 to 0:00:25.391322048, can you provide the whole log?

[awennersteen]

It’s the start of the log, I think it starts there because we run the init of gstreamer quite some time before pushing the pipeline to PLAY.

The log before this comes from our Python application