Adding NvInfer to my pipeline is causing segfault

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson AGX Xavier
• DeepStream Version 5.1
• JetPack Version (valid for Jetson only) 4.5.1
• TensorRT Version 7.1.3

Hello! I have manually formed a batch from single source using custom my plugin named batcher. With a tiler and sink, it works without any issue but when I add nvinfer it is giving segfault after running for some time.

Pipeline
  gst_bin_add_many (GST_BIN (pipeline), source, bayer2rgb, videoconvert,
          nvvideoconvert, nvdewarper, batcher, NULL);
  if (!gst_element_link_many (source, bayer2rgb, videoconvert, nvvideoconvert, nvdewarper, batcher, NULL)) {
    g_printerr ("Elements could not be linked. Exiting.\n");
    return -1;
  }

  gst_bin_add_many (GST_BIN (pipeline), pgie, tiler, sink, NULL);
  if (!gst_element_link_many (batcher, pgie, tiler, sink, NULL)) {
    g_printerr ("Elements could not be linked 1. Exiting.\n");
    return -1;
  }
Error log:
0:01:09.308594481 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<batcher0:sink> calling chainfunction &gst_base_transform_chain with buffer buffer: 0x55c372ba40, pts 0:01:05.203018630, dts 99:99:99.999999999, dur 0:00:00.019608352, size 64, offset none, offset_end none, flags 0x40
0:01:09.308607058 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2273:gst_base_transform_chain:<batcher0> got DISCONT buffer 0x55c372ba40
0:01:09.308620179 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<batcher0> handling buffer 0x55c372ba40 of size 64, PTS 0:01:05.203018630 and offset NONE
0:01:09.308636339 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<batcher0> calling prepare buffer
0:01:09.308826620 21414   0x55c486f0f0 DEBUG             GST_MEMORY gstmemory.c:139:gst_memory_init: new memory 0x7f073fd8a0, maxsize:64 offset:0 size:64
0:01:09.308871935 21414   0x55c486f0f0 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f14024308 (NvDsMeta) of size 72
0:01:09.308888767 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<batcher0> using allocated buffer in 0x55c372ba40, out 0x7f00afa6b0
0:01:09.308899616 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<batcher0> doing non-inplace transform
0:01:09.308919489 21414   0x55c486f0f0 DEBUG                batcher gstbatcher.cpp:504:gst_batcher_transform:<batcher0> Processing Frame 0 Surface 0x7f06fcfdf0

0:01:09.311849708 21414   0x55947664a0 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f073f2528 (NvDsMeta) of size 72
0:01:09.311921072 21414   0x55947664a0 DEBUG             GST_MEMORY gstmemory.c:88:_gst_memory_free: free memory 0x7f073fd9c0
0:01:09.311944881 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:2311:gst_base_transform_chain:<nvtiler> we have a pending DISCONT
0:01:09.311970450 21414   0x55947664a0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<nvoverlaysink-nvoverlaysink0:sink> calling chainfunction &gst_base_sink_chain with buffer buffer: 0x55c372bd70, pts 0:01:05.163802118, dts 99:99:99.999999999, dur 0:00:00.019608600, size 64, offset none, offset_end none, flags 0x40
0:01:09.311994771 21414   0x55947664a0 DEBUG               basesink gstbasesink.c:3421:gst_base_sink_chain_unlocked:<nvoverlaysink-nvoverlaysink0> got times start: 0:01:05.163802118, end: 0:01:05.183410718
0:01:09.312014196 21414   0x55947664a0 DEBUG               basesink gstbasesink.c:2006:gst_base_sink_get_sync_times:<nvoverlaysink-nvoverlaysink0> got times start: 0:01:05.163802118, stop: 0:01:05.183410718, do_sync 1
0:01:09.312029397 21414   0x55947664a0 DEBUG               basesink gstbasesink.c:2591:gst_base_sink_do_sync:<nvoverlaysink-nvoverlaysink0> reset rc_time to time 0:01:05.163802118
0:01:09.312045206 21414   0x55947664a0 DEBUG               basesink gstbasesink.c:2603:gst_base_sink_do_sync:<nvoverlaysink-nvoverlaysink0> possibly waiting for clock to reach 0:01:05.163802118, adjusted 0:01:05.163802118
0:01:09.312059030 21414   0x55947664a0 DEBUG              GST_CLOCK gstclock.c:537:gst_clock_id_wait:<GstSystemClock> waiting on clock entry 0x7ee8005de0
0:01:09.312075831 21414   0x55947664a0 DEBUG              GST_CLOCK gstclock.c:1046:gst_clock_get_internal_time:<GstSystemClock> internal time 30:30:47.989362348
0:01:09.312089784 21414   0x55947664a0 DEBUG              GST_CLOCK gstclock.c:1091:gst_clock_get_time:<GstSystemClock> adjusted time 30:30:47.989362348
0:01:09.312131066 21414   0x55947664a0 DEBUG              GST_CLOCK gstsystemclock.c:719:gst_system_clock_id_wait_jitter_unlocked: entry 0x7ee8005de0 time 30:30:47.931396359 now 30:30:47.989362348 diff (time-now) -57965989
0:01:09.312145211 21414   0x55947664a0 DEBUG              GST_CLOCK gstclock.c:547:gst_clock_id_wait:<GstSystemClock> done waiting entry 0x7ee8005de0, res: 1 (early)
0:01:09.312162683 21414   0x55947664a0 DEBUG               basesink gstbasesink.c:2610:gst_base_sink_do_sync:<nvoverlaysink-nvoverlaysink0> clock returned 1, jitter  0:00:00.057965989
0:01:09.312187709 21414   0x55947664a0 DEBUG        GST_PERFORMANCE gstbasesink.c:2894:gst_base_sink_is_too_late:<nvoverlaysink-nvoverlaysink0> buffer is too late 0:01:05.221768107 > 0:01:05.203410718
0:01:09.312199965 21414   0x55947664a0 DEBUG               basesink gstbasesink.c:3622:gst_base_sink_chain_unlocked:<nvoverlaysink-nvoverlaysink0> buffer late, dropping
0:01:09.312219742 21414   0x55947664a0 DEBUG                GST_QOS gstbasesink.c:3636:gst_base_sink_chain_unlocked:<nvoverlaysink-nvoverlaysink0> qos: dropped buffer rt 0:01:05.163802118, st 0:01:05.163802118, ts 0:01:05.163802118, dur 0:00:00.019608600
0:01:09.312232351 21414   0x55947664a0 DEBUG                GST_QOS gstbasesink.c:3639:gst_base_sink_chain_unlocked:<nvoverlaysink-nvoverlaysink0> qos: rendered 64, dropped 1659
0:01:09.312318147 21414   0x55947664a0 DEBUG                GST_BUS gstbus.c:318:gst_bus_post:<bus0> [msg 0x7ee8008400] posting on bus qos message: 0x7ee8008400, time 99:99:99.999999999, seq-num 5109, element 'nvoverlaysink-nvoverlaysink0', GstMessageQOS, live=(boolean)true, running-time=(guint64)65163802118, stream-time=(guint64)65163802118, timestamp=(guint64)65163802118, duration=(guint64)19608600, jitter=(gint64)57965989, proportion=(double)0.99089311022987914, quality=(int)1000000, format=(GstFormat)GST_FORMAT_BUFFERS, processed=(guint64)64, dropped=(guint64)1659;
0:01:09.312336772 21414   0x55947664a0 DEBUG                    bin gstbin.c:3717:gst_bin_handle_message_func:<object-tracker-pipeline> [msg 0x7ee8008400] handling child nvoverlaysink-nvoverlaysink0 message of type qos
0:01:09.312348772 21414   0x55947664a0 DEBUG                    bin gstbin.c:4054:gst_bin_handle_message_func:<object-tracker-pipeline> posting message upward
0:01:09.312380678 21414   0x55947664a0 DEBUG                GST_BUS gstbus.c:318:gst_bus_post:<bus1> [msg 0x7ee8008400] posting on bus qos message: 0x7ee8008400, time 99:99:99.999999999, seq-num 5109, element 'nvoverlaysink-nvoverlaysink0', GstMessageQOS, live=(boolean)true, running-time=(guint64)65163802118, stream-time=(guint64)65163802118, timestamp=(guint64)65163802118, duration=(guint64)19608600, jitter=(gint64)57965989, proportion=(double)0.99089311022987914, quality=(int)1000000, format=(GstFormat)GST_FORMAT_BUFFERS, processed=(guint64)64, dropped=(guint64)1659;
0:01:09.312394054 21414   0x55947664a0 DEBUG                GST_BUS gstbus.c:357:gst_bus_post:<bus1> [msg 0x7ee8008400] pushing on async queue
0:01:09.312423592 21414   0x55947664a0 DEBUG                GST_BUS gstbus.c:360:gst_bus_post:<bus1> [msg 0x7ee8008400] pushed on async queue
0:01:09.312437032 21414   0x55947664a0 DEBUG                GST_BUS gstbus.c:353:gst_bus_post:<bus0> [msg 0x7ee8008400] dropped
0:01:09.312460106 21414   0x55947664a0 DEBUG                GST_QOS gstbasesink.c:2751:gst_base_sink_perform_qos:<nvoverlaysink-nvoverlaysink0> start: 0:01:05.163802118, stop 0:01:05.183410718, entered 0:01:05.221768107, left 0:01:05.221768107, pt: 0:00:00.035877005, duration 0:00:00.037632334,jitter 57965989
0:01:09.312479403 21414   0x55947664a0 DEBUG                GST_QOS gstbasesink.c:2755:gst_base_sink_perform_qos:<nvoverlaysink-nvoverlaysink0> avg_pt: 0:00:00.037065313, avg_rate: 0.990893
0:01:09.312494219 21414   0x5593ba3e00 DEBUG                GST_BUS gstbus.c:546:gst_bus_timed_pop_filtered:<bus1> got message 0x7ee8008400, qos from nvoverlaysink-nvoverlaysink0, type mask is 4294967295
0:01:09.312564879 21414   0x5593ba3e00 DEBUG                GST_BUS gstbus.c:839:gst_bus_source_dispatch:<bus1> source 0x5594764fd0 calling dispatch with qos message: 0x7ee8008400, time 99:99:99.999999999, seq-num 5109, element 'nvoverlaysink-nvoverlaysink0', GstMessageQOS, live=(boolean)true, running-time=(guint64)65163802118, stream-time=(guint64)65163802118, timestamp=(guint64)65163802118, duration=(guint64)19608600, jitter=(gint64)57965989, proportion=(double)0.99089311022987914, quality=(int)1000000, format=(GstFormat)GST_FORMAT_BUFFERS, processed=(guint64)64, dropped=(guint64)1659;
0:01:09.312597072 21414   0x5593ba3e00 DEBUG                GST_BUS gstbus.c:844:gst_bus_source_dispatch:<bus1> source 0x5594764fd0 handler returns 1
0:01:09.312496267 21414   0x55947664a0 DEBUG                GST_QOS gstbasesink.c:2785:gst_base_sink_perform_qos:<nvoverlaysink-nvoverlaysink0> updated: avg_pt: 0:00:00.036916774, avg_rate: 0.980986
0:01:09.312653555 21414   0x55947664a0 DEBUG                GST_QOS gstbasesink.c:2679:gst_base_sink_send_qos:<nvoverlaysink-nvoverlaysink0> qos: type 1, proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.312674708 21414   0x55947664a0 DEBUG              GST_EVENT gstevent.c:306:gst_event_new_custom: creating new event 0x7f145af0f0 qos 48641
0:01:09.312708821 21414   0x55947664a0 DEBUG              GST_EVENT gstpad.c:5693:gst_pad_send_event_unchecked:<nvtiler:src> have event type qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.312739447 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1941:gst_base_transform_src_eventfunc:<nvtiler> handling event 0x7f145af0f0 qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.312787865 21414   0x55947664a0 DEBUG                GST_QOS gstbasetransform.c:2645:gst_base_transform_update_qos:<nvtiler> qos: proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.312814842 21414   0x55947664a0 DEBUG              GST_EVENT gstpad.c:5693:gst_pad_send_event_unchecked:<nvinfer0:src> have event type qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.312842972 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1941:gst_base_transform_src_eventfunc:<nvinfer0> handling event 0x7f145af0f0 qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.312862493 21414   0x55947664a0 DEBUG                GST_QOS gstbasetransform.c:2645:gst_base_transform_update_qos:<nvinfer0> qos: proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.312888542 21414   0x55947664a0 DEBUG              GST_EVENT gstpad.c:5693:gst_pad_send_event_unchecked:<batcher0:src> have event type qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.312914815 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1941:gst_base_transform_src_eventfunc:<batcher0> handling event 0x7f145af0f0 qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.312933920 21414   0x55947664a0 DEBUG                GST_QOS gstbasetransform.c:2645:gst_base_transform_update_qos:<batcher0> qos: proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.312959297 21414   0x55947664a0 DEBUG              GST_EVENT gstpad.c:5693:gst_pad_send_event_unchecked:<nvdewarper0:src> have event type qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.312985603 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1941:gst_base_transform_src_eventfunc:<nvdewarper0> handling event 0x7f145af0f0 qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.313004035 21414   0x55947664a0 DEBUG                GST_QOS gstbasetransform.c:2645:gst_base_transform_update_qos:<nvdewarper0> qos: proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.313028325 21414   0x55947664a0 DEBUG              GST_EVENT gstpad.c:5693:gst_pad_send_event_unchecked:<nvvideoconvert0:src> have event type qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.313055238 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1941:gst_base_transform_src_eventfunc:<nvvideoconvert0> handling event 0x7f145af0f0 qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.313077127 21414   0x55947664a0 DEBUG                GST_QOS gstbasetransform.c:2645:gst_base_transform_update_qos:<nvvideoconvert0> qos: proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.313103624 21414   0x55947664a0 DEBUG              GST_EVENT gstpad.c:5693:gst_pad_send_event_unchecked:<videoconvert0:src> have event type qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.313153003 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1941:gst_base_transform_src_eventfunc:<videoconvert0> handling event 0x7f145af0f0 qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.313172140 21414   0x55947664a0 DEBUG                GST_QOS gstbasetransform.c:2645:gst_base_transform_update_qos:<videoconvert0> qos: proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.313197229 21414   0x55947664a0 DEBUG              GST_EVENT gstpad.c:5693:gst_pad_send_event_unchecked:<bayer2rgb0:src> have event type qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.313223662 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1941:gst_base_transform_src_eventfunc:<bayer2rgb0> handling event 0x7f145af0f0 qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.313242031 21414   0x55947664a0 DEBUG                GST_QOS gstbasetransform.c:2645:gst_base_transform_update_qos:<bayer2rgb0> qos: proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.313266864 21414   0x55947664a0 DEBUG              GST_EVENT gstpad.c:5693:gst_pad_send_event_unchecked:<aravis0:src> have event type qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.313995283 21414   0x55c486f0f0 DEBUG             GST_BUFFER gstbuffer.c:2384:gst_buffer_foreach_meta: remove metadata 0x7f073f2248 (NvDsMeta)
0:01:09.320308864 21414   0x55947664a0 DEBUG                basesrc gstbasesrc.c:2027:gst_base_src_default_event:<aravis0> handle event qos event: 0x7f145af0f0, time 99:99:99.999999999, seq-num 5110, GstEventQOS, type=(GstQOSType)GST_QOS_TYPE_UNDERFLOW, proportion=(double)0.98098550039442145, diff=(gint64)57965989, timestamp=(guint64)65163802118;
0:01:09.320317888 21414   0x55c486f0f0 DEBUG               nvdspool gstnvdsbufferpool.c:524:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool1> release_buffer
0:01:09.320339105 21414   0x55947664a0 DEBUG                GST_QOS gstbasesrc.c:2013:gst_base_src_update_qos:<aravis0> qos: proportion: 0.980986, diff 57965989, timestamp 0:01:05.163802118
0:01:09.320344225 21414   0x55c486f0f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:09.320368866 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2311:gst_base_transform_chain:<batcher0> we have a pending DISCONT
0:01:09.320395940 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2313:gst_base_transform_chain:<batcher0> marking DISCONT on output buffer
0:01:09.320380195 21414   0x55947664a0 DEBUG               GST_PADS gstpad.c:5776:gst_pad_send_event_unchecked:<aravis0:src> sent event, ret ok
0:01:09.320436198 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<nvinfer0:sink> calling chainfunction &gst_base_transform_chain with buffer buffer: 0x7f00afa6b0, pts 0:01:05.203018630, dts 99:99:99.999999999, dur 0:00:00.019608352, size 64, offset none, offset_end none, flags 0x40
0:01:09.320457415 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2273:gst_base_transform_chain:<nvinfer0> got DISCONT buffer 0x7f00afa6b0
0:01:09.320441318 21414   0x55947664a0 DEBUG               GST_PADS gstpad.c:5776:gst_pad_send_event_unchecked:<bayer2rgb0:src> sent event, ret ok
0:01:09.320497416 21414   0x55947664a0 DEBUG               GST_PADS gstpad.c:5776:gst_pad_send_event_unchecked:<videoconvert0:src> sent event, ret ok
0:01:09.320512393 21414   0x55947664a0 DEBUG               GST_PADS gstpad.c:5776:gst_pad_send_event_unchecked:<nvvideoconvert0:src> sent event, ret ok
0:01:09.320525674 21414   0x55947664a0 DEBUG               GST_PADS gstpad.c:5776:gst_pad_send_event_unchecked:<nvdewarper0:src> sent event, ret ok
0:01:09.320537962 21414   0x55947664a0 DEBUG               GST_PADS gstpad.c:5776:gst_pad_send_event_unchecked:<batcher0:src> sent event, ret ok
0:01:09.320550283 21414   0x55947664a0 DEBUG               GST_PADS gstpad.c:5776:gst_pad_send_event_unchecked:<nvinfer0:src> sent event, ret ok
0:01:09.320562668 21414   0x55947664a0 DEBUG               GST_PADS gstpad.c:5776:gst_pad_send_event_unchecked:<nvtiler:src> sent event, ret ok
0:01:09.320575180 21414   0x55947664a0 DEBUG               basesink gstbasesink.c:3577:gst_base_sink_chain_unlocked:<nvoverlaysink-nvoverlaysink0> object unref after render 0x55c372bd70
0:01:09.320592621 21414   0x55947664a0 DEBUG             GST_BUFFER gstbuffer.c:2384:gst_buffer_foreach_meta: remove metadata 0x7f073f2528 (NvDsMeta)
0:01:09.320640303 21414   0x55947664a0 DEBUG               nvdspool gstnvdsbufferpool.c:524:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool2> release_buffer
0:01:09.320661936 21414   0x55947664a0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:09.320680241 21414   0x55947664a0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<nvoverlaysink-nvoverlaysink0:sink> called chainfunction &gst_base_sink_chain with buffer 0x55c372bd70, returned ok
0:01:09.320695282 21414   0x55947664a0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<nvtiler:sink> called chainfunction &gst_base_transform_chain with buffer 0x7f06fe27f0, returned ok
0:01:09.322316255 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<nvinfer0:sink> called chainfunction &gst_base_transform_chain with buffer 0x7f00afa6b0, returned ok
0:01:09.322339456 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<batcher0:sink> called chainfunction &gst_base_transform_chain with buffer 0x55c372ba40, returned ok
0:01:09.322352289 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<nvdewarper0:sink> called chainfunction &gst_base_transform_chain with buffer 0x7f06fe24c0, returned ok
0:01:09.322367842 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<nvvideoconvert0:sink> called chainfunction &gst_base_transform_chain with buffer 0x7f06fe2c30, returned ok
0:01:09.322380866 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<videoconvert0:sink> called chainfunction &gst_base_transform_chain with buffer 0x7f06fe2c30, returned ok
0:01:09.322396003 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<bayer2rgb0:sink> called chainfunction &gst_base_transform_chain with buffer 0x7f06fe26e0, returned ok
0:01:09.322415012 21414   0x55c486f0f0 DEBUG                basesrc gstbasesrc.c:2506:gst_base_src_get_range:<aravis0> calling create offset 18446744073709551615 length 4096, time 0
0:01:09.322441477 21414   0x55c486f0f0 DEBUG             GST_MEMORY gstmemory.c:139:gst_memory_init: new memory 0x7f00af8ea0, maxsize:1000000 offset:0 size:1000000
0:01:09.322466886 21414   0x55c486f0f0 DEBUG              GST_CLOCK gstclock.c:249:gst_clock_entry_new:<GstSystemClock> created entry 0x7f00aa9ed0, time 30:30:47.990221463
0:01:09.322479559 21414   0x55c486f0f0 DEBUG              GST_CLOCK gstclock.c:537:gst_clock_id_wait:<GstSystemClock> waiting on clock entry 0x7f00aa9ed0
0:01:09.322496232 21414   0x55c486f0f0 DEBUG              GST_CLOCK gstclock.c:1046:gst_clock_get_internal_time:<GstSystemClock> internal time 30:30:47.999783068
0:01:09.322510312 21414   0x55c486f0f0 DEBUG              GST_CLOCK gstclock.c:1091:gst_clock_get_time:<GstSystemClock> adjusted time 30:30:47.999783068
0:01:09.322530505 21414   0x55c486f0f0 DEBUG              GST_CLOCK gstsystemclock.c:719:gst_system_clock_id_wait_jitter_unlocked: entry 0x7f00aa9ed0 time 30:30:47.990221463 now 30:30:47.999783068 diff (time-now) -9561605
0:01:09.322547274 21414   0x55c486f0f0 DEBUG              GST_CLOCK gstclock.c:547:gst_clock_id_wait:<GstSystemClock> done waiting entry 0x7f00aa9ed0, res: 1 (early)
0:01:09.322559627 21414   0x55c486f0f0 DEBUG              GST_CLOCK gstclock.c:352:_gst_clock_id_free: freed entry 0x7f00aa9ed0
0:01:09.322573483 21414   0x55c486f0f0 DEBUG                basesrc gstbasesrc.c:2593:gst_base_src_get_range:<aravis0> buffer too late!, returning anyway
0:01:09.322598477 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<bayer2rgb0:sink> calling chainfunction &gst_base_transform_chain with buffer buffer: 0x7f00afa7c0, pts 0:01:05.222627222, dts 99:99:99.999999999, dur 0:00:00.019608592, size 1000000, offset none, offset_end none, flags 0x0
0:01:09.322621998 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<bayer2rgb0> handling buffer 0x7f00afa7c0 of size 1000000, PTS 0:01:05.222627222 and offset NONE
0:01:09.322636494 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<bayer2rgb0> calling prepare buffer
0:01:09.322668432 21414   0x55c486f0f0 DEBUG               GST_CAPS gstpad.c:2733:gst_pad_get_current_caps:<bayer2rgb0:sink> get current pad caps video/x-bayer, format=(string)rggb, width=(int)1000, height=(int)1000, framerate=(fraction)51/1
0:01:09.322695793 21414   0x55c486f0f0 DEBUG               GST_CAPS gstpad.c:2733:gst_pad_get_current_caps:<bayer2rgb0:src> get current pad caps video/x-raw, width=(int)1000, height=(int)1000, framerate=(fraction)51/1, format=(string)BGRx
0:01:09.322749716 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1635:default_prepare_output_buffer:<bayer2rgb0> getting output size for alloc
0:01:09.322778229 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:598:gst_base_transform_transform_size:<bayer2rgb0> asked to transform size 1000000 for caps video/x-bayer, format=(string)rggb, width=(int)1000, height=(int)1000, framerate=(fraction)51/1 to size for caps video/x-raw, width=(int)1000, height=(int)1000, framerate=(fraction)51/1, format=(string)BGRx in direction SINK
0:01:09.322793686 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1801:gst_base_transform_get_unit_size:<bayer2rgb0> returned 1000000 from first cache
0:01:09.322803222 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:532:gst_base_transform_default_transform_size:<bayer2rgb0> input size 1000000, input unit size 1000000
0:01:09.322811479 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1807:gst_base_transform_get_unit_size:<bayer2rgb0> returned 4000000 from second cached
0:01:09.322819127 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:549:gst_base_transform_default_transform_size:<bayer2rgb0> transformed size to 4000000
0:01:09.322828216 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1647:default_prepare_output_buffer:<bayer2rgb0> doing alloc of size 4000000
0:01:09.322850105 21414   0x55c486f0f0 DEBUG             GST_MEMORY gstmemory.c:139:gst_memory_init: new memory 0x7f00e7b700, maxsize:4000007 offset:0 size:4000000
0:01:09.322865081 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<bayer2rgb0> copying metadata
0:01:09.322882362 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<bayer2rgb0> using allocated buffer in 0x7f00afa7c0, out 0x7f00afa490
0:01:09.322894235 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<bayer2rgb0> doing non-inplace transform
0:01:09.322901307 21414   0x55c486f0f0 DEBUG              bayer2rgb gstbayer2rgb.c:462:gst_bayer2rgb_transform: transforming buffer
0:01:09.322997856 21414   0x7efc003180 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:09.326989982 21414   0x55c486f0f0 DEBUG             GST_MEMORY gstmemory.c:88:_gst_memory_free: free memory 0x7f00af8ea0
0:01:09.327064801 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<videoconvert0:sink> calling chainfunction &gst_base_transform_chain with buffer buffer: 0x7f00afa490, pts 0:01:05.222627222, dts 99:99:99.999999999, dur 0:00:00.019608592, size 4000000, offset none, offset_end none, flags 0x0
0:01:09.327097187 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<videoconvert0> handling buffer 0x7f00afa490 of size 4000000, PTS 0:01:05.222627222 and offset NONE
0:01:09.327111300 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<videoconvert0> calling prepare buffer
0:01:09.327122340 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1592:default_prepare_output_buffer:<videoconvert0> passthrough: reusing input buffer
0:01:09.327134949 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<videoconvert0> using allocated buffer in 0x7f00afa490, out 0x7f00afa490
0:01:09.327145701 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2120:default_generate_output:<videoconvert0> element is in passthrough
0:01:09.327167046 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<nvvideoconvert0:sink> calling chainfunction &gst_base_transform_chain with buffer buffer: 0x7f00afa490, pts 0:01:05.222627222, dts 99:99:99.999999999, dur 0:00:00.019608592, size 4000000, offset none, offset_end none, flags 0x0
0:01:09.327186439 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvvideoconvert0> handling buffer 0x7f00afa490 of size 4000000, PTS 0:01:05.222627222 and offset NONE
0:01:09.327197608 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvvideoconvert0> calling prepare buffer
0:01:09.327208776 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvvideoconvert0> using pool alloc
0:01:09.327225289 21414   0x55c486f0f0 DEBUG               nvdspool gstnvdsbufferpool.c:474:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool0> acquire_buffer
0:01:09.327243242 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvvideoconvert0> copying metadata
0:01:09.327257259 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvvideoconvert0> using allocated buffer in 0x7f00afa490, out 0x7f06fe22a0
0:01:09.327267883 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvvideoconvert0> doing non-inplace transform
0:01:09.327288268 21414   0x55c486f0f0 DEBUG         nvvideoconvert gstnvvideoconvert.c:2628:gst_nvvideoconvert_transform:<nvvideoconvert0> NvDsBatchMeta not found for output buffer
0:01:09.330876215 21414   0x55c486f0f0 DEBUG             GST_MEMORY gstmemory.c:88:_gst_memory_free: free memory 0x7f00e7b700
0:01:09.330944346 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<nvdewarper0:sink> calling chainfunction &gst_base_transform_chain with buffer buffer: 0x7f06fe22a0, pts 0:01:05.222627222, dts 99:99:99.999999999, dur 0:00:00.019608592, size 64, offset none, offset_end none, flags 0x0
0:01:09.330974940 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvdewarper0> handling buffer 0x7f06fe22a0 of size 64, PTS 0:01:05.222627222 and offset NONE
0:01:09.330988764 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvdewarper0> calling prepare buffer
0:01:09.331004189 21414   0x55c486f0f0 DEBUG               nvdspool gstnvdsbufferpool.c:474:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool1> acquire_buffer
0:01:09.331024510 21414   0x55c486f0f0 DEBUG             nvdewarper gstnvdewarper.cpp:1036:gst_nvdewarper_prepare_output_buffer:<nvdewarper0> gst_nvdewarper_prepare_output_buffer : Frame=1724 Gst-OutBuf=0x55c372b710

0:01:09.331041855 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvdewarper0> using allocated buffer in 0x7f06fe22a0, out 0x55c372b710
0:01:09.331049343 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvdewarper0> doing non-inplace transform
0:01:09.331061664 21414   0x55c486f0f0 DEBUG             nvdewarper gstnvdewarper.cpp:960:gst_nvdewarper_transform:<nvdewarper0> gst_nvdewarper_transform : Frame=1725 InBuf=0x7f06fe22a0 OutBuf=0x55c372b710

0:01:09.331085313 21414   0x55c486f0f0 DEBUG             nvdewarper gstnvdewarper.cpp:968:gst_nvdewarper_transform:<nvdewarper0> transform
0:01:09.345317479 21414   0x55947664a0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<nvtiler:sink> calling chainfunction &gst_base_transform_chain with buffer buffer: 0x7f00afa6b0, pts 0:01:05.203018630, dts 99:99:99.999999999, dur 0:00:00.019608352, size 64, offset none, offset_end none, flags 0x40
0:01:09.345362185 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:2273:gst_base_transform_chain:<nvtiler> got DISCONT buffer 0x7f00afa6b0
0:01:09.345424524 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<nvtiler> handling buffer 0x7f00afa6b0 of size 64, PTS 0:01:05.203018630 and offset NONE
0:01:09.345499376 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<nvtiler> calling prepare buffer
0:01:09.345696569 21414   0x55c486f0f0 INFO                 default nvdewarper.cpp:453:Dewarp_VertRadCylBuffer:<nvdewarper0>  Dewarp_VertRadCylBuffer Frame=1725 Dewarp_VertRadCyl for Views=0 cudaErr=0 Stream=0x55a9155dc0 Completed
0:01:09.346425372 21414   0x55c486f0f0 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f073f2248 (NvDsMeta) of size 72
0:01:09.346466334 21414   0x55c486f0f0 INFO              nvdewarper gstnvdewarper.cpp:1001:gst_nvdewarper_transform:<nvdewarper0>  : source_id 0 Frame=1725 OUT-BUFFER 0:01:05.222627222
0:01:09.346544833 21414   0x55c486f0f0 DEBUG               nvdspool gstnvdsbufferpool.c:524:gst_nvds_buffer_pool_release_buffer:<nvdsbufferpool0> release_buffer
0:01:09.346569667 21414   0x55c486f0f0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:01:09.346604260 21414   0x55c486f0f0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<batcher0:sink> calling chainfunction &gst_base_transform_chain with buffer buffer: 0x55c372b710, pts 0:01:05.222627222, dts 99:99:99.999999999, dur 0:00:00.019608592, size 64, offset none, offset_end none, flags 0x0
0:01:09.346625957 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:1990:default_submit_input_buffer:<batcher0> handling buffer 0x55c372b710 of size 64, PTS 0:01:05.222627222 and offset NONE
0:01:09.346659687 21414   0x55c486f0f0 DEBUG          basetransform gstbasetransform.c:2102:default_generate_output:<batcher0> calling prepare buffer
0:01:09.346770700 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1605:default_prepare_output_buffer:<nvtiler> using pool alloc
0:01:09.346803566 21414   0x55947664a0 DEBUG               nvdspool gstnvdsbufferpool.c:474:gst_nvds_buffer_pool_acquire_buffer:<nvdsbufferpool2> acquire_buffer
0:01:09.346817678 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1745:default_copy_metadata:<nvtiler> copying metadata
0:01:09.346839824 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:1717:foreach_metadata:<nvtiler> transformed metadata NvDsMetaAPI: copy: 0
0:01:09.346852432 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:2109:default_generate_output:<nvtiler> using allocated buffer in 0x7f00afa6b0, out 0x55c372bc60
0:01:09.346860016 21414   0x55947664a0 DEBUG          basetransform gstbasetransform.c:2129:default_generate_output:<nvtiler> doing non-inplace transform
Segmentation fault (core dumped)
My custom batcher's prepare output buffer:
static GstFlowReturn
gst_batcher_prepare_output_buffer(GstBaseTransform * btrans, GstBuffer * inbuf, GstBuffer ** outbuf)
{
  GstBatcher *batcher = GST_BATCHER (btrans);
  GstFlowReturn flow_ret = GST_FLOW_ERROR;	
  NvBufSurfaceCreateParams create_params;
  NvDsBatchMeta *batch_meta = NULL;
  NvDsFrameMeta *frame_meta = NULL;
  NvDsMeta *meta = NULL;

  //g_print ("Prepare start\n");	

  /* allocate buffer for crop destination */
  create_params.gpuId  = batcher->gpu_id;
  create_params.width  = 300;
  create_params.height = 300;
  create_params.size = 0;
  create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
  create_params.layout = NVBUF_LAYOUT_PITCH;

  if(batcher->is_integrated) {
    create_params.memType = NVBUF_MEM_DEFAULT;
  } else {
    create_params.memType = NVBUF_MEM_CUDA_PINNED;
  }

  if (batcher->inter_buf)
    NvBufSurfaceDestroy (batcher->inter_buf);
  batcher->inter_buf = NULL;	

  if (NvBufSurfaceCreate (&batcher->inter_buf, 4,
        &create_params) != 0) {
    GST_ERROR ("Error: Could not allocate internal buffer for batcher");
    goto error;
  }

  *outbuf = NULL;
  *outbuf = gst_buffer_new_wrapped_full (GST_MEMORY_FLAG_ZERO_PREFIXED, batcher->inter_buf,
                                         sizeof(NvBufSurface), 0, sizeof(NvBufSurface), NULL, NULL);    	

  // Create and fill batch meta
  batch_meta = nvds_create_batch_meta(4);    	
  if (batch_meta == NULL) {
    GST_ERROR ("Error: Could not create batch meta");
    goto error;
  }
  batch_meta->base_meta.batch_meta = batch_meta;
  batch_meta->base_meta.copy_func = nvds_batch_meta_copy_func;
  batch_meta->base_meta.release_func = nvds_batch_meta_release_func;
  batch_meta->max_frames_in_batch = 4;

  for(int i = 0;i < 4;i++) {
    frame_meta = nvds_acquire_frame_meta_from_pool(batch_meta);
    if (frame_meta == NULL) {
      GST_ERROR ("Error: Could not allocate internal buffer for batcher");
      goto error;
    }
    // FIXME: Properly set the buf pts, ntp timestamp, frame_num
    frame_meta->pad_index = i;
    frame_meta->source_id = 0;
    frame_meta->buf_pts = 0;
    frame_meta->ntp_timestamp = 0;
    frame_meta->frame_num = 0;
    frame_meta->batch_id = i;
    frame_meta->source_frame_width = 300;
    frame_meta->source_frame_height = 300;
    frame_meta->num_surfaces_per_frame = 1;
    nvds_add_frame_meta_to_batch(batch_meta, frame_meta);
    //g_print("%d\n", batch_meta->num_frames_in_batch);
  }
  meta = gst_buffer_add_nvds_meta (*outbuf , batch_meta, NULL, nvds_batch_meta_copy_func, nvds_batch_meta_release_func);
  meta->meta_type = NVDS_BATCH_GST_META;

  flow_ret = GST_FLOW_OK;	
  //g_print ("Prepare OK\n");	
error:	

  return flow_ret;	
}

It processes and visualizes previous frames well but as you can see in the last few lines of the error log, prepare_output_buffer was not completely executed and transform() was not even called, and nvtiler tries to read the output causing the SEGFAULT. But I am not sure what is causing the issue. I would really appreciate any sort of input. Thank you!

Sometimes it also seems to give different errors as well:

    NvRmChannelSubmit: NvError_IoctlFailed with error code 22
    NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 45, SyncPointValue = 0)
    fence_set_name ioctl failed with 22

log.txt (15.4 MB)

Hey, customer
It’s really hard to debug your issue since you are using your custom plugin to generate the batch.
May I know why nvstreammux cannot meet your requirement for generating a batch.

We have discussed about usage of nvstreammux in this topic. And I fixed my segfault by fixing some parts in the prepare_output_buffer as I explained here

Great work!