We are currently running a custom TLT model through Deepstream on a live stream and we consistently reach a segmentation fault error after a given period of time. The seg fault appears to happen in the dequeueOutputBatch call that is in the gst_nvinfer_output_loop method of gstnvinfer.cpp. The call stack for the fault is given below:
<unknown> 0x00007fdeabbdccff
std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<float> stl_algobase.h:368
std::__copy_move_a<true, float*, float*> stl_algobase.h:386
std::__copy_move_a2<true, float*, float*> stl_algobase.h:424
std::copy<std::move_iterator<float*>, float*> stl_algobase.h:456
std::__uninitialized_copy<true>::__uninit_copy<std::move_iterator<float*>, float*> stl_uninitialized.h:101
std::uninitialized_copy<std::move_iterator<float*>, float*> stl_uninitialized.h:134
std::__uninitialized_copy_a<std::move_iterator<float*>, float*, float> stl_uninitialized.h:289
std::__uninitialized_move_if_noexcept_a<float*, float*, std::allocator<float> > stl_uninitialized.h:312
std::vector<float, std::allocator<float> >::_M_realloc_insert<float> vector.tcc:431
std::vector<float, std::allocator<float> >::emplace_back<float> vector.tcc:105
std::vector<float, std::allocator<float> >::push_back stl_vector.h:954
NvDsInferParseCustomFrcnnTLT nvdsinfer_custombboxparser_frcnn_tlt.cpp:325
nvdsinfer::DetectPostprocessor::fillDetectionOutput nvdsinfer_context_impl_output_parsing.cpp:721
nvdsinfer::DetectPostprocessor::parseEachBatch nvdsinfer_context_impl.cpp:711
nvdsinfer::InferPostprocessor::postProcessHost nvdsinfer_context_impl.cpp:584
nvdsinfer::NvDsInferContextImpl::dequeueOutputBatch nvdsinfer_context_impl.cpp:1577
gst_nvinfer_output_loop gstnvinfer.cpp:2014
<unknown> 0x00007fdeaddfd2a5
start_thread 0x00007fdeace176db
clone 0x00007fdeabb6f71f
This sometime occurs after a few minutes, sometime it occurs after several hours. However, it regularly occurs during the stream processing