Tee + queue after nvinferserver causes random segmentation faults

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU) - GPU
• DeepStream Version - 6.1.1
• TensorRT Version - whatever comes with DeepStream 6.1.1
• NVIDIA GPU Driver Version (valid for GPU only) - 510.47.03
• Issue Type( questions, new requirements, bugs) - possibly a bug?

I have the following pipeline:

pipeline
pipeline17431×820 857 KB

Purpose of this pipeline is to process live video and at the same time collect sample data with a fixed interval (bottom branch after tee). Interval is achieved with the identity element, which has a probe attached. The probe looks like this (python):

def _filter_frame(self, pad, info):
    diff = time.time() - self._last_frame_timestamp

    if diff >= self._interval:
        self._last_frame_timestamp = time.time()

        detections = [
            {
                "label": object_meta.obj_label,
                "bbox": [
                    int(object_meta.rect_params.left),
                    int(object_meta.rect_params.top),
                    int(object_meta.rect_params.width),
                    int(object_meta.rect_params.height),
                ],
                "score": object_meta.confidence,
            }
            for object_meta in get_object_meta(
                pyds.gst_buffer_get_nvds_batch_meta(hash(info.get_buffer()))
            )
        ]

       # get_object_meta is a helper function which iterates over batch meta and yields individual detections

        frame_meta = {
            "detections": detections,
            "timestamp": datetime.datetime.utcnow().isoformat() + "Z",
        }

        self._frame_metas.put(frame_meta)

        return Gst.PadProbeReturn.OK

    return Gst.PadProbeReturn.DROP

This code might run for 10 minutes or 5 hours before segfaulting. I have also collected core dumps and they usually point to the second queue element, however they also happen elsewhere. It doesn’t look video stream specific as I have tested it with at least 10 different streams.

Here are some stacktraces taken from core dumps:

# this happened in the queue2:src thread
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f4bb5ea500b in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f4b94abc700 (LWP 1460168))]
#0  0x00007f4bb5ea500b in raise () at /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb5ea5090 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f4bb416e824 in nvds_copy_obj_meta () at /opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_meta.so
#3  0x00007f4bb416ea7a in nvds_copy_obj_meta_list () at /opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_meta.so
#4  0x00007f4bb416ebaa in nvds_copy_frame_meta () at /opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_meta.so
#5  0x00007f4bb416ec09 in nvds_copy_frame_meta_list () at /opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_meta.so
#6  0x00007f4bb416ecb1 in batch_meta_copy () at /opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_meta.so
#7  0x00007f4bb3e77062 in  () at /opt/nvidia/deepstream/deepstream-6.1/lib/libnvdsgst_meta.so
#8  0x00007f4bb502e2c2 in gst_buffer_copy_into () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#9  0x00007f4bb502e482 in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#10 0x00007f4bb39355b6 in  () at /lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#11 0x00007f4bb3933ab8 in  () at /lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#12 0x00007f4bb3933426 in  () at /lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#13 0x00007f4bb5066fef in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#14 0x00007f4bb5069051 in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#15 0x00007f4bb506fe63 in gst_pad_push () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#16 0x00007f4b96ed1774 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstcoreelements.so
#17 0x00007f4bb509e1e7 in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#18 0x00007f4bb58da374 in  () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#19 0x00007f4bb58d9ad1 in  () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#20 0x00007f4bb5e47609 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007f4bb5f81133 in clone () at /lib/x86_64-linux-gnu/libc.so.6

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f630860c00b in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f62d5e59700 (LWP 1420065))]
#0  0x00007f630860c00b in raise () at /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f630860c090 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f62dd9d28b9 in NvTrackerProc::fillMOTFrame(unsigned long, NvTrackerProc::ProcParams const&, _NvDsFrameMeta const&, _NvMOTFrame&, _NvMOTTrackedObjList&) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#3  0x00007f62dd9d6808 in NvTrackerProc::processBatch() () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#4  0x00007f630649ede4 in  () at /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f63085ae609 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f63086e8133 in clone () at /lib/x86_64-linux-gnu/libc.so.6

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f7b1676b00b in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f7a57fff700 (LWP 1486436))]
#0  0x00007f7b1676b00b in raise () at /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f7b1676b090 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f7b167c21fe in malloc () at /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f7b145d1b39 in operator new(unsigned long) () at /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f7afdd3b70d in std::_Function_base::_Base_manager<nvdsinferserver::BufferPool<std::unique_ptr<nvdsinferserver::SurfaceBuffer, std::default_delete<nvdsinferserver::SurfaceBuffer> > >::acquireBuffer()::{lambda(nvdsinferserver::SurfaceBuffer*)#1}>::_M_manager(std::_Any_data&, std::_Function_base::_Base_manager<nvdsinferserver::BufferPool<std::unique_ptr<nvdsinferserver::SurfaceBuffer, std::default_delete<nvdsinferserver::SurfaceBuffer> > >::acquireBuffer()::{lambda(nvdsinferserver::SurfaceBuffer*)#1}> const&, std::_Manager_operation) () at ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_infer_server.so
#5  0x00007f7afdd39f23 in nvdsinferserver::CropSurfaceConverter::requestOutBuffer(std::shared_ptr<nvdsinferserver::BaseBatchBuffer>&) () at ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_infer_server.so
#6  0x00007f7afdd62177 in std::_Function_handler<bool (std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >), nvdsinferserver::ThreadPreprocessor<nvdsinferserver::CropSurfaceConverter>::ThreadPreprocessor<int const&>(int const&)::{lambda(std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >)#1}>::_M_invoke(std::_Any_data const&, std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >&&) () at ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_infer_server.so
#7  0x00007f7afdd66e42 in nvdsinferserver::QueueThread<std::vector<std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >, std::allocator<std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> > > > >::threadLoop() () at ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_infer_server.so
#8  0x00007f7afdd6728f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<nvdsinferserver::QueueThread<std::vector<std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >, std::allocator<std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> > > > >::QueueThread(std::function<bool (std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}> > >::_M_run() () at ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_infer_server.so
#9  0x00007f7b145fdde4 in  () at /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007f7b1670d609 in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007f7b16847133 in clone () at /lib/x86_64-linux-gnu/libc.so.6
3

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Can you debug with your code and find out the simplest pipeline to reproduce the failure? It is hard to tell what happened with what you post.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.