Nvdsinfer related Core dump after running 20 days

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) 4.4
• TensorRT Version 7.1.3

We have a gst-launch-1.0 pipeline running with Deepstream plugins. We have a system out in the field with it running consistently for 20 days. Then it threw a core dump error and the gst pipeline stopped running. And we applied the following patch to build the lib_nvdsinfer.so.

The diagnostic is below:

Using host libthread_db library “/lib/aarch64-linux-gnu/libthread_db.so.1”.

Core was generated by `gst-launch-1.0 udpsrc do-timestamp=true port=6000 buffer-size=60000000 ! applic’.

Program terminated with signal SIGABRT, Aborted.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

[Current thread is 1 (Thread 0x7fae6e0a50 (LWP 7921))]

(gdb) bt

#0  0x0000007fae28e4d8 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

#1  0x0000007fae28f8b4 in __GI_abort () at abort.c:79

#2  0x0000007fad11610c in __gnu_cxx::__verbose_terminate_handler() () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6

#3  0x0000007fad113c54 in  () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6

#4  0x0000007fad112b5c in  () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6

#5  0x0000007fad113340 in __gxx_personality_v0 () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6

#6  0x0000007fad06c9b4 in  () at /lib/aarch64-linux-gnu/libgcc_s.so.1

#7  0x0000007fad06cf6c in _Unwind_Resume () at /lib/aarch64-linux-gnu/libgcc_s.so.1

#8  0x0000007f831978d0 in nvinfer1::throwNvmError(char const*, char const*, int, int, char const*) () at /usr/lib/aarch64-linux-gnu/libnvinfer.so.7

#9  0x0000007f8317b3bc in  () at /usr/lib/aarch64-linux-gnu/libnvinfer.so.7

#10 0x0000007f83193728 in nvinfer1::trtCudaFree(nvinfer1::IGpuAllocator*, void*, char const*, char const*, int) () at /usr/lib/aarch64-linux-gnu/libnvinfer.so.7

#11 0x0000007f8316ca60 in nvinfer1::rt::SafeExecutionContext::~SafeExecutionContext() () at /usr/lib/aarch64-linux-gnu/libnvinfer.so.7

#12 0x0000007f82eecaa4 in nvinfer1::rt::ExecutionContext::~ExecutionContext() () at /usr/lib/aarch64-linux-gnu/libnvinfer.so.7

#13 0x0000007f82eecb94 in nvinfer1::rt::ExecutionContext::~ExecutionContext() () at /usr/lib/aarch64-linux-gnu/libnvinfer.so.7

#14 0x0000007fa3016358 in nvdsinfer::UniquePtrWDestroy<nvinfer1::IExecutionContext>::UniquePtrWDestroy(nvinfer1::IExecutionContext*)::{lambda(nvinfer1::IExecutionContext*)#1}::operator()(nvinfer1::IExecutionContext*) const ()

    at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#15 0x0000007fa301637c in nvdsinfer::UniquePtrWDestroy<nvinfer1::IExecutionContext>::UniquePtrWDestroy(nvinfer1::IExecutionContext*)::{lambda(nvinfer1::IExecutionContext*)#1}::_FUN(nvinfer1::IExecutionContext*) ()

    at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#16 0x0000007fa3016280 in std::unique_ptr<nvinfer1::IExecutionContext, void (*)(nvinfer1::IExecutionContext*)>::reset(nvinfer1::IExecutionContext*) () at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#17 0x0000007fa30141c4 in nvdsinfer::TrtBackendContext::~TrtBackendContext() () at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#18 0x0000007fa301615c in nvdsinfer::FullDimTrtBackendContext::~FullDimTrtBackendContext() () at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#19 0x0000007fa301617c in nvdsinfer::FullDimTrtBackendContext::~FullDimTrtBackendContext() () at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#20 0x0000007fa2fcbca0 in std::default_delete<nvdsinfer::BackendContext>::operator()(nvdsinfer::BackendContext*) const () at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#21 0x0000007fa2fc763c in std::unique_ptr<nvdsinfer::BackendContext, std::default_delete<nvdsinfer::BackendContext> >::reset(nvdsinfer::BackendContext*) () at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#22 0x0000007fa2fbbb90 in nvdsinfer::NvDsInferContextImpl::~NvDsInferContextImpl() () at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#23 0x0000007fa2fbc01c in nvdsinfer::NvDsInferContextImpl::~NvDsInferContextImpl() () at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_infer.so

#24 0x0000007fa30bead0 in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so

#25 0x0000007fa30b6040 in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so

#26 0x0000007faddb739c in  () at /usr/lib/aarch64-linux-gnu/libgstbase-1.0.so.0

#27 0x0000005588e44460 in  ()

(gdb)

Hi @wdw0908 , we cannot locate the cause of the crash only from the log attached. Could you update your JetPack Version and deepstream version to the latest?

Hi Yuweiw,

I am sorry this unit is out in the field and it cannot be updated at this moment. And the jetpack version is 4.4 and this version doesn’t support image-based OTA, as far as I know. I wonder if this error above is related to the patch below. We applied the patch to adapt it to dynamic input batch size.

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

We cannot confirm it only by the crash info posted. But you can refer our latest code from ds6.1.0, the patch is already in the officially released version. And the crash info is destructor api. So maybe it’s not the root cause.
Could you try to duplacate it and provide more crash info and the log info to us?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.