Segmentation fault (core dumped) deepstream on Xavier NX

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi Team,

I am using Xavier NX 8GB(R35.4.1 L4T BSP).

I am working on custom yolov8n, converted to trt engine. deployed in the deepstream(6.3). But no luck, getting segmentation fault (core dumped) and there are no errors generated during the process. I am attaching the log FYR.

Note: fetching the video source from local storage(internal)

$deepstream-app -c deepstream_app_config_copy.txt
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
~~ CLOG[/dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvmultiobjecttracker/include/modules/NvMultiObjectTracker/NvTrackerParams.hpp, getConfigRoot() @line 52]: [NvTrackerParams::getConfigRoot()] !!![WARNING] Invalid low-level config file caused an exception, but will go ahead with the default config values
~~ CLOG[/dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvmultiobjecttracker/include/modules/NvMultiObjectTracker/NvTrackerParams.hpp, getConfigRoot() @line 52]: [NvTrackerParams::getConfigRoot()] !!![WARNING] Invalid low-level config file caused an exception, but will go ahead with the default config values
[NvMultiObjectTracker] Initialized
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:05.332033531 9304 0xaaaae6c99730 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/home/ninad/hump/models/hump_fp16.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT images 3x640x640
1 OUTPUT kFLOAT output0 5x8400

0:00:05.401426731 9304 0xaaaae6c99730 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /home/ninad/hump/models/hump_fp16.engine
0:00:05.412628192 9304 0xaaaae6c99730 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/ninad/hump/config_infer_primary_yoloV8.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:239>: Pipeline ready

**PERF: FPS 0 (Avg)
**PERF: 0.00 (0.00)
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
** INFO: <bus_callback:225>: Pipeline running

Segmentation fault (core dumped)

5

Could you use gdb tool to do a preliminary analysis?

1.$gdb --args deepstream-app -c deepstream_app_config_copy.txt
2.$r
3.$bt
6

Hi Yuweiw,

please find the results after executing the above commands(GDB tool):

$ gdb --args deepstream-app -c deepstream_app_config_copy.txt
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type “show copying” and “show warranty” for details.
This GDB was configured as “aarch64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from deepstream-app…
(No debugging symbols found in deepstream-app)
(gdb) r
Starting program: /usr/bin/deepstream-app -c deepstream_app_config_copy.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/aarch64-linux-gnu/libthread_db.so.1”.
[Detaching after fork from child process 3810]
[New Thread 0xffffc317a900 (LWP 3811)]
[New Thread 0xffff8e6f0900 (LWP 3812)]
[New Thread 0xffff8dcb1900 (LWP 3813)]
[New Thread 0xffff8d39b900 (LWP 3814)]
[New Thread 0xffff8cb9a900 (LWP 3815)]
[New Thread 0xffff7fff9900 (LWP 3816)]
[New Thread 0xffff7f7f8900 (LWP 3817)]
[New Thread 0xffff7eff7900 (LWP 3818)]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
~~ CLOG[/dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvmultiobjecttracker/include/modules/NvMultiObjectTracker/NvTrackerParams.hpp, getConfigRoot() @line 52]: [NvTrackerParams::getConfigRoot()] !!![WARNING] Invalid low-level config file caused an exception, but will go ahead with the default config values
~~ CLOG[/dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvmultiobjecttracker/include/modules/NvMultiObjectTracker/NvTrackerParams.hpp, getConfigRoot() @line 52]: [NvTrackerParams::getConfigRoot()] !!![WARNING] Invalid low-level config file caused an exception, but will go ahead with the default config values
[NvMultiObjectTracker] Initialized
[New Thread 0xffff7e57e900 (LWP 3819)]
[New Thread 0xffff7dd7d900 (LWP 3820)]
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:44.096212972 3800 0xaaaaabc81d20 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/home/ninad/hump/models/hump_fp16.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT images 3x640x640
1 OUTPUT kFLOAT output0 5x8400

0:00:44.186805964 3800 0xaaaaabc81d20 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /home/ninad/hump/models/hump_fp16.engine
[New Thread 0xffff57cce900 (LWP 3915)]
[New Thread 0xffff574cd900 (LWP 3916)]
[New Thread 0xffff54d90900 (LWP 3917)]
0:00:44.363469298 3800 0xaaaaabc81d20 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/ninad/hump/config_infer_primary_yoloV8.txt sucessfully
[New Thread 0xffff1cba7900 (LWP 3918)]
[New Thread 0xffff17ff9900 (LWP 3919)]
[New Thread 0xffff177f8900 (LWP 3920)]
[New Thread 0xffff16ff7900 (LWP 3921)]
[New Thread 0xffff167f6900 (LWP 3922)]

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg)
**PERF: 0.00 (0.00)
** INFO: <bus_callback:239>: Pipeline ready

[New Thread 0xffff15ff5900 (LWP 3923)]
[New Thread 0xffff157f4900 (LWP 3924)]
[New Thread 0xffff14ff3900 (LWP 3925)]
[Detaching after vfork from child process 3926]
[Detaching after vfork from child process 3929]
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
[New Thread 0xfffef3ff9900 (LWP 3933)]
[New Thread 0xfffef37f8900 (LWP 3934)]
[New Thread 0xfffef2ff7900 (LWP 3935)]
NvMMLiteBlockCreate : Block : BlockType = 261
[New Thread 0xfffef27f6900 (LWP 3936)]
[New Thread 0xfffef1ff5900 (LWP 3937)]
** INFO: <bus_callback:225>: Pipeline running

[New Thread 0xfffef17f4900 (LWP 3938)]
[New Thread 0xfffef0ff3900 (LWP 3939)]
[New Thread 0xfffee3ff9900 (LWP 3940)]
**PERF: 0.00 (0.00)
**PERF: 0.00 (0.00)
**PERF: 0.00 (0.00)

Thread 12 “deepstream-app” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffff57cce900 (LWP 3915)]
0x0000ffff7d51ab68 in decodeTensorYolo(float const*, float const*, float const*, unsigned int const&, unsigned int const&, unsigned int const&, std::vector<float, std::allocator > const&) ()
from /home/ninad/hump/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
(gdb) bt
#0 0x0000ffff7d51ab68 in decodeTensorYolo(float const*, float const*, float const*, unsigned int const&, unsigned int const&, unsigned int const&, std::vector<float, std::allocator > const&) ()
at /home/ninad/hump/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
#1 0x0000ffff7d51afc4 in NvDsInferParseCustomYolo(std::vector<NvDsInferLayerInfo, std::allocator > const&, NvDsInferNetworkInfo const&, NvDsInferParseDetectionParams const&, std::vector<NvDsInferObjectDetectionInfo, std::allocator >&) () at /home/ninad/hump/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
#2 0x0000ffff7d51b280 in NvDsInferParseYolo () at /home/ninad/hump/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
#3 0x0000ffffc1b5c1d0 in () at /opt/nvidia/deepstream/deepstream-6.3/lib/libnvds_infer.so
#4 0x0000ffffc1b3f1d4 in () at /opt/nvidia/deepstream/deepstream-6.3/lib/libnvds_infer.so
#5 0x0000ffffc1b3f438 in () at /opt/nvidia/deepstream/deepstream-6.3/lib/libnvds_infer.so
#6 0x0000ffffc1b433b8 in nvdsinfer::NvDsInferContextImpl::dequeueOutputBatch(NvDsInferContextBatchOutput&) () at /opt/nvidia/deepstream/deepstream-6.3/lib/libnvds_infer.so
#7 0x0000ffffc1c53e7c in () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
#8 0x0000fffff7cd547c in () at /lib/aarch64-linux-gnu/libglib-2.0.so.0
#9 0x0000fffff70db624 in start_thread (arg=0xfffff7cf8e30) at pthread_create.c:477
#10 0x0000fffff6f6f49c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

7

It shows the crash is in the following code decodeTensorYolo, you need to check it according to your own code.

8

Hi Yuweiw,

I have complied them(custom library) with the respective CUDA version on my Jetson platform. Is there anything that i need to still?

9

You need to add some log to analyze your code in the decodeTensorYolo. There may be an NULL pointer or memory overread problem.

10

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.