DeepStream Pipeline Crashes with “Segmentation Fault (Core Dumped)” After Adding Custom Alignment Plugin

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Hello @junshengy,

I’ve been working on a DeepStream-based face recognition pipeline, which was functioning properly until I integrated a custom NVIDIA plugin for face alignment, as suggested by the GitHub author. The custom nvinfer plugin was installed from the following GitHub repository: gst-nvinfer-custom. The goal was to improve embedding similarity scores, but I had to abandon this approach due to compatibility issues. Installing the custom nvinfer plugin replaced the default version of nvinfer that comes with DeepStream 7.0, causing several related issues.

You can find more details and prior discussions regarding this issue here: Issues with Face Recognition - NVIDIA Developer Forum.

Steps Taken:

  1. Issue Encountered:
  • After installing the custom nvinfer plugin, DeepStream started throwing errors related to nvinfer.
  1. Resolution Attempted:
  • I uninstalled the custom nvinfer and reinstalled the original version of nvinfer that comes with DeepStream 7.0.
  • This restored compatibility and resolved some of the errors, but now the pipeline crashes with a “Segmentation fault (core dumped)” error.

Pipeline Behavior:

  • The issue is non-deterministic, meaning every run produces slightly different logs, but the pipeline consistently crashes.
  • The crash occurs even with previously working scripts.

Logs:

Here are the logs when running the pipeline:

Log 1:

dstream@tokkio:~/Documents/Deep_Stream_App$ GST_DEBUG=3 python3 app.py

Creating DeepStream Face Detection Pipeline

Pipeline created successfully

streammux set_property gpu_id 0

streammux set_property batch-size 1

streammux set_property width 1920

streammux set_property height 1080

/home/dstream/Documents/Deep_Stream_App/app.py:88: DeprecationWarning: Gst.Element.get_request_pad is deprecated

sinkpad = streammux.get_request_pad(“sink_0”)

Linking Source Bin to StreamMuxer

pgie set_property config-file-path /home/dstream/Documents/Deep_Stream_App/configs/cofig_infer_yolov8_forum.txt

After creating elements linking of elements is started

gstnvtracker: Loading low-level lib at /home/dstream/Desktop/DEEP_STREAM/libraries/libnvds_nvmultiobjecttracker.so

[NvMultiObjectTracker] Initialized

…

ERROR egladaption ext/eglgles/gstegladaptation.c:669:gst_egl_adaptation_choose_config: Could not find matching framebuffer config

ERROR egladaption ext/eglgles/gstegladaptation.c:683:gst_egl_adaptation_choose_config: Couldn’t choose an usable config

ERROR nveglglessink ext/eglgles/gsteglglessink.c:3233:gst_eglglessink_configure_caps: Couldn’t choose EGL config

ERROR nveglglessink ext/eglgles/gsteglglessink.c:3293:gst_eglglessink_configure_caps: Configuring caps failed

ERROR nveglglessink ext/eglgles/gsteglglessink.c:3338:gst_eglglessink_setcaps: Failed to configure caps

Segmentation fault (core dumped)

Log 2 (for additional context):

dstream@tokkio:~/Documents/Deep_Stream_App$ python3 app.py

Creating DeepStream Face Detection Pipeline

Pipeline created successfully

streammux set_property gpu_id 0

streammux set_property batch-size 1

streammux set_property width 1920

streammux set_property height 1080

/home/dstream/Documents/Deep_Stream_App/app.py:88: DeprecationWarning: Gst.Element.get_request_pad is deprecated

sinkpad = streammux.get_request_pad(“sink_0”)

Linking Source Bin to StreamMuxer

pgie set_property config-file-path /home/dstream/Documents/Deep_Stream_App/configs/cofig_infer_yolov8_forum.txt

After creating elements linking of elements is started

libEGL warning: DRI2: failed to authenticate

gstnvtracker: Loading low-level lib at /home/dstream/Desktop/DEEP_STREAM/libraries/libnvds_nvmultiobjecttracker.so

[NvMultiObjectTracker] Initialized

0:00:08.500441814 19128 0x561a229a6c00 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/home/dstream/Documents/Deep_Stream_App/models/yolov8n/yolov8n-face-dynamic_batch_640.onnx_b1_gpu0_fp32.engine

WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.

INFO: [Implicit Engine Info]: layers num: 2

0 INPUT kFLOAT images 3x640x640

1 OUTPUT kFLOAT trans_out 8400x5

0:00:08.724623038 19128 0x561a229a6c00 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2003> [UID = 1]: Use deserialized engine model: /home/dstream/Documents/Deep_Stream_App/models/yolov8n/yolov8n-face-dynamic_batch_640.onnx_b1_gpu0_fp32.engine

0:00:08.730989907 19128 0x561a229a6c00 INFO nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus: [UID 1]: Load new model:/home/dstream/Documents/Deep_Stream_App/configs/cofig_infer_yolov8_forum.txt sucessfully

Decodebin child added: source

Decodebin child added: decodebin0

Decodebin child added: nvjpegdec0

Using GPU 0 (NVIDIA RTXA6000-24C, 84 SMs, 1536 th/SM max, CC 8.6, ECC on)

New pad callback

Pad type: video/x-raw

Segmentation fault (core dumped)

Observations:

  • The pipeline fails with “Segmentation fault (core dumped)” error.

  • The nveglglessink component throws multiple errors related to EGL configuration failure, such as:

    • “Could not find matching framebuffer config”

    • “Couldn’t choose a usable config”

    • “Configuring caps failed”

Questions:

  1. Could there be lingering effects from the custom plugin installation that are still causing conflicts with the video sink, especially in terms of EGL configuration or pad activation during the pipeline’s execution?
  • Although the custom nvinfer plugin has been uninstalled, the pipeline is still encountering issues with rendering video through nveglglessink. Could there be residual configuration changes or incorrect EGL settings that are affecting the video sink?
  1. What are the recommended steps to analyze and resolve core dumps in a DeepStream application? Any best practices for debugging the issue?
  • What tools or techniques can be used to debug and analyze core dumps in DeepStream applications to pinpoint the source of the crash?
  1. Can you suggest the root cause of the gst_eglglessink_setcaps errors, and what are potential solutions for this issue?
  • Could these errors be linked to misconfigurations in the EGL context or an incompatibility with the custom nvinfer plugin? How can this be fixed to get the pipeline running again?
3

Please back up your data and re-burn Jetpack and Deepstream. You have damaged system compatibility and we cannot determine the problem.

4

"My environment is:

  • Hardware Platform (Jetson / GPU): A6000 VGPU
  • DeepStream Version: 7
  • JetPack Version (valid for Jetson only): N/A
  • TensorRT Version: 8.6.1
  • CUDA Version: 12.2
  • NVIDIA GPU Driver Version: 535.216.01

I’m not working on a dedicated GPU; I am working on a VGPU.

I believe that in this case, we can’t modify the driver. What else can we re-burn along with DeepStream?

Regarding the driver: While I can modify the driver, the version for dedicated GPUs differs from that for vGPUs. Installing a dedicated GPU driver on a vGPU setup might cause version conflicts and other issues, as these drivers are optimized for different hardware configurations. Would it be advisable to install the correct vGPU driver version, or should we avoid modifying the driver altogether? Additionally, if we can modify the driver version for the vGPU, could you please confirm the exact vGPU driver version I should install for my setup?

5

In the previous topics, you are using the Jetson platform, but this problem occurs on the vGPU, right?

Generally speaking, there should be no difference between vGPU and physical GPU. If you install the driver and nvidia-container-toolkit, and run deepstream-test1 in the container, does it work properly?

If this problem is caused by the custom nvinfer plugin, you need to ask the author on github. So let’s first confirm that nvinfer can work properly on your vGPU

6

I ran the sample app deepstream-test1, which is working fine, and I am able to see the output. I deleted the custom nvinfer plugin and reinstalled the nvinfer plugin.

How can I find the exact issue I’m facing and solve it?

7

Forget about the custom nvinfer plugin for now, it is not relevant here.

Reinstall deepstream sdk first.Before installation, execute the following script

sudo  /opt/nvidia/deepstream/deepstream/uninstall.sh

If it worked before, please try to reinstall the GPU driver and dependencies, maybe some packages are damaged.

8

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.