deepstream SDK 4.0 Yolo Segmentation fault (core dumped)

che1428 · September 4, 2019, 2:07am

Hi, I am new to deepstream.

After the installation of tensorrt and deepstream, I tried to run yolo but failed.
Below are the details and the error message I got.

System details

Deepstream SDK 4.0
GPU: Tesla P40
Ubuntu 18.04
Nvidia driver 430.26
Cuda 10.1
Tensorrt 5.1.2.2

Yolo details

deepstream_app_config_yoloV3_tiny.txt (original one, didn’t make any adjustments)
yolov3-tiny.weights (original one, generated by prebuild.sh)
yolov3-tiny.cfg (original one, generated by prebuild.sh)

Error Message

XX:/opt/nvidia/deepstream/deepstream-4.0/sources/objectDetec tor_Yolo$ deepstream-app -c deepstream_app_config_yoloV3_tiny.txt
Creating LL OSD context new
0:00:09.116669062 8262 0x561d26e2d010 INFO nvinfer gstnvinfer.c pp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initi alize(): Trying to create engine from model files
Loading pre-trained weights…
Loading complete!
Total Number of weights read : 8858734
layer inp_size out_size weightPtr
(1) conv-bn-leaky 3 x 416 x 416 16 x 416 x 416 496
(2) maxpool 16 x 416 x 416 16 x 208 x 208 496
(3) conv-bn-leaky 16 x 208 x 208 32 x 208 x 208 5232
(4) maxpool 32 x 208 x 208 32 x 104 x 104 5232
(5) conv-bn-leaky 32 x 104 x 104 64 x 104 x 104 23920
(6) maxpool 64 x 104 x 104 64 x 52 x 52 23920
(7) conv-bn-leaky 64 x 52 x 52 128 x 52 x 52 98160
(8) maxpool 128 x 52 x 52 128 x 26 x 26 98160
(9) conv-bn-leaky 128 x 26 x 26 256 x 26 x 26 394096
(10) maxpool 256 x 26 x 26 256 x 13 x 13 394096
(11) conv-bn-leaky 256 x 13 x 13 512 x 13 x 13 1575792
(12) maxpool 512 x 13 x 13 512 x 13 x 13 1575792
(13) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 6298480
(14) conv-bn-leaky 1024 x 13 x 13 256 x 13 x 13 6561648
(15) conv-bn-leaky 256 x 13 x 13 512 x 13 x 13 7743344
(16) conv-linear 512 x 13 x 13 255 x 13 x 13 7874159
(17) yolo 255 x 13 x 13 255 x 13 x 13 7874159
(18) route - 256 x 13 x 13 7874159
(19) conv-bn-leaky 256 x 13 x 13 128 x 13 x 13 7907439
(20) upsample 128 x 13 x 13 128 x 26 x 26 -
(21) route - 384 x 26 x 26 7907439
(22) conv-bn-leaky 384 x 26 x 26 256 x 26 x 26 8793199
(23) conv-linear 256 x 26 x 26 255 x 26 x 26 8858734
(24) yolo 255 x 26 x 26 255 x 26 x 26 8858734
Output blob names :
yolo_17
yolo_24
Total number of layers: 50
Total number of layers on DLA: 0
Building the TensorRT Engine…
Building complete!
0:00:23.709590831 8262 0x561d26e2d010 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/model_b1_fp32.engine
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24

Runtime commands:
h: Print this help
q: Quit

    p: Pause
    r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg)
**PERF: 0.00 (0.00)
** INFO: <bus_callback:163>: Pipeline ready

Creating LL OSD context new
** INFO: <bus_callback:149>: Pipeline running

Segmentation fault (core dumped)

Please inform if any additional information is needed
Thanks a lot!

CJR · September 4, 2019, 6:36pm

Hi,

Have you verified if the default samples with resnet 10/18 work ?

che1428 · September 5, 2019, 2:51am

Sorry, not quite sure where is the resnet 10/18 located at nor how to test it
but I did try config_infer_primary.txt in the samples folder, and there is error too

As I am working on a ubuntu server without connecting any camera or streaming sources, thus I ignored the error and test the yolo model by videos.

The error messages are shown below

XX:/opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app$ deepstream-app -c config_infer_primary.txt

(deepstream-app:30279): GStreamer-CRITICAL **: 02:38:58.829: gst_element_get_static_pad: assertion ‘GST_IS_ELEMENT (element)’ failed
Segmentation fault (core dumped)

FYI, I have just run the yolo again but with debug mode this time (#1 - log all fatal errors)

XX:/opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo$ deepstream-app -c deepstream_app_config_yoloV3_tiny.txt --gst-debug=1

(skip)

** INFO: <bus_callback:163>: Pipeline ready

0:00:22.957843864 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.957887522 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.957936722 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.957947644 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.958022247 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.958045810 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.958104213 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.958127785 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.958180670 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:22.958205873 29512 0x7f0160013c50 ERROR v4l2 gstv4l2object.c:2072:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

Creating LL OSD context new
** INFO: <bus_callback:149>: Pipeline running

Segmentation fault (core dumped)

CJR · September 5, 2019, 9:23pm

Hi,

What installation method did you use to install Deepstream ? Can you try running the sample this config file - “source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt” and confirm that it works ?

che1428 · September 6, 2019, 2:39am

I used the DeepStream tar package method by the instructions.
https://docs.nvidia.com/metropolis/deepstream/4.0/dev-guide/index.html

I have also tried source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt, it doesn’t work.
Here is the output

XX:~/deepstream_sdk_v4.0_x86_64/samples/configs/deepstream-app$ deepstream-a pp -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
Creating LL OSD context new
0:00:09.125886936 19288 0x55bacb83a810 WARN nvinfer gstnvinfer.cpp:515:gst_nvinf er_logger:<secondary_gie_2> NvDsInferContext[UID 6]:useEngineFile(): Failed to read from model e ngine file
0:00:09.126035921 19288 0x55bacb83a810 INFO nvinfer gstnvinfer.cpp:519:gst_nvinf er_logger:<secondary_gie_2> NvDsInferContext[UID 6]:initialize(): Trying to create engine from m odel files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:19.694745347 19288 0x55bacb83a810 INFO nvinfer gstnvinfer.cpp:519:gst_nvinf er_logger:<secondary_gie_2> NvDsInferContext[UID 6]:generateTRTModel(): Storing the serialized c uda engine to file at /home/ubuntu/deepstream_sdk_v4.0_x86_64/samples/models/Secondary_CarMake/r esnet18.caffemodel_b16_int8.engine
0:00:19.710786806 19288 0x55bacb83a810 WARN nvinfer gstnvinfer.cpp:515:gst_nvinf er_logger:<secondary_gie_1> NvDsInferContext[UID 5]:useEngineFile(): Failed to read from model e ngine file
0:00:19.710807514 19288 0x55bacb83a810 INFO nvinfer gstnvinfer.cpp:519:gst_nvinf er_logger:<secondary_gie_1> NvDsInferContext[UID 5]:initialize(): Trying to create engine from m odel files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:27.948289470 19288 0x55bacb83a810 INFO nvinfer gstnvinfer.cpp:519:gst_nvinf er_logger:<secondary_gie_1> NvDsInferContext[UID 5]:generateTRTModel(): Storing the serialized c uda engine to file at /home/ubuntu/deepstream_sdk_v4.0_x86_64/samples/models/Secondary_CarColor/ resnet18.caffemodel_b16_int8.engine
0:00:27.958422719 19288 0x55bacb83a810 WARN nvinfer gstnvinfer.cpp:515:gst_nvinf er_logger:<secondary_gie_0> NvDsInferContext[UID 4]:useEngineFile(): Failed to read from model e ngine file
0:00:27.958445676 19288 0x55bacb83a810 INFO nvinfer gstnvinfer.cpp:519:gst_nvinf er_logger:<secondary_gie_0> NvDsInferContext[UID 4]:initialize(): Trying to create engine from m odel files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:35.614116883 19288 0x55bacb83a810 INFO nvinfer gstnvinfer.cpp:519:gst_nvinf er_logger:<secondary_gie_0> NvDsInferContext[UID 4]:generateTRTModel(): Storing the serialized c uda engine to file at /home/ubuntu/deepstream_sdk_v4.0_x86_64/samples/models/Secondary_VehicleTy pes/resnet18.caffemodel_b16_int8.engine
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt .so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
0:00:35.629899707 19288 0x55bacb83a810 WARN nvinfer gstnvinfer.cpp:515:gst_nvinf er_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:useEngineFile(): Failed to read from model engine file
0:00:35.629929872 19288 0x55bacb83a810 INFO nvinfer gstnvinfer.cpp:519:gst_nvinf er_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:40.199685583 19288 0x55bacb83a810 INFO nvinfer gstnvinfer.cpp:519:gst_nvinf er_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Storing the seria lized cuda engine to file at /home/ubuntu/deepstream_sdk_v4.0_x86_64/samples/models/Primary_Dete ctor/resnet10.caffemodel_b4_int8.engine

Runtime commands:
h: Print this help
q: Quit

    p: Pause
    r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the sour ce.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg)
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:163>: Pipeline ready

Creating LL OSD context new
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
Segmentation fault (core dumped)

CJR · September 6, 2019, 5:35pm

Hi,

Did you remove gstreamer cache as suggested in the README ? If not, can you try it again ?
Can you check if any of the plugins have been blacklisted ? You can use the command “gst-inspect-1.0 -b”
If you still see the segfault can you try running it using gdb and check if you can get a backtrace ?

che1428 · September 9, 2019, 8:52am

Sorry, I miss telling one important thing. I did all these processes by accessing the server through putty and xming. As far as I know, gstreamer requires special configurations to project the output remotely. So I am not sure if this may affect or even cause the error.

I have deleted the cache and tried to run again. The error remains the same.
No blacklisted plugins
Sorry, I totally have no idea how to use gdb. Would you like to show me how if this is essential?

CJR · September 16, 2019, 6:38pm

If you are working with a headless server, you can switch the sink from EGL sink to either fake sink / file output / RTSP streaming.

These configurations are present in your “source4_1080p_dec_infer-esnet_tracker_sgie_tiled_display_int8.txt” file but EGL sink is enabled by default. You can also find more information about sink group config options here - [url]DeepStream Development Guide

che1428 · September 24, 2019, 7:22am

The app run successfully after switch from EGL sink to fake sink. (source4_1080p_dec_infer-esnet_tracker_sgie_tiled_display_int8.txt)
However, there is no video output at all. Is this normal?

P.S. The app run failed after switch to file output

Also, one more important question, as my ultimate goal is to use yolov3 deepstream.
It still failed after switching to fake sink

Creating LL OSD context new
0:00:02.864178222 11866 0x55cb71816240 ERROR nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Could not open custom lib: /home/ubuntu/deepstream_sdk_v4.0_x86_64/sources/objectDetector_Yolo/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so: cannot open shared object file: No such file or directory
0:00:02.864267559 11866 0x55cb71816240 WARN nvinfer gstnvinfer.cpp:692:gst_nvinfer_start:<primary_gie_classifier> error: Failed to create NvDsInferContext instance
0:00:02.864284742 11866 0x55cb71816240 WARN nvinfer gstnvinfer.cpp:692:gst_nvinfer_start:<primary_gie_classifier> error: Config file path: /home/ubuntu/deepstream_sdk_v4.0_x86_64/sources/objectDetector_Yolo/config_infer_primary_yoloV3.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED
** ERROR: main:651: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie_classifier: Failed to create NvDsInferContext instance
Debug info: gstnvinfer.cpp(692): gst_nvinfer_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie_classifier:
Config file path: /home/ubuntu/deepstream_sdk_v4.0_x86_64/sources/objectDetector_Yolo/config_infer_primary_yoloV3.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED
App run failed

CJR · September 24, 2019, 6:07pm

Hi,

fakesink by default does not ouput anything, so the behavior you are seeing is expected. Regarding yolov3 error, the message says “NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED”

Can you check if you built the custom library as mentioned in the yolo sample README ?

Compile the custom library:
  # Export right CUDA version (e.g. 10.1, 10.0)
  $ export CUDA_VER=10.1
  $ make -C nvdsinfer_custom_impl_Yolo