My problem: I come across the error with message Segmentation Fault (core dumped) in every DeepStream pipeline I run in Jetson AGX Orin. These pipelines were successfully executing in x86 devices but I cannot make any of them running on Jetson AGX Orin.
For this issue, I tried to run deepstream-test5 but I also tested DeepStream-YOLO examples and other samples, and came across the same error: Segmentation Fault (core dumped) and I couldn’t find the root cause yet. I did not perform any action other than sample READMEs and official docs.
Any kind of help is appreciated.
Output really confused me because I confirmed that my RTSP link is valid since I can play it using VLC player. Here are the command and the logs.
./deepstream-test5-app -c configs/test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt -p 0
0:00:07.136738154 7822 0xaaaae080ed30 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/deepstream-test5/configs/../../../../../samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b2_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x544x960
1 OUTPUT kFLOAT output_bbox/BiasAdd 16x34x60
2 OUTPUT kFLOAT output_cov/Sigmoid 4x34x60
0:00:07.540946288 7822 0xaaaae080ed30 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/deepstream-test5/configs/../../../../../samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b2_gpu0_int8.engine
0:00:07.549883941 7822 0xaaaae080ed30 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/deepstream-test5/configs/../../../../../samples/configs/deepstream-app/config_infer_primary.txt sucessfully
Runtime commands:
h: Print this help
q: Quit
p: Pause
r: Resume
NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.
Active sources : 0
**PERF: FPS 0 (Avg) FPS 1 (Avg)
Thu May 30 07:35:07 2024
**PERF: 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:301>: Pipeline ready
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
Stream format not found, dropping the frame
** INFO: <bus_callback:287>: Pipeline running
Segmentation fault (core dumped)
My configuration:
Here are the differences w/ /opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/deepstream-test5/configs/test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt. I disabled few things to make pipeline simpler.
- RTSP links are updated in
source0andsource1. sink1is disabled. (cloud messages)trackeris disabled.secondary-gie0andsecondary-gie1are disabled.
################################################################################
# Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################
[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl
[tiled-display]
enable=1
rows=1
columns=2
width=1280
height=720
gpu-id=0
nvbuf-memory-type=0
[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://172.16.44.172:8554/mystream
num-sources=1
gpu-id=0
nvbuf-memory-type=0
# smart record specific fields, valid only for source type=4
# 0 = disable, 1 = through cloud events, 2 = through cloud + local events
#smart-record=1
# 0 = mp4, 1 = mkv
#smart-rec-container=0
#smart-rec-file-prefix
#smart-rec-dir-path
# smart record cache size in seconds
#smart-rec-cache
# default duration of recording in seconds.
#smart-rec-default-duration
# duration of recording in seconds.
# this will override default value.
#smart-rec-duration
# seconds before the current time to start recording.
#smart-rec-start-time
# value in seconds to dump video stream.
#smart-rec-interval
[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://172.16.44.172:8554/mystream
num-sources=1
gpu-id=0
nvbuf-memory-type=0
# smart record specific fields, valid only for source type=4
# 0 = disable, 1 = through cloud events, 2 = through cloud + local events
#smart-record=1
# 0 = mp4, 1 = mkv
#smart-rec-container=0
#smart-rec-file-prefix
#smart-rec-dir-path
# smart record cache size in seconds
#smart-rec-cache
[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0
[sink1]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvdrmvideosink 6=MsgConvBroker
type=6
msg-conv-config=dstest5_msgconv_sample_config.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(2): PAYLOAD_DEEPSTREAM_PROTOBUF - Deepstream schema protobuf encoded payload
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM - Custom schema payload
msg-conv-payload-type=0
#(0): Create payload using NvdsEventMsgMeta
#(1): New Api to create payload using NvDsFrameMeta
msg-conv-msg2p-new-api=0
#Frame interval at which payload is generated
msg-conv-frame-interval=30
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so
#Provide your msg-broker-conn-str here
msg-broker-conn-str=<host>;<port>;<topic>
topic=<topic>
#Optional:
#msg-broker-config=/opt/nvidia/deepstream/deepstream/sources/libs/kafka_protocol_adaptor/cfg_kafka.txt
#new-api=0
#(0) Use message adapter library api's
#(1) Use new msgbroker library api's
[sink2]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265 3=mpeg4
## only SW mpeg4 is supported right now.
codec=3
sync=1
bitrate=2000000
output-file=out.mp4
source-id=0
# sink type = 6 by default creates msg converter + broker.
# To use multiple brokers use this group for converter and use
# sink type = 6 with disable-msgconv = 1
[message-converter]
enable=0
msg-conv-config=dstest5_msgconv_sample_config.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(2): PAYLOAD_DEEPSTREAM_PROTOBUF - Deepstream schema protobuf payload
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM - Custom schema payload
msg-conv-payload-type=0
# Name of library having custom implementation.
#msg-conv-msg2p-lib=<val>
# Id of component in case only selected message to parse.
#msg-conv-comp-id=<val>
# Configure this group to enable cloud message consumer.
[message-consumer0]
enable=0
proto-lib=/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so
conn-str=<host>;<port>
config-file=/opt/nvidia/deepstream/deepstream/sources/libs/kafka_protocol_adaptor/cfg_kafka.txt
subscribe-topic-list=<topic1>;<topic2>;<topicN>
# Use this option if message has sensor name as id instead of index (0,1,2 etc.).
#sensor-list-file=dstest5_msgconv_sample_config.txt
[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0
[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=2
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1
# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
#Required to display the PGIE labels, should be added even when using config-file
#property
batch-size=2
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
#Required by the app for SGIE, when used along with config-file property
gie-unique-id=1
nvbuf-memory-type=0
model-engine-file=../../../../../samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b2_gpu0_int8.engine
labelfile-path=../../../../../samples/models/Primary_Detector/labels.txt
config-file=../../../../../samples/configs/deepstream-app/config_infer_primary.txt
#infer-raw-output-dir=../../../../../samples/primary_detector_raw_output/
[tracker]
enable=0
# For NvDCF and NvDeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=960
tracker-height=544
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
# ll-config-file required to set different tracker types
# ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_IOU.yml
# ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_NvSORT.yml
ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml
# ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_NvDCF_accuracy.yml
# ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_NvDeepSORT.yml
gpu-id=0
display-tracking-id=1
[secondary-gie0]
enable=0
gpu-id=0
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;
batch-size=16
config-file=../../../../../samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt
labelfile-path=../../../../../samples/models/Secondary_VehicleTypes/labels.txt
model-engine-file=../../../../../samples/models/Secondary_VehicleTypes/resnet18_vehicletypenet.etlt_b16_gpu0_int8.engine
[secondary-gie1]
enable=0
gpu-id=0
gie-unique-id=5
operate-on-gie-id=1
operate-on-class-ids=0;
batch-size=16
config-file=../../../../../samples/configs/deepstream-app/config_infer_secondary_vehiclemake.txt
labelfile-path=../../../../../samples/models/Secondary_VehicleMake/labels.txt
model-engine-file=../../../../../samples/models/Secondary_VehicleMake/resnet18_vehiclemakenet.etlt_b16_gpu0_int8.engine
[tests]
file-loop=0
Steps to recreate environment & reproduce the issue:
First, I start a Docker container as mentioned here.
xhost +
docker run -it --rm --runtime nvidia --detach --net=host --gpus all -e DISPLAY=$DISPLAY --device /dev/snd -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/deepstream:6.4-triton-multiarch
docker ps
docker exec -it <CONTAINER-ID> bash
Then I run these commands to install some of the requirements.
apt-get update
apt-get install -y kmod # for the logs about lsmod and so
/opt/nvidia/deepstream/deepstream-6.4/user_additional_install.sh # for the errors about decoders
I go into the example deepstream-test5 and install & build the required packages.
# from the README of deepstream-test5
apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev \
libgstrtspserver-1.0-dev libx11-dev libjson-glib-dev libyaml-cpp-dev -y
# from the README of kafka_protocol_adaptor
git clone https://github.com/confluentinc/librdkafka.git
cd librdkafka
git checkout tags/v2.2.0
./configure # --enable-ssl
make
make install
cp /usr/local/lib/librdkafka* /opt/nvidia/deepstream/deepstream/lib/
ldconfig
# from the README of kafka_protocol_adaptor
apt-get install libglib2.0 libglib2.0-dev -y
apt-get install libjansson4 libjansson-dev -y
apt-get install libssl-dev -y
apt-get install protobuf-compiler -y
# from the README of deepstream-test5
cd /opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/deepstream-test5
export CUDA_VER=12.2
make
Then I configure the pipeline /opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/deepstream-test5/configs/test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt as I stated in the beginning of the post and run the pipeline. These were all the steps I go through before running the pipeline.
My setup:
• Hardware Platform: Jetson AGX Orin 64 Gb
• DeepStream Version: 6.4 - Docker image: nvcr.io/nvidia/deepstream:6.4-triton-multiarch
• JetPack Version: 6
• TensorRT Version: 8.6.2.3-1+cuda12.2
• Issue Type: Bug