Deepstream 6.4 issue

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

Ubuntu 22.04

GStreamer 1.20.3

NVIDIA driver 535.104.12

CUDA 12.2

TensorRT 8.6.1.6

Hi,
Problem with deepstream 6.4

0:01:19.970225789 1577 0x55e78a3e2390 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<face(‘face’, ‘tracker_face’)> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2138> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-6.4/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b30_gpu0_int8.engine successfully
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x544x960
1 OUTPUT kFLOAT output_bbox/BiasAdd 16x34x60
2 OUTPUT kFLOAT output_cov/Sigmoid 4x34x60

0:01:20.127324107 1577 0x55e78a3e2390 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<face(‘face’, ‘tracker_face’)> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-6.4/nvodin23/models/assets/face/face.txt sucessfully
terminate called after throwing an instance of ‘std::runtime_error’
what(): Unable to read configuration
Aborted (core dumped)

2

Hi,
I am waiting for your response.

3

Can you share the reproducing steps before this issue happens?
Is the sample deepstream-test1 able to run in your host?

5

I am not facing this while running the deepstream-6.4 sample apps, this error occurs while i am running my custom python code.
The error is not clear to debug, and I have also checked all the config file paths as well.

6

Could you use gdb tool to initially locate the Aborted (core dumped) issue?

7

Could you suggest the steps?

8 
$ gdb your_python.py
$ r  your_parameters
$ bt
9

I dont have any parameters

10

This is the output log, If this means anything to you?

Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type “show copying” and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from python3…
(No debugging symbols found in python3)
Starting program: /usr/bin/python3 server.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.
[New Thread 0x7ffff3dff640 (LWP 2067)]
[New Thread 0x7ffff35fe640 (LWP 2068)]
[New Thread 0x7ffff0dfd640 (LWP 2069)]
[New Thread 0x7fffee5fc640 (LWP 2070)]
[New Thread 0x7fffebdfb640 (LWP 2071)]
[New Thread 0x7fffe75fa640 (LWP 2072)]
[New Thread 0x7fffe4df9640 (LWP 2073)]
[New Thread 0x7fffe45f8640 (LWP 2074)]
[New Thread 0x7fffe3df7640 (LWP 2075)]
[New Thread 0x7fffdf5f6640 (LWP 2076)]
[New Thread 0x7fffdcdf5640 (LWP 2077)]
[New Thread 0x7fffda5f4640 (LWP 2078)]
[New Thread 0x7fffd7df3640 (LWP 2079)]
[New Thread 0x7fffd55f2640 (LWP 2080)]
[New Thread 0x7fffd2df1640 (LWP 2081)]
[New Thread 0x7fff50984640 (LWP 2082)]
[New Thread 0x7fff50183640 (LWP 2083)]
[New Thread 0x7fff4d982640 (LWP 2084)]
[New Thread 0x7fff4b181640 (LWP 2085)]
[New Thread 0x7fff46980640 (LWP 2086)]
[New Thread 0x7fff4617f640 (LWP 2087)]
[New Thread 0x7fff4397e640 (LWP 2088)]
[New Thread 0x7fff4117d640 (LWP 2089)]
[New Thread 0x7fff3e97c640 (LWP 2090)]
[New Thread 0x7fff3c17b640 (LWP 2091)]
[New Thread 0x7fff3997a640 (LWP 2092)]
[New Thread 0x7fff37179640 (LWP 2093)]
[New Thread 0x7fff32978640 (LWP 2094)]
[New Thread 0x7fff30177640 (LWP 2095)]
[New Thread 0x7fff2d976640 (LWP 2096)]
[Detaching after vfork from child process 2097]
[Detaching after vfork from child process 2098]
[New Thread 0x7fff1cfff640 (LWP 2099)]
[New Thread 0x7fff17fff640 (LWP 2100)]
SERVER CONFIGGG <lib.config_reader.ConfigReader object at 0x7fff2a734400>
MODELLLLLLLLL face
PROCESSORRR <frame_processor.FramesProcessor object at 0x7fff2a34d2a0>
[New Thread 0x7fff177fe640 (LWP 2101)]
[New Thread 0x7fff16ffd640 (LWP 2102)]
[New Thread 0x7fff167fc640 (LWP 2103)]

[New Thread 0x7fff15ffb640 (LWP 2104)]
CAllin g the processorr
[New Thread 0x7fff157fa640 (LWP 2105)]
[New Thread 0x7fff14ff9640 (LWP 2106)]
[New Thread 0x7ffef7fff640 (LWP 2107)]
[New Thread 0x7ffef77fe640 (LWP 2108)]
[New Thread 0x7ffef6ffd640 (LWP 2109)]
[New Thread 0x7ffef67fc640 (LWP 2110)]
[New Thread 0x7ffef5ffb640 (LWP 2111)]
[New Thread 0x7ffef57fa640 (LWP 2112)]
[New Thread 0x7ffef4ff9640 (LWP 2113)]
[New Thread 0x7ffed7fff640 (LWP 2114)]
Creating streamux

[New Thread 0x7ffed49ff640 (LWP 2115)]
Creating source_bin 0

Creating Tee

Creating nvstreamdemuxface defaultdict(<class ‘list’>, {‘face’: [‘cam007’]})
[New Thread 0x7ffebbbff640 (LWP 2116)]
Saving pipeline graph in folder None
SET CALL BACK CALLED 1
Starting pipeline
[New Thread 0x7ffeba3dd640 (LWP 2117)]
[New Thread 0x7ffeb8bbb640 (LWP 2118)]
[New Thread 0x7ffeadfff640 (LWP 2119)]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-6.4/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Initialized
[New Thread 0x7ffead7fe640 (LWP 2120)]
[New Thread 0x7ffeacffd640 (LWP 2121)]
0:00:00.262534026 2058 0x5555584b9520 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<face(‘face’, ‘tracker_face’)> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1243> [UID = 2]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
0:00:04.420638828 2058 0x5555584b9520 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<face(‘face’, ‘tracker_face’)> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 2]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.4/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b30_gpu0_int8.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x544x960
1 OUTPUT kFLOAT output_bbox/BiasAdd 16x34x60
2 OUTPUT kFLOAT output_cov/Sigmoid 4x34x60

0:00:04.540157882 2058 0x5555584b9520 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<face(‘face’, ‘tracker_face’)> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 2]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.4/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b30_gpu0_int8.engine
[New Thread 0x7ffea3fff640 (LWP 2122)]
[New Thread 0x7ffea37fe640 (LWP 2123)]
[New Thread 0x7ffea2ffd640 (LWP 2124)]
0:00:04.545466271 2058 0x5555584b9520 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<face(‘face’, ‘tracker_face’)> [UID 2]: Load new model:/opt/nvidia/deepstream/deepstream-6.4/models/assets/face/face.txt sucessfully
[New Thread 0x7ffea27fc640 (LWP 2125)]
[New Thread 0x7ffea1ffb640 (LWP 2126)]
[New Thread 0x7ffea17fa640 (LWP 2127)]
[Thread 0x7fff2d976640 (LWP 2096) exited]
[Thread 0x7fff30177640 (LWP 2095) exited]
[Thread 0x7fff32978640 (LWP 2094) exited]
[Thread 0x7fff37179640 (LWP 2093) exited]
[Thread 0x7fff3997a640 (LWP 2092) exited]
[Thread 0x7fff3c17b640 (LWP 2091) exited]
[Thread 0x7fff3e97c640 (LWP 2090) exited]
[Thread 0x7fff4117d640 (LWP 2089) exited]
[Thread 0x7fff4397e640 (LWP 2088) exited]
[Thread 0x7fff4617f640 (LWP 2087) exited]
[Thread 0x7fff46980640 (LWP 2086) exited]
[Thread 0x7fff4b181640 (LWP 2085) exited]
[Thread 0x7fff4d982640 (LWP 2084) exited]
[Thread 0x7fff50183640 (LWP 2083) exited]
[Thread 0x7fff50984640 (LWP 2082) exited]
[Thread 0x7fffd2df1640 (LWP 2081) exited]
[Thread 0x7fffd55f2640 (LWP 2080) exited]
[Thread 0x7fffd7df3640 (LWP 2079) exited]
[Thread 0x7fffda5f4640 (LWP 2078) exited]
[Thread 0x7fffdcdf5640 (LWP 2077) exited]
[Thread 0x7fffdf5f6640 (LWP 2076) exited]
[Thread 0x7fffe3df7640 (LWP 2075) exited]
[Thread 0x7fffe45f8640 (LWP 2074) exited]
[Thread 0x7fffe4df9640 (LWP 2073) exited]
[Thread 0x7fffe75fa640 (LWP 2072) exited]
[Thread 0x7fffebdfb640 (LWP 2071) exited]
[Thread 0x7fffee5fc640 (LWP 2070) exited]
[Thread 0x7ffff0dfd640 (LWP 2069) exited]
[Thread 0x7ffff35fe640 (LWP 2068) exited]
[Thread 0x7ffff3dff640 (LWP 2067) exited]
[Detaching after fork from child process 2128]
[New Thread 0x7fffd2df1640 (LWP 2129)]
terminate called after throwing an instance of ‘std::runtime_error’
what(): Unable to read configuration

Thread 61 “pool-python3” received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffd2df1640 (LWP 2129)]
0x00007ffff7cde9fc in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

11

@yingliu, @yuweiw, please help me find a solution to the problem!

12

It is a cpp error, you have let me know what could possibly be causing this?

13

Could you use bt as I attached to print out the stack information? Or you can provide us an environment in which we can reproduce this problem.

14

Starting pipeline
[New Thread 0x7ffeba3dd640 (LWP 11696)]
[New Thread 0x7ffeb9bdc640 (LWP 11697)]
[New Thread 0x7ffeadfff640 (LWP 11698)]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-6.4/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Initialized
[New Thread 0x7ffead7fe640 (LWP 11699)]
[New Thread 0x7ffeacffd640 (LWP 11700)]
0:00:00.251654599 11637 0x5555581b2e00 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<face(‘face’, ‘tracker_face’)> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1243> [UID = 2]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
b0:00:04.446475068 11637 0x5555581b2e00 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<face(‘face’, ‘tracker_face’)> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 2]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.4/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b30_gpu0_int8.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x544x960
1 OUTPUT kFLOAT output_bbox/BiasAdd 16x34x60
2 OUTPUT kFLOAT output_cov/Sigmoid 4x34x60

0:00:04.565629212 11637 0x5555581b2e00 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<face(‘face’, ‘tracker_face’)> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 2]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.4/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b30_gpu0_int8.engine
[New Thread 0x7ffea5fff640 (LWP 11701)]
[New Thread 0x7ffea57fe640 (LWP 11702)]
[New Thread 0x7ffea4ffd640 (LWP 11703)]
0:00:04.571082202 11637 0x5555581b2e00 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<face(‘face’, ‘tracker_face’)> [UID 2]: Load new model:/opt/nvidia/deepstream/deepstream-6.4/nvodin23/models/assets/face/face.txt sucessfully
[New Thread 0x7ffe7edfd640 (LWP 11704)]
[New Thread 0x7ffe7e5fc640 (LWP 11705)]
[New Thread 0x7ffe7ddfb640 (LWP 11706)]
[Thread 0x7fff2f976640 (LWP 11675) exited]
[Thread 0x7fff30177640 (LWP 11674) exited]
[Thread 0x7fff32978640 (LWP 11673) exited]
[Thread 0x7fff35179640 (LWP 11672) exited]
[Thread 0x7fff3997a640 (LWP 11671) exited]
[Thread 0x7fff3c17b640 (LWP 11670) exited]
[Thread 0x7fff3c97c640 (LWP 11669) exited]
[Thread 0x7fff3f17d640 (LWP 11668) exited]
[Thread 0x7fff4a980640 (LWP 11665) exited]
[Thread 0x7fff4b982640 (LWP 11663) exited]
[Thread 0x7fff50984640 (LWP 11661) exited]
[Thread 0x7fff4397e640 (LWP 11667) exited]
[Thread 0x7fff4417f640 (LWP 11666) exited]
[Thread 0x7fff50183640 (LWP 11662) exited]
[Thread 0x7ffff0dfd640 (LWP 11648) exited]
[Thread 0x7fff4b181640 (LWP 11664) exited]
[Thread 0x7ffff35fe640 (LWP 11647) exited]
[Thread 0x7fffe4df9640 (LWP 11652) exited]
[Thread 0x7fffe75fa640 (LWP 11651) exited]
[Thread 0x7fffd85f4640 (LWP 11657) exited]
[Thread 0x7fffd0df1640 (LWP 11660) exited]
[Thread 0x7fffd35f2640 (LWP 11659) exited]
[Thread 0x7fffd5df3640 (LWP 11658) exited]
[Thread 0x7fffdadf5640 (LWP 11656) exited]
[Thread 0x7fffdd5f6640 (LWP 11655) exited]
[Thread 0x7fffe1df7640 (LWP 11654) exited]
[Thread 0x7fffe25f8640 (LWP 11653) exited]
[Thread 0x7fffe9dfb640 (LWP 11650) exited]
[Thread 0x7fffee5fc640 (LWP 11649) exited]
[Thread 0x7ffff3dff640 (LWP 11646) exited]
[Detaching after fork from child process 11707]
[New Thread 0x7fffd0df1640 (LWP 11708)]
terminate called after throwing an instance of ‘std::runtime_error’
what(): Unable to read configuration

Thread 61 “pool-python3” received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffd0df1640 (LWP 11708)]
0x00007ffff7cde9fc in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff7cde9fc in pthread_kill () at /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff7c8a476 in raise () at /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff7c707f3 in abort () at /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff476bb9e in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff477720c in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff47761e9 in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff4776959 in __gxx_personality_v0 () at /lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fff79076fe9 in __libunwind_Unwind_Resume () at /lib/x86_64-linux-gnu/libunwind.so.8
#8 0x00007ffff3dd386d in () at /lib/x86_64-linux-gnu/libproxy.so.1
#9 0x00007ffff3ddc827 in px_proxy_factory_get_proxies () at /lib/x86_64-linux-gnu/libproxy.so.1
#10 0x00007fff56e53827 in () at /usr/lib/x86_64-linux-gnu/gio/modules/libgiolibproxy.so
#11 0x00007ffff7367194 in () at /lib/x86_64-linux-gnu/libgio-2.0.so.0
#12 0x00007ffff76156b4 in () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x00007ffff7612a51 in () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007ffff7cdcac3 in () at /lib/x86_64-linux-gnu/libc.so.6
#15 0x00007ffff7d6ea40 in () at /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff7cde9fc in pthread_kill () at /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff7c8a476 in raise () at /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff7c707f3 in abort () at /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff476bb9e in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff477720c in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff47761e9 in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff4776959 in __gxx_personality_v0 () at /lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fff79076fe9 in __libunwind_Unwind_Resume () at /lib/x86_64-linux-gnu/libunwind.so.8
#8 0x00007ffff3dd386d in () at /lib/x86_64-linux-gnu/libproxy.so.1
#9 0x00007ffff3ddc827 in px_proxy_factory_get_proxies () at /lib/x86_64-linux-gnu/libproxy.so.1
#10 0x00007fff56e53827 in () at /usr/lib/x86_64-linux-gnu/gio/modules/libgiolibproxy.so
#11 0x00007ffff7367194 in () at /lib/x86_64-linux-gnu/libgio-2.0.so.0
#12 0x00007ffff76156b4 in () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x00007ffff7612a51 in () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007ffff7cdcac3 in () at /lib/x86_64-linux-gnu/libc.so.6
#15 0x00007ffff7d6ea40 in () at /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff7cde9fc in pthread_kill () at /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff7c8a476 in raise () at /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff7c707f3 in abort () at /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff476bb9e in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff477720c in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff47761e9 in () at /lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff4776959 in __gxx_personality_v0 () at /lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fff79076fe9 in __libunwind_Unwind_Resume () at /lib/x86_64-linux-gnu/libunwind.so.8
#8 0x00007ffff3dd386d in () at /lib/x86_64-linux-gnu/libproxy.so.1
#9 0x00007ffff3ddc827 in px_proxy_factory_get_proxies () at /lib/x86_64-linux-gnu/libproxy.so.1
#10 0x00007fff56e53827 in () at /usr/lib/x86_64-linux-gnu/gio/modules/libgiolibproxy.so
#11 0x00007ffff7367194 in () at /lib/x86_64-linux-gnu/libgio-2.0.so.0
#12 0x00007ffff76156b4 in () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x00007ffff7612a51 in () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007ffff7cdcac3 in () at /lib/x86_64-linux-gnu/libc.so.6
#15 0x00007ffff7d6ea40 in () at /lib/x86_64-linux-gnu/libc.so.6

15

Any update?

17

From the stack info, it crashed in the following module, libgio-2.0.so.0. There may be a problem with your environment. What environment are you running deepstream, your host, in a docker or virtual machine?
Could you attach your code or pipeline to us ?

18

How do I fix it?

19

Docker on my Local machine!

20

I can share my pipeline with you.

deepstream_6_4
deepstream_6_49145×352 129 KB

I am trying to achieve deepstream-parallel-inference using python, which I have already achieved on deepstream6.1.1, but I am unable to get a single PGIE to run on deepstream-6.4

21

@yuweiw, I am looking for a solution.

22

It’s a system API, not a DeepStream API libgio-2-0. And we cannot reproduce this issue.
So you can provide us with your project that can reproduce this problem, or narrow down the problem step by step with the gst-launch-1.0 command yourself.