Segmentation fault when loading deepstream dependencies

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU)
Jetson orin-nx
• DeepStream Version
8.2
• JetPack Version (valid for Jetson only)
5.1.1
• TensorRT Version
8.5.2

• Issue Type( questions, new requirements, bugs)
Bugs

I have a deepstream python application packaged with PyInstaller, that is running without issues on 90% of Jetson Orin-NX in our fleet. The remaining 10% fail with “Segmentation fault (core dumped)”. All jetsons are flashed using the same script, and therefore should be identical.
Running with GST_DEBUG=7 suggests that it fails when attempting to load libnvdsgst_inferaudio.so

ry0> file /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_inferaudio.so not yet in registry
0:00:00.991147154 364077      0x9ef9f50 DEBUG     GST_PLUGIN_LOADING gstplugin.c:760:_priv_gst_plugin_load_file_for_registry: attempt to load plugin "/usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_inferaudio.so"
Segmentation fault (core dumped)

Running with LD_DEBUG=all the final output before the crash is:

    363596:     object=/usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so [0]
    363596:      scope 18: /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_inferaudio.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libglib-2.0.so.0 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgobject-2.0.so.0 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstreamer-1.0.so.0 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstaudio-1.0.so.0 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_meta.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_meta.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_inferbase.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_audiotransform.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_tensor.so /lib/aarch64-linux-gnu/libdl.so.2 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libstdc++.so.6 /lib/aarch64-linux-gnu/libpthread.so.0 /lib/aarch64-linux-gnu/librt.so.1 /lib/aarch64-linux-gnu/libc.so.6 /lib/ld-linux-aarch64.so.1 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpcre.so.3 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libffi.so.7 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgmodule-2.0.so.0 /lib/aarch64-linux-gnu/libm.so.6 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgsttag-1.0.so.0 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstbase-1.0.so.0 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/liborc-0.4.so.0 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_infer.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcufft.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgcc_s.so.1 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libz.so.1 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvparsers.so.8 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvonnxparser.so.8 /lib/aarch64-linux-gnu/libnvinfer.so.8 /lib/aarch64-linux-gnu/libnvinfer_plugin.so.8 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_inferutils.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_inferlogger.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdla_compiler.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcudla.so.1 /usr/local/cuda/targets/aarch64-linux/lib/libcublas.so.11 /usr/local/cuda/targets/aarch64-linux/lib/libcublasLt.so.11 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcudnn.so.8 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcrypto.so.1.1 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvos.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdla_runtime.so /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1 /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_host1x.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_mem.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvsocsys.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_gpu.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_sync.so /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_chip.so /usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so
    363596:
    363596:
    363596:     calling init: /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcudnn.so.8
    363596:
    363596:
    363596:     calling init: /usr/local/cuda/targets/aarch64-linux/lib/libcublasLt.so.11
    363596:
    363596:
    363596:     calling init: /usr/local/cuda/targets/aarch64-linux/lib/libcublas.so.11
    363596:
    363596:
    363596:     calling init: /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_inferlogger.so
    363596:
    363596:
    363596:     calling init: /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvparsers.so.8
    363596:
    363596:
    363596:     calling init: /lib/aarch64-linux-gnu/libnvinfer_plugin.so.8
    363596:

Running the app with gdb gives the following backtrace:

0x0000ffff940a25bc in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.8
(gdb) bt
#0  0x0000ffff940a25bc in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.8
#1  0x0000ffff940a2ee8 in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.8
#2  0x0000ffffb42f0eec in ?? () from /lib/aarch64-linux-gnu/libnvinfer_plugin.so.8
#3  0x0000fffff7fda8b4 in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0xfffffffff308,
    env=env@entry=0x22413b0) at dl-init.c:72
#4  0x0000fffff7fda9b4 in call_init (env=0x22413b0, argv=0xfffffffff308, argc=1, l=<optimized out>) at dl-init.c:30
#5  _dl_init (main_map=0x32cf400, argc=1, argv=0xfffffffff308, env=0x22413b0) at dl-init.c:119
#6  0x0000fffff7ed920c in __GI__dl_catch_exception (exception=0x0, operate=0xfffff7fddcc8 <call_dl_init>,
    args=0xffffffffb490) at dl-error-skeleton.c:182
#7  0x0000fffff7fdea14 in dl_open_worker (a=a@entry=0xffffffffb6e0) at dl-open.c:758
#8  0x0000fffff7ed91ac in __GI__dl_catch_exception (exception=0xffffffffb6c8, operate=0xfffff7fde540 <dl_open_worker>,
    args=0xffffffffb6e0) at dl-error-skeleton.c:208
#9  0x0000fffff7fde1a4 in _dl_open (
    file=0x2376e40 "/usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_inferaudio.so", mode=-2147483646,
    caller_dlopen=0xffffe99a493c <g_module_open+748>, nsid=-2, argc=1, argv=0xfffffffff308, env=0x22413b0) at dl-open.c:837
#10 0x0000fffff7f9e09c in dlopen_doit (a=a@entry=0xffffffffb9a8) at dlopen.c:66
#11 0x0000fffff7ed91ac in __GI__dl_catch_exception (exception=exception@entry=0xffffffffb920,
    operate=0xfffff7f9e038 <dlopen_doit>, args=0xffffffffb9a8) at dl-error-skeleton.c:208
#12 0x0000fffff7ed9278 in __GI__dl_catch_error (objname=0x425890, errstring=0x425898, mallocedp=0x425888,
    operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:227
#13 0x0000fffff7f9e838 in _dlerror_run (operate=operate@entry=0xfffff7f9e038 <dlopen_doit>, args=args@entry=0xffffffffb9a8)
    at dlerror.c:170
#14 0x0000fffff7f9e140 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#15 0x0000ffffe99a493c in g_module_open ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgmodule-2.0.so.0
#16 0x0000ffffe9bc1728 in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstreamer-1.0.so.0
#17 0x0000ffffe9bd566c in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstreamer-1.0.so.0
#18 0x0000ffffe9bd7348 in gst_update_registry ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstreamer-1.0.so.0
#19 0x0000ffffe9b72384 in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstreamer-1.0.so.0
#20 0x0000ffffd9a7d9bc in g_option_context_parse ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libglib-2.0.so.0
#21 0x0000ffffe9b73040 in gst_init_check ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstreamer-1.0.so.0
#22 0x0000ffffe9b73094 in gst_init () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstreamer-1.0.so.0
#23 0x0000fffff7fb8ff8 in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libffi.so.7
#24 0x0000fffff7fb87cc in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libffi.so.7
#25 0x0000ffffe9726984 in ?? ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/gi/_gi.cpython-38-aarch64-linux-gnu.so
#26 0x0000ffffe971d49c in ?? ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/gi/_gi.cpython-38-aarch64-linux-gnu.so
#27 0x0000fffff7b0d350 in _PyObject_MakeTpCall ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#28 0x0000fffff78ed72c in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#29 0x0000fffff78f244c in _PyEval_EvalFrameDefault ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#30 0x0000fffff78f7724 in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#31 0x0000fffff78ed6b8 in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#32 0x0000fffff78f40d8 in _PyEval_EvalFrameDefault ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#33 0x0000fffff78f7724 in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#34 0x0000fffff78ed6b8 in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#35 0x0000fffff78ee630 in _PyEval_EvalFrameDefault ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#36 0x0000fffff7a351f4 in _PyEval_EvalCodeWithName ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
--Type <RET> for more, q to quit, c to continue without paging--
#37 0x0000fffff7b0c908 in _PyFunction_Vectorcall ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#38 0x0000fffff7b0e798 in _PyObject_FastCallDict ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#39 0x0000fffff7b0e93c in _PyObject_Call_Prepend ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#40 0x0000fffff7aa9d0c in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#41 0x0000fffff7aaf2ac in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#42 0x0000fffff7b0d350 in _PyObject_MakeTpCall ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#43 0x0000fffff78ed72c in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#44 0x0000fffff78eecb0 in _PyEval_EvalFrameDefault ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#45 0x0000fffff7a351f4 in _PyEval_EvalCodeWithName ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#46 0x0000fffff7b0c908 in _PyFunction_Vectorcall ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#47 0x0000fffff78ed6b8 in ?? () from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#48 0x0000fffff78eecb0 in _PyEval_EvalFrameDefault ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#49 0x0000fffff7a351f4 in _PyEval_EvalCodeWithName ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#50 0x0000fffff7a35610 in PyEval_EvalCodeEx ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#51 0x0000fffff7a359ec in PyEval_EvalCode ()
   from /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpython3.8.so.1.0
#52 0x0000000000402884 in ?? ()
#53 0x0000000000402f78 in ?? ()

#54 0x0000fffff7defe10 in __libc_start_main (main=0x401a80, argc=1, argv=0xfffffffff308, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#55 0x0000000000401ac4 in ?? ()
#56 0x0000fffffffff5e7 in ?? ()

I’ve run out of ideas of what could be wrong, so any ideas or tips would be appreciated.

3

Do you mean DS-6.2?

This seems to be a deployment issue, probably caused by differences in environment

Try the following command, this problem may be caused by the lack of some shared libraries.

ldd /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferaudio.so
4

Sorry yes DS-6.2

The deepstream .so files are packed as part of the pyinstaller package as they are not installed on the jetson’s in the fleet. So they are located in /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_inferlogger.so

	linux-vdso.so.1 (0x0000ffff9e8f2000)
	libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff9e6ae000)
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff9e53b000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff9e8c2000)
	libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff9e490000)
	libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff9e46c000)

I’ve compared all files in the filesystem on a working device, and a non-working one and they all have the same libraries. I have however not checked the checksum of all files, but libnvds_inferlogger.so and /lib/aarch64-linux-gnu/libnvinfer_plugin.so.8 have the same checksum.

5

Is libnvdsgst_inferaudio.so located in /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/?

If you execute ldd libnvdsgst_inferaudio.so, can you see some shared libraries are missing?

Try rm -rf ~/.cache/gstreamer-1.0 to clean cache

6

Yes, libnvdsgst_inferaudio.so is located in /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/

ldd /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_inferaudio.so outputs:

	linux-vdso.so.1 (0x0000ffff90cde000)
	libglib-2.0.so.0 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libglib-2.0.so.0 (0x0000ffff90ab3000)
	libgobject-2.0.so.0 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgobject-2.0.so.0 (0x0000ffff90a41000)
	libgstreamer-1.0.so.0 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstreamer-1.0.so.0 (0x0000ffff908d3000)
	libgstaudio-1.0.so.0 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstaudio-1.0.so.0 (0x0000ffff9084f000)
	libnvds_meta.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_meta.so (0x0000ffff90838000)
	libnvdsgst_meta.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_meta.so (0x0000ffff90822000)
	libnvdsgst_inferbase.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_inferbase.so (0x0000ffff9075b000)
	libnvds_audiotransform.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_audiotransform.so (0x0000ffff90684000)
	libnvdsgst_tensor.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdsgst_tensor.so (0x0000ffff905d2000)
	libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff905a3000)
	libstdc++.so.6 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libstdc++.so.6 (0x0000ffff903be000)
	libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff9038d000)
	librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000ffff90375000)
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff90202000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff90cae000)
	libpcre.so.3 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libpcre.so.3 (0x0000ffff90190000)
	libffi.so.7 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libffi.so.7 (0x0000ffff90177000)
	libgmodule-2.0.so.0 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgmodule-2.0.so.0 (0x0000ffff90163000)
	libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff900b8000)
	libgsttag-1.0.so.0 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgsttag-1.0.so.0 (0x0000ffff9006d000)
	libgstbase-1.0.so.0 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgstbase-1.0.so.0 (0x0000ffff8ffe8000)
	liborc-0.4.so.0 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/liborc-0.4.so.0 (0x0000ffff8ff59000)
	libnvds_infer.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_infer.so (0x0000ffff8fe59000)
	libcufft.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcufft.so (0x0000ffff84ef1000)
	libgcc_s.so.1 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libgcc_s.so.1 (0x0000ffff84ecd000)
	libz.so.1 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libz.so.1 (0x0000ffff84ea3000)
	libnvparsers.so.8 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvparsers.so.8 (0x0000ffff84b32000)
	libnvonnxparser.so.8 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvonnxparser.so.8 (0x0000ffff84865000)
	libnvinfer.so.8 => /lib/aarch64-linux-gnu/libnvinfer.so.8 (0x0000ffff7770c000)
	libnvinfer_plugin.so.8 => /lib/aarch64-linux-gnu/libnvinfer_plugin.so.8 (0x0000ffff762e9000)
	libnvds_inferutils.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_inferutils.so (0x0000ffff762c0000)
	libnvds_inferlogger.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvds_inferlogger.so (0x0000ffff762aa000)
	libnvdla_compiler.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdla_compiler.so (0x0000ffff75b6f000)
	libcudla.so.1 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcudla.so.1 (0x0000ffff75b32000)
	libcublas.so.11 => /usr/local/cuda/targets/aarch64-linux/lib/libcublas.so.11 (0x0000ffff6ba59000)
	libcublasLt.so.11 => /usr/local/cuda/targets/aarch64-linux/lib/libcublasLt.so.11 (0x0000ffff544b8000)
	libcudnn.so.8 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcudnn.so.8 (0x0000ffff54481000)
	libcrypto.so.1.1 => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libcrypto.so.1.1 (0x0000ffff541ec000)
	libnvos.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvos.so (0x0000ffff541cc000)
	libnvdla_runtime.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvdla_runtime.so (0x0000ffff53b74000)
	libcuda.so.1 => /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1 (0x0000ffff5251d000)
	libnvrm_host1x.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_host1x.so (0x0000ffff524fa000)
	libnvrm_mem.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_mem.so (0x0000ffff524e2000)
	libnvsocsys.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvsocsys.so (0x0000ffff524ce000)
	libnvrm_gpu.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_gpu.so (0x0000ffff52461000)
	libnvrm_sync.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_sync.so (0x0000ffff5244a000)
	libnvrm_chip.so => /usr/local/snakebrain/debugging/usr/local/snakebrain/deps/libnvrm_chip.so (0x0000ffff52434000)
	libnvsciipc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so (0x0000ffff5240f000)

Deleting the cache with rm -rf ~/.cache/gstreamer-1.0 yields the same results.

7

For some context: All devices are currently able to run a gstreamer pipeline and tensorrt model separately, which I now try to integrate into a single deepstream pipeline.

8

Very strange, I don’t know what happened, but why are the checksums of libnvds_inferlogger.so and /lib/aarch64-linux-gnu/libnvinfer_plugin.so.8 the same?

libnvinfer_plugin.so.8 should just be a soft link to libnvinfer_plugin.so.8.x.x.

Can you share the packaged application ?

9

Sorry I meant deps/libnvinfer_plugin.so.8 and /lib/aarch64-linux-gnu/libnvinfer_plugin.so.8 have the same checksum.

The issue occours even for this simple example. It crashes during Gst.init(None)

simple.py

import sys
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GLib

print("calling Gst.init(None)")
Gst.init(None)

print("creating pipeline")
# Create the pipeline
pipeline = Gst.Pipeline()

It’s packaged using pyinstaller, where we include the deepstream libraries
pyinstaller --add-binary "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/*.so:." --distpath dist --name simple simple.py

In the jetson’s with the issue it outputs:
$ GST_DEBUG=7 ./simple

...
(simple:224695): GStreamer-WARNING **: 13:30:02.599: Failed to load plugin '/home/nightingale/dist/simple/_internal/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
0:00:00.216183809 224695      0x3ca1b50 LOG             GST_REGISTRY gstregistry.c:1295:gst_registry_scan_path_level:<registry0> file /home/nightingale/dist/simple/_internal/libnvscf.so looks like a possible module
0:00:00.216195138 224695      0x3ca1b50 DEBUG           GST_REGISTRY gstregistry.c:1356:gst_registry_scan_path_level:<registry0> file /home/nightingale/dist/simple/_internal/libnvscf.so not yet in registry
0:00:00.216201602 224695      0x3ca1b50 DEBUG     GST_PLUGIN_LOADING gstplugin.c:760:_priv_gst_plugin_load_file_for_registry: attempt to load plugin "/home/nightingale/dist/simple/_internal/libnvscf.so"
0:00:00.219408936 224695      0x3ca1b50 DEBUG     GST_PLUGIN_LOADING gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: Could not find symbol 'gst_plugin_nvscf_get_desc', falling back to gst_plugin_desc
0:00:00.219442090 224695      0x3ca1b50 DEBUG     GST_PLUGIN_LOADING gstplugin.c:819:_priv_gst_plugin_load_file_for_registry: Could not find plugin entry point in "/home/nightingale/dist/simple/_internal/libnvscf.so"
0:00:00.220605690 224695      0x3ca1b50 LOG             GST_REGISTRY gstregistry.c:1295:gst_registry_scan_path_level:<registry0> file /home/nightingale/dist/simple/_internal/libnvdla_runtime.so looks like a possible module
0:00:00.220636380 224695      0x3ca1b50 DEBUG           GST_REGISTRY gstregistry.c:1356:gst_registry_scan_path_level:<registry0> file /home/nightingale/dist/simple/_internal/libnvdla_runtime.so not yet in registry
0:00:00.220652604 224695      0x3ca1b50 DEBUG     GST_PLUGIN_LOADING gstplugin.c:760:_priv_gst_plugin_load_file_for_registry: attempt to load plugin "/home/nightingale/dist/simple/_internal/libnvdla_runtime.so"
0:00:00.220707903 224695      0x3ca1b50 DEBUG     GST_PLUGIN_LOADING gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: Could not find symbol 'gst_plugin_nvdla_runtime_get_desc', falling back to gst_plugin_desc
0:00:00.220724159 224695      0x3ca1b50 DEBUG     GST_PLUGIN_LOADING gstplugin.c:819:_priv_gst_plugin_load_file_for_registry: Could not find plugin entry point in "/home/nightingale/dist/simple/_internal/libnvdla_runtime.so"
0:00:00.220789890 224695      0x3ca1b50 LOG             GST_REGISTRY gstregistry.c:1295:gst_registry_scan_path_level:<registry0> file /home/nightingale/dist/simple/_internal/libnvdsgst_inferaudio.so looks like a possible module
0:00:00.220802851 224695      0x3ca1b50 DEBUG           GST_REGISTRY gstregistry.c:1356:gst_registry_scan_path_level:<registry0> file /home/nightingale/dist/simple/_internal/libnvdsgst_inferaudio.so not yet in registry
0:00:00.220814979 224695      0x3ca1b50 DEBUG     GST_PLUGIN_LOADING gstplugin.c:760:_priv_gst_plugin_load_file_for_registry: attempt to load plugin "/home/nightingale/dist/simple/_internal/libnvdsgst_inferaudio.so"
Segmentation fault (core dumped)

The entire output can be seen here:
output.log (99.2 KB)

The package is built on an orin-nx that is running the same jetpack version as the devices in the fleet.

10

1.If you install deepstreamsdk and python binding on the device where the problem occurs, and then run the deepstream-python sample app, will it run correctly?

2.Are there any differences when comparing logs on different devices?

sudo apt-get install strace
rm -rf /home/*/.cache/gstreamer-1.0/
rm -rf /root/.cache/gstreamer-1.0/
strace ./simple > log.log 2>&1
11
  1. If I install the deepstreamsdk and python bindings I’m able to run my application correctly.

If I package it using pyinstaller on the same device, and try to run it, I get the same error, so it seems like something goes wrong in the packaging.

  1. Up until the seg fault I’m not able to see any obvious differences between the two logs.
12

I finally managed to find a solution:

  • I make sure pyinstaller doesn’t include any gstreamer plugins in the binary by setting
        hooksconfig={
     "gstreamer": {
         "include_plugins": []
     }
 },
  • I no longer tell pyinstaller to add the binaries. So removing: binaries=[('/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/*.so', '.')]
  • I manually copy over only the files needed by the jetson at runtime, which for my application was:
   cp /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/*.so dis
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvds_meta.so
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvds_inferlogger.so
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvdsgst_helper.so
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvds_logger.so
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvds_infer.so
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvdsgst_inferbase.so
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvds_inferutils.so
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvdsgst_meta.so
   cp /opt/nvidia/deepstream/deepstream-6.2/lib/libnvdsgst_customhelper.so
   cp /usr/lib/aarch64-linux-gnu/libyaml-cpp.so.0.6

If i copy over all files in /opt/nvidia/deepstream/deepstream-6.2/lib I get the segmentation fault. So there must be some file in there that is causing it, but it turned out it is not needed to run the application.
Thank you for looking into it.

1 Like
13

Thanks for you sharing.

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.