SCF Execution ERROR after hours of camera capture

We are using Jetson Orin Nano with Jetpack 5.1.2-b104. The Nano board is inserted into a Leopard Imaging carrier board, and then connected to 3 of Leopard Imaging DS90UB954 FPD-Link III Deserializer, then to 5 or 6 cameras with DS90UB953 Serializer and Cameras.

We adapt the camera codes from the Argus sample code. After we get the image, we resize it, pad it and run inference with CUDA, and send the result out.

The system does work for some time, but after 30mins~10hours of capturing, the code with crash. GDB print a backtrace into tegra libraries.

How can we trouble shoot the problem? What information should we provide? Here we attached some backtrace print.

BT1:
Thread 16 “SCF Execution” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffb4165900 (LWP 28513)]
tcache_get (tc_idx=) at malloc.c:2937
2937 malloc.c: No such file or directory.
(gdb) bt
#0 tcache_get (tc_idx=) at malloc.c:2937
#1 __GI___libc_malloc (bytes=88) at malloc.c:3051
#2 0x0000ffffea6c3b4c in operator new(unsigned long) ()
from /lib/aarch64-linux-gnu/libstdc++.so.6
#3 0x0000ffffe94b8110 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#4 0x0000ffffe94b8b3c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#5 0x0000ffffe9499ea4 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#6 0x0000ffffe945ebb4 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#7 0x0000ffffe946144c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000ffffe94442bc in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000ffffe9413190 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000ffffe941331c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#11 0x0000ffffe9411d48 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#12 0x0000ffffea3de114 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#13 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0) at pthread_create.c:477
#14 0x0000ffffea55e49c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

BT2:
Thread 11 “SCF Execution” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffd3588900 (LWP 640207)]
0x0000ffffe9413538 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
(gdb) bt
#0 0x0000ffffe9413538 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#1 0x0000ffffe9413228 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#2 0x0000ffffe941331c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#3 0x0000ffffe9411d48 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#4 0x0000ffffea3de114 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#5 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0)
at pthread_create.c:477
#6 0x0000ffffea55e49c in thread_start ()
at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

BT3:
Thread 13 “SCF Execution” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffd2586900 (LWP 640517)]
__memcpy_generic () at …/sysdeps/aarch64/multiarch/…/memcpy.S:254
254 …/sysdeps/aarch64/multiarch/…/memcpy.S: No such file or directory.
(gdb) bt
#0 __memcpy_generic () at …/sysdeps/aarch64/multiarch/…/memcpy.S:254
#1 0x0000ffffea75cd28 in std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_replace(unsigned long, unsigned long, char const*, unsigned long) () from /lib/aarch64-linux-gnu/libstdc++.so.6
#2 0x0000ffffeab80af8 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#3 0x0000ffffeabad6f8 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#4 0x0000ffffeabaf1e0 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#5 0x0000ffffeab93444 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#6 0x0000ffffe94da188 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#7 0x0000ffffe94da63c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000ffffe94dab40 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000ffffe94020c4 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000ffffe942c7e0 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#11 0x0000ffffe94442bc in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#12 0x0000ffffe9413190 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#13 0x0000ffffe941331c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#14 0x0000ffffe9411d48 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#15 0x0000ffffea3de114 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#16 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0)
at pthread_create.c:477
#17 0x0000ffffea55e49c in thread_start ()
at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

BT4:
Thread 8 “SCF Execution” received signal SIGABRT, Aborted.
[Switching to Thread 0xffffd4d8b900 (LWP 640726)]
__GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:50
50 …/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000ffffea4adaac in __GI_abort () at abort.c:79
#2 0x0000ffffea4ba490 in __assert_fail_base (
fmt=0xffffea5b5738 “%s%s%s:%u: %s%sAssertion `%s’ failed.\n%n”,
assertion=assertion@entry=0xffffeac62f08 “mutex->__data.__owner == 0”,
file=file@entry=0xffffeac63088 “pthread_mutex_lock.c”, line=line@entry=81,
function=function@entry=0xffffeac62f88 <PRETTY_FUNCTION.11555> “__pthread_mutex_lock”) at assert.c:92
#3 0x0000ffffea4ba4f4 in __GI___assert_fail (
assertion=assertion@entry=0xffffeac62f08 “mutex->__data.__owner == 0”,
file=file@entry=0xffffeac63088 “pthread_mutex_lock.c”, line=line@entry=81,
function=function@entry=0xffffeac62f88 <PRETTY_FUNCTION.11555> “__pthread_mutex_lock”) at assert.c:101
#4 0x0000ffffeac56e18 in __GI___pthread_mutex_lock (mutex=0xfffee0003460)
at pthread_mutex_lock.c:81
#5 0x0000ffffea3ddb48 in NvOsMutexLock ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#6 0x0000ffffe9363204 in nvcamerautils::Mutex::lock(char const*, unsigned int) const () from /usr/lib/aarch64-linux-gnu/tegra/libnvcamerautils.so
#7 0x0000ffffe947dd18 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000ffffe93f7e64 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000ffffe93f78c4 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000ffffe9402a38 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#11 0x0000ffffe942c644 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#12 0x0000ffffe94442ec in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#13 0x0000ffffe9413190 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#14 0x0000ffffe941331c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#15 0x0000ffffe9411d48 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#16 0x0000ffffea3de114 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#17 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0) at pthread_create.c:477
#18 0x0000ffffea55e49c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

BT5:
#0 __memcpy_generic () at …/sysdeps/aarch64/multiarch/…/memcpy.S:252
#1 0x0000ffffea75cd28 in std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_replace(unsigned long, unsigned long, char const*, unsigned long) () from /lib/aarch64-linux-gnu/libstdc++.so.6
#2 0x0000ffffeab80af8 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#3 0x0000ffffeabad6f8 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#4 0x0000ffffeabaf1e0 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#5 0x0000ffffeab93444 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#6 0x0000ffffe94da188 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#7 0x0000ffffe94da63c in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000ffffe94dab40 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000ffffe94020c4 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000ffffe942c7e0 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#11 0x0000ffffe94442bc in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#12 0x0000ffffe9413190 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#13 0x0000ffffe941331c in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#14 0x0000ffffe9411d48 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#15 0x0000ffffea3de114 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#16 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0) at pthread_create.c:477
#17 0x0000ffffea55e62c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

hello Deqing_S,

please moving to the latest Jetpack-5 public release version.
FYI,
we’ve some stability updates based-on Jetpack-5.1.3/ l4t-r35.5.0
please refer to Topic 293597 for camera firmware update to improve CSI stability