SCF Execution ERROR after hours of camera capture

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

We are using Jetson Orin Nano with Jetpack 5.1.2-b104. The Nano board is inserted into a Leopard Imaging carrier board, and then connected to 3 of Leopard Imaging DS90UB954 FPD-Link III Deserializer, then to 5 or 6 cameras with DS90UB953 Serializer and Cameras.

We adapt the camera codes from the Argus sample code. After we get the image, we resize it, pad it and run inference with CUDA, and send the result out.

The system does work for some time, but after 30mins~10hours of capturing, the code with crash. GDB print a backtrace into tegra libraries.

How can we trouble shoot the problem? What information should we provide? Here we attached some backtrace print.

BT1:
Thread 16 “SCF Execution” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffb4165900 (LWP 28513)]
tcache_get (tc_idx=) at malloc.c:2937
2937 malloc.c: No such file or directory.
(gdb) bt
#0 tcache_get (tc_idx=) at malloc.c:2937
#1 __GI___libc_malloc (bytes=88) at malloc.c:3051
#2 0x0000ffffea6c3b4c in operator new(unsigned long) ()
from /lib/aarch64-linux-gnu/libstdc++.so.6
#3 0x0000ffffe94b8110 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#4 0x0000ffffe94b8b3c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#5 0x0000ffffe9499ea4 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#6 0x0000ffffe945ebb4 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#7 0x0000ffffe946144c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000ffffe94442bc in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000ffffe9413190 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000ffffe941331c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#11 0x0000ffffe9411d48 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#12 0x0000ffffea3de114 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#13 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0) at pthread_create.c:477
#14 0x0000ffffea55e49c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

BT2:
Thread 11 “SCF Execution” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffd3588900 (LWP 640207)]
0x0000ffffe9413538 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
(gdb) bt
#0 0x0000ffffe9413538 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#1 0x0000ffffe9413228 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#2 0x0000ffffe941331c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#3 0x0000ffffe9411d48 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#4 0x0000ffffea3de114 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#5 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0)
at pthread_create.c:477
#6 0x0000ffffea55e49c in thread_start ()
at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

BT3:
Thread 13 “SCF Execution” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffd2586900 (LWP 640517)]
__memcpy_generic () at …/sysdeps/aarch64/multiarch/…/memcpy.S:254
254 …/sysdeps/aarch64/multiarch/…/memcpy.S: No such file or directory.
(gdb) bt
#0 __memcpy_generic () at …/sysdeps/aarch64/multiarch/…/memcpy.S:254
#1 0x0000ffffea75cd28 in std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_replace(unsigned long, unsigned long, char const*, unsigned long) () from /lib/aarch64-linux-gnu/libstdc++.so.6
#2 0x0000ffffeab80af8 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#3 0x0000ffffeabad6f8 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#4 0x0000ffffeabaf1e0 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#5 0x0000ffffeab93444 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#6 0x0000ffffe94da188 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#7 0x0000ffffe94da63c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000ffffe94dab40 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000ffffe94020c4 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000ffffe942c7e0 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#11 0x0000ffffe94442bc in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#12 0x0000ffffe9413190 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#13 0x0000ffffe941331c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#14 0x0000ffffe9411d48 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#15 0x0000ffffea3de114 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#16 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0)
at pthread_create.c:477
#17 0x0000ffffea55e49c in thread_start ()
at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

BT4:
Thread 8 “SCF Execution” received signal SIGABRT, Aborted.
[Switching to Thread 0xffffd4d8b900 (LWP 640726)]
__GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:50
50 …/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000ffffea4adaac in __GI_abort () at abort.c:79
#2 0x0000ffffea4ba490 in __assert_fail_base (
fmt=0xffffea5b5738 “%s%s%s:%u: %s%sAssertion `%s’ failed.\n%n”,
assertion=assertion@entry=0xffffeac62f08 “mutex->__data.__owner == 0”,
file=file@entry=0xffffeac63088 “pthread_mutex_lock.c”, line=line@entry=81,
function=function@entry=0xffffeac62f88 <PRETTY_FUNCTION.11555> “__pthread_mutex_lock”) at assert.c:92
#3 0x0000ffffea4ba4f4 in __GI___assert_fail (
assertion=assertion@entry=0xffffeac62f08 “mutex->__data.__owner == 0”,
file=file@entry=0xffffeac63088 “pthread_mutex_lock.c”, line=line@entry=81,
function=function@entry=0xffffeac62f88 <PRETTY_FUNCTION.11555> “__pthread_mutex_lock”) at assert.c:101
#4 0x0000ffffeac56e18 in __GI___pthread_mutex_lock (mutex=0xfffee0003460)
at pthread_mutex_lock.c:81
#5 0x0000ffffea3ddb48 in NvOsMutexLock ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#6 0x0000ffffe9363204 in nvcamerautils::Mutex::lock(char const*, unsigned int) const () from /usr/lib/aarch64-linux-gnu/tegra/libnvcamerautils.so
#7 0x0000ffffe947dd18 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000ffffe93f7e64 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000ffffe93f78c4 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000ffffe9402a38 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#11 0x0000ffffe942c644 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#12 0x0000ffffe94442ec in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#13 0x0000ffffe9413190 in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#14 0x0000ffffe941331c in ?? ()
from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#15 0x0000ffffe9411d48 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#16 0x0000ffffea3de114 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#17 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0) at pthread_create.c:477
#18 0x0000ffffea55e49c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

BT5:
#0 __memcpy_generic () at …/sysdeps/aarch64/multiarch/…/memcpy.S:252
#1 0x0000ffffea75cd28 in std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_replace(unsigned long, unsigned long, char const*, unsigned long) () from /lib/aarch64-linux-gnu/libstdc++.so.6
#2 0x0000ffffeab80af8 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#3 0x0000ffffeabad6f8 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#4 0x0000ffffeabaf1e0 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#5 0x0000ffffeab93444 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
#6 0x0000ffffe94da188 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#7 0x0000ffffe94da63c in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000ffffe94dab40 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000ffffe94020c4 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000ffffe942c7e0 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#11 0x0000ffffe94442bc in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#12 0x0000ffffe9413190 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#13 0x0000ffffe941331c in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#14 0x0000ffffe9411d48 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#15 0x0000ffffea3de114 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#16 0x0000ffffeac54624 in start_thread (arg=0xffffea3de0e0) at pthread_create.c:477
#17 0x0000ffffea55e62c in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb)

3

hello Deqing_S,

please moving to the latest Jetpack-5 public release version.
FYI,
we’ve some stability updates based-on Jetpack-5.1.3/ l4t-r35.5.0
please refer to Topic 293597 for camera firmware update to improve CSI stability

5

Thank you for your help.

I tried to get the 7z file from AGX Orin camera issues - #23 by JerryChang
It seems it only contains a “camera-rtcpu-t234-rce.img”

We are using a jetson Orin nano with fresh install of 5.1.3 on NVMe harddrive, the only folder containing “bootloader” is

/usr/src/linux-headers-5.10.192-tegra-ubuntu20.04_aarch64/kernel-5.10/include/config/tegra/bootloader

And there is only a debug folder and a debug.h

I also tried to search for all img files and I did not find any similar img file.

How should we use the img file and update the firmware? We did not find flash.sh either.

6

hello Deqing_S,

since it’s Orin Nano, it cannot use flash script to perform image flashing.
please see-also Topic 260583 for the steps to replace rce-fw with initrd.

7

Thank you I would try. Would the camera-rtcpu-t234-rce.img for AGX in the other post also work for orin nano?

8

hello Deqing_S,

Orin series (AGX Orin/ Orin Nano/ Orin NX) they’re using the same RCE firmware, camera-rtcpu-t234-rce.img.

since it’s rce-fw update for JP-5.1.3 / l4t-r35.5.0,
please double check you’re based-on the same release version before updating rce-fw.
for instance,
$ cat /etc/nv_tegra_release
# R35 (release), REVISION: 5.0, GCID: 35550185, BOARD: t186ref, EABI: aarch64, DATE: Tue Feb 20 04:46:31 UTC 2024

9

Hi Jerry,

It seems my nano does not come with anything called *flash.sh or kernel_flash
How do I get these scripts?

Also another question. After updating to 5.1.3. It seems this is no more memory crash. But we still get error after hours. Like

SCF: Error InvalidState:  Corr Error Received for sensor 4 .. Continuing!
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 654)

How can this be debugged?

10

hello Deqing_S,

those script files coming from JetPack-5.1.3, which by default installed to your host machine locally.
anyways, thanks for moving to JP-5.1.3, let’s have a new thread for following-up your new issues.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.