Hi
We are running our GStreamer based application on Jetson Nano.
System information:
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION=“Ubuntu 18.04.5 LTS”
$ cat /etc/nv_tegra_release
# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t210ref, EABI: aarch64, DATE: Fri Oct 16 19:44:43 UTC 2020
During long running we saw that the nvargus-daemon is crashing, we are chasing this for a while and now we were able to catch it with logs and recording.
So the system is started almost a 24 hours before:
$ uptime
15:06:24 up 1 day, 1:09, 9 users, load average: 2,00, 2,35, 2,50
Based on the logs the crash is generated around 14:46:28. Based on the recording the issues started around 14:46:16 with large purple stripes on the video and some noisy pixels:
After this the screen gone black.
The backtarce in the coredump is not telling me a lot:
Using host libthread_db library “/lib/aarch64-linux-gnu/libthread_db.so.1”.
Core was generated by `/usr/sbin/nvargus-daemon’.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0001000003000300 in ?? ()
[Current thread is 1 (Thread 0x7f8e2a01d0 (LWP 26451))]
(gdb) bt
#0 0x0001000003000300 in ?? ()
#1 0x0000007f9459e150 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#2 0x0000007f9459f928 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#3 0x0000007f944cf538 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#4 0x0000007f9451f85c in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#5 0x0000007f94530ab8 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#6 0x0000007f94514478 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#7 0x0000007f944e0c4c in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#8 0x0000007f944e0e84 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#9 0x0000007f944dfa50 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
#10 0x0000007f950be628 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#11 0x0000007f94e2e088 in start_thread (arg=0x7f8f2a12cf) at pthread_create.c:463
#12 0x0000007f952a0ffc in thread_start () at …/sysdeps/unix/sysv/linux/aarch64/clone.S:78
Im attaching the crash dump:
_usr_sbin_nvargus-daemon.0.crash (21.4 MB)
After this the our recovery mechanism kicks in shuts down our pipelines and restarts the nvargus-daemon.
In the nvargus-daemon log I see the following:
may 24 14:46:28 falconprod nvargus-daemon[25496]: === gst-launch-1.0[25701]: Connection closed (7F8F2A21D0)=== gst-launch-1.0[25701]: WARNING: CameraProvider was not destroyed before client connection terminated.=== gst-launch-1.0[25701]: The client may have abnormally terminated. Destroying CameraProvider…=== gst-launch-1.0[25701]: CameraProvider destroyed (0x7f88b5c610)=== gst-launch-1.0[25701]: WARNING: Cleaning up 1 outstanding requests…=== gst-launch-1.0[25701]: WARNING: Cleaning up 1 outstanding streams…SCF: Error InvalidState: 3 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 306)
may 24 14:46:28 falconprod nvargus-daemon[25496]: SCF: Error Disconnected: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 539)
may 24 14:46:28 falconprod nvargus-daemon[25496]: SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
may 24 14:46:28 falconprod nvargus-daemon[25496]: === gst-launch-1.0[25701]: WARNING: Cleaning up 1 outstanding stream settings…=== gst-launch-1.0[25701]: WARNING: Cleaning up 1 outstanding sessions…(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/gl/EGLStreamProducer.cpp:212) (in Mutex.cpp, function lock(), line 79)
may 24 14:46:28 falconprod systemd[1]: Stopping Argus daemon…
Before this there is nothing nor in the dmesg log.
Do you have any idea what can cause this? Or how could we advance to find out the reason?
Thanks!
Bests,
peter