Nvargus-daemon hangs after "SCF: Error InvalidState: Corr Error Received for sensor 4 .. Continuing!"

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
,
1

We are using the Jetson AGX Xavier with a camera carrier board from Leopard Imaging (LI-JXAV-MIPI-ADPT-4CAM V1.0) with 3 cameras connected. The Jetson AGX Xavier is running Jetson Linux release: R35.4.1 JetPack release: 5.1.2 built and deployed with Yocto based on the Kirkstone branch of meta-tegra. We are also running a custom driver supplied by Leopard Imaging on top of the Jetson Linux release.

With this setup we are currently seeing errors in the nvargus-daemon after anything from 20 minutes to several hours when running gstreamer video pipelines. The first error lines are usually the following:

Nov 27 12:25:35 E2-028 nvargus-daemon[1954291]: SCF: Error InvalidState:  Corr Error Received for sensor 4 .. Continuing!
Nov 27 12:25:35 E2-028 nvargus-daemon[1954291]:  (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 643)
Nov 27 12:25:36 E2-028 nvargus-daemon[1954291]: SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
Nov 27 12:25:36 E2-028 nvargus-daemon[1954291]: SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
Nov 27 12:25:36 E2-028 nvargus-daemon[1954291]: SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 126)
Nov 27 12:25:36 E2-028 nvargus-daemon[1954291]: SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
Nov 27 12:25:36 E2-028 nvargus-daemon[1954291]: SCF: Error Timeout: Sending critical error event for Session 4
Nov 27 12:25:36 E2-028 nvargus-daemon[1954291]:  (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
Nov 27 12:25:39 E2-028 nvargus-daemon[1954291]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
Nov 27 12:25:39 E2-028 nvargus-daemon[1954291]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)

Attached are the script used to start these pipelines, the logs from the gstreamer pipeline and the log from nvargus-daemon. When nvargus-daemon crashes we need to first stop the gstreamer application, then restart nvargus-daemon, then we can start the gstreamer pipeline again.

We can currently not run our camera stack since nvargus-daemon hangs after these errors and have to be restarted.

test_dual_argus.txt (937 Bytes)
crash_nvargus_daemon.txt (1.3 MB)
crash_client.txt (147.7 KB)

3

Did you boost the system and clocks to try.

sudo nvpmodel -m 0
sudo jetson_clocks
sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vic/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vic/state
cat /sys/kernel/debug/bpmp/debug/clk/vic/max_rate > /sys/kernel/debug/bpmp/debug/clk/vic/rate
4

Hi Shane,

We tried doing this today, and the only effect it seems to have is that we now get an error even quicker, within a few minutes of starting the gstreamer application.
However I’m not sure if the jetson_clocks call worked, since it prints the following error:
tee: /sys/devices/system/cpu/cpu0/cpuidle/state0/disable: I/O error

The errors also looks slightly different now, logs from nvargus-daemon are attached.
new-nvargus-errors.txt (10.8 KB)

5

Replace the libs at /usr/lib/aarch64-linux-gnu/tegra to try.

Bug4236032_Nov22_libnvscf.zip (2.3 MB)

6

Hi Shane,

Unfortunately we still see the same errors after having replaced the libnvscf binary as well.

7

Please try this lib.
libnvscf.so (8.4 MB)

9

I am experiencing an identical problem in JP5.1.1. Is there a libnvscf.so fix for my version of JP?

Thank you very much in advance

10

I would suggest upgrade to r35.4.1 for it.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.