We have some specific hardware that processes the MIPI signal, we reset that chip.
I will try to toggle the debug node to see if I can get the same results with that!
The directory /sys/kernel/debug/camera-video0 does not exist.
/sys/kernel/debug/camera-video0/streaming: No such file or directory
hello pepijn.vanheiningen,
please examine release tag, $ cat /etc/nv_tegra_release
I’ve confirm this debug node is created on JP-5.1
for example,
/sys/kernel/debug# ll camera-video*
camera-video0:
total 0
drwxr-xr-x 2 root root 0 Feb 15 07:45 ./
drwx------ 96 root root 0 Feb 15 07:46 ../
-rw-r--r-- 1 root root 0 Feb 15 07:45 streaming
camera-video1:
total 0
drwxr-xr-x 2 root root 0 Feb 15 07:45 ./
drwx------ 96 root root 0 Feb 15 07:46 ../
-rw-r--r-- 1 root root 0 Feb 15 07:45 streaming
or…
may I know what’s the camera type you’re using? is it a bayer sensor using CSI interface camera?
We are actually running 35.1/JP5.1:
head -n 1 /etc/nv_tegra_release
# R35 (release), REVISION: 1.0, GCID: 31250864, BOARD: t186ref, EABI: aarch64, DATE: Thu Aug 11 03:37:46 UTC 2022
Yes, we’re using a bayer sensor and use the CSI interface, but still not getting the camera-video folders:
/sys/kernel/debug# ll camera-video*
ls: cannot access 'camera-video*': No such file or directory
Looks like that debugfs node is not implemented in the driver for our sensor.
All right, so I implemented the camera-video* endpoint in our sensor driver. Starting and stopping briefly the sensor as follows, while in another terminal I run the gstreamer pipeline.
#!/bin/bash
echo 0 > /sys/kernel/debug/camera-video0/streaming
sleep 0.1
echo 1 > /sys/kernel/debug/camera-video0/streaming
This gives me the same Timeout waiting on frame start sensor guid 0 error as before. Do you see the same results on your camera?
P.S. I’m also getting an error in dmesg: [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=1760342946848 sof_ts=1760343144128 gerror_code=2 gerror_data=400 notify_bits=0"
hello pepijn.vanheiningen,
FYI, I can reproduce the same issue on reference camera board.
for example,
Feb 21 13:38:34 nvidia-desktop nvargus-daemon[1789]: === gst-launch-1.0[18536]: CameraProvider initialized (0xffffa8684d70)SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 942 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 524)
Feb 21 13:38:34 nvidia-desktop nvargus-daemon[1789]: SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Feb 21 13:38:34 nvidia-desktop nvargus-daemon[1789]: SCF: Error InvalidState: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Feb 21 13:38:35 nvidia-desktop nvargus-daemon: Module_id 30 Severity 2 : (fusa) Error: Timeout propagating from:/capture/src/fusaViHandler.cpp 776
Feb 21 13:38:35 nvidia-desktop nvargus-daemon[1789]: SCF: Error Timeout: (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 610)
Feb 21 13:38:35 nvidia-desktop nvargus-daemon[1789]: SCF: Error Timeout: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Feb 21 13:38:35 nvidia-desktop nvargus-daemon[1789]: SCF: Error Timeout: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
Feb 21 13:38:37 nvidia-desktop nvargus-daemon[1789]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
Feb 21 13:38:37 nvidia-desktop nvargus-daemon[1789]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
Feb 21 13:38:37 nvidia-desktop nvargus-daemon[1789]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
this is a regression, let me arrange resources for checking this.
in the meanwhile.
please have below as temporary solution to kill and restart nvargus-daemon service to restore the camera functionality.
$ sudo pkill nvargus-daemon
$ sudo systemctl start nvargus-daemon
Thank you for reproducing it! Unfortunately the temporary solution doesn’t work for us, since this can happen many times per hour. Restarting the nvargus-daemon and our pipelines will take a while, where we lose video.
Do you have any idea when a fix might be available? This is currently blocking the production of our new cameras.
hello pepijn.vanheiningen,
we haven’t root cause the issue, and it may take some time to figure-out the solution.
let me arrange resources for investigation, you should also expect this won’t be fix soon.
Thanks for your continued effort on this.
One additional question: do you have some insight into how easy it will be to update our Xaviers over-the-air later? Or do you think this fix will need to be applied when flashing the device?
Please keep me up-to-date if you learn more about the root cause of the problem!
hello pepijn.vanheiningen,
I cannot give you a solid answer since we have not root caused the issue.
let’s assume it’s the bug on camera stack,
for example, you may update the pre-built binary, (i.e. /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so) this can be done remotely via ssh.
in this case, it’ll be simple to apply the fix. the update will take effect after a system reboot.
1 Like
Hey Jerry,
Do you happen to have any update on the topic? Did you get some resources for investigation? Any updates on a potential timeline?
If there is anything I can do to help out, let me know!
Thanks in advance!
hello pepijn.vanheiningen,
please refer to JetPack 5.1.1 is now live,
please have a test with the latest release image.
the error recovery mechanism should be functional with Jetpack 5.1.1 / l4t-35.3.1
moreover,
here shows test steps in brief.
step1) launch gst pipeline to enable camera preview
$ gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),framerate=30/1,format=NV12' ! nvvidconv ! xvimagesink
step2) sending commands on the terminal to shutdown the stream,
# cd /sys/kernel/debug/camera-video0
# echo 0 > streaming
the expected behavior is camera app it should terminate gracefully and we should not need to restart argus daemon service to restore the camera functionality.
1 Like
I just tried it on 35.3.1, but it still is stalling for me.
I’m doing the exact same steps as mentioned here, but the pipeline doesn’t continue to run. I’m also not seeing the NVCSI error handling that happened on 32.4.4.
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: TIMEOUT
Additional debug info:
Argus Error Status
Execution ended after 0:00:08.178493900
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Logs from nvargus-daemon:
Mar 30 07:42:58 camera nvargus-daemon[6549]: SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 80 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 514)
Mar 30 07:42:58 camera nvargus-daemon[6549]: SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Mar 30 07:42:58 camera nvargus-daemon[6549]: SCF: Error InvalidState: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Mar 30 07:42:58 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 600)
Mar 30 07:42:58 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Mar 30 07:42:58 camera nvargus-daemon[6549]: SCF: Error Timeout: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
Mar 30 07:42:58 camera nvargus-daemon[6549]: Module_id 30 Severity 2 : (fusa) Error: Timeout propagating from:/capture/src/fusaViHandler.cpp 776
Mar 30 07:43:00 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
Mar 30 07:43:00 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
Mar 30 07:43:00 camera nvargus-daemon[6549]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
Mar 30 07:43:03 camera nvargus-daemon[6549]: waitForIdleLocked remaining request 180
Mar 30 07:43:03 camera nvargus-daemon[6549]: waitForIdleLocked remaining request 179
Mar 30 07:43:03 camera nvargus-daemon[6549]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
Mar 30 07:43:03 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/api/Session.cpp, function abortCaptures(), line 888)
Mar 30 07:43:59 camera nvargus-daemon[6549]: SCF: Error InvalidState: 2 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
Mar 30 07:43:59 camera nvargus-daemon[6549]: SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 49)
Mar 30 07:44:04 camera nvargus-daemon[6549]: waitForIdleLocked remaining request 180
Mar 30 07:44:04 camera nvargus-daemon[6549]: waitForIdleLocked remaining request 179
Mar 30 07:44:04 camera nvargus-daemon[6549]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
Mar 30 07:44:04 camera nvargus-daemon[6549]: (Argus) Error Timeout: (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 216)
Mar 30 07:44:05 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
Mar 30 07:44:05 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
Mar 30 07:44:05 camera nvargus-daemon[6549]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
Mar 30 07:44:09 camera nvargus-daemon[6549]: waitForIdleLocked remaining request 180
Mar 30 07:44:09 camera nvargus-daemon[6549]: waitForIdleLocked remaining request 179
Mar 30 07:44:09 camera nvargus-daemon[6549]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
Mar 30 07:44:09 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/api/Session.cpp, function abortCaptures(), line 888)
Mar 30 07:44:09 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/api/Session.cpp, function shutdown(), line 407)
Mar 30 07:44:10 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
Mar 30 07:44:10 camera nvargus-daemon[6549]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
Mar 30 07:44:10 camera nvargus-daemon[6549]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
hello pepijn.vanheiningen,
thanks for sharing test results.
may I know how you enable camera stream,
could you please try with Argus example, i.e. Argus/public/samples/userAutoExposure
Still the same as before:
gst-launch-1.0 nvarguscamerasrc ee-mode=0 tnr-mode=0 aeantibanding=0 silent=false ! fakesink
I have tried the userAutoExposure, but it stops working when the sensors are reset as well.
hello pepijn.vanheiningen,
please see-also forum topic, How to make Argus in Jetson 35.2.1 recover after a corrupted frame?
it looks error handling mechanism did not works, we do reproduce the issue locally.
test environment… l4t-r35.3.1 + AGX Orin + IMX274.
let us check this issue internally.
Hi,
We hit the exact same issue. Any updates on this?
hello casperlyngesen.mogensen,
here’s pre-built update, Topic243051_Jun05.7z (1.8 MB)
could you please based-on JetPack-5.1.1/l4t-r35.3.1 to update the binaries with attachment,
you may preform a warm-reboot, i.e. $ sudo reboot after replace the binary files.