Argus camera stop after USB-storage disconnect

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hi,
if simultaneously started
A) argus stream:
“gst-launch-1.0 nvarguscamerasrc ! ‘video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1’ ! fakesink silent=false -v”
B) USB storage write
“dd if=/dev/zero of=/mnt/flash_mount/tmp.raw bs=1M count=10000”

Argus stream is stopped if USB disconnected (physically)

with argus log:
SCF: Error Timeout: FuSa Capture Status failed! (in src/services/capture/FusaCaptureIspHw.cpp, function waitIspFrameEnd(), line 365)
SCF: Error InvalidState: (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 593)
SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
SCF: Error InvalidState: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function startCaptureInternal(), line 809)
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCapture(), line 530)
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 477)
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1291)
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1122)
SCF: Error ResourceAlreadyInUse: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
SCF: Error ResourceAlreadyInUse: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 643)
SCF: Error Timeout: (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 373)
SCF: Error Timeout: (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
SCF_AutocontrolACSync failed to wait for an earlier frame to complete.

SCF: Error Timeout: (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 126)
SCF: Error Timeout: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 979)
SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)

3

hello konstantin.sobolev2,

is there any kernel panic reported while you remove USB storage device?
besides, may I know which Jetpack release version you’re working with?

4

Jetpack Linux 35.1.0 ( 5.0.2 )

5

is it possible for moving to the latest JetPack 5.1.2 release version for confirmation?

6

this event reported in both consoles(after USB drive disconnect)
Message from syslogd@test at Feb 21 07:06:48 …
kernel:[ 3024.231752] EXT4-fs (sdb1): failed to convert unwritten extents to written extents – potential data loss! (inode 12, error -30)

Not sure we can move to 5.1.x now because of BSP , camera drivers , et.c.

7

hello konstantin.sobolev2,

could you please also setup a terminal for gathering logs for further checking.
for example, $ dmesg --follow to keep monitor kernel status, and reproduce the issue again to gather complete error logs.

8

Hi, here’s the logs

xavier_logs.log (44.1 KB)

9

hello konstantin.sobolev2,

I don’t see any camera errors reported by kernel layer.
it should be system gating by below I/O errors…

[ 1301.084227] blk_update_request: I/O error, dev sda, sector 12630928 op 0x1:(WRITE) flags 0x4000 phys_seg 30 prio class 0
[ 1301.087283] blk_update_request: I/O error, dev sda, sector 12631168 op 0x1:(WRITE) flags 0x4000 phys_seg 30 prio class 0
[ 1301.098264] blk_update_request: I/O error, dev sda, sector 12631408 op 0x1:(WRITE) flags 0x4000 phys_seg 30 prio class 0
[ 1301.109474] blk_update_request: I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 1301.119612] Buffer I/O error on dev sda1, logical block 0, lost sync page write

may I know what’s the real use-case for having USB storage write but disconnect physically?

10

The system should not be stopped by user interaction ( remove storage)

Interesting that pipeline with “nvv4l2camerasrc” works without interruption …
Can we check lower layers of camera API (lower than argus library)?

11

hello konstantin.sobolev2,

it might be low-level socket communication failed by those I/O errors.

for issue narrow down,
let’s configure DISABLE_MULTIPROCESS for testing with single-process mode, it does not require nvargus-daemon service, (i.e. /tmp/argus_socket for communication)
please see-also /usr/src/jetson_multimedia_api/argus/README.TXT for details.
you may try to rebuild the argus_camera application with… $ cmake -DDISABLE_MULTIPROCESS=ON ...

12

Tested with -DDISABLE_MULTIPROCESS build “/usr/src/jetson_multimedia_api/argus/”

demo@test:/usr/src/jetson_multimedia_api/argus/build$ grep ‘DISABLE_MULTIPROCESS’ . -r
./CMakeCache.txt:DISABLE_MULTIPROCESS:UNINITIALIZED=ON

Argus service was stopped
system.log (11.4 KB)
camera_app_try_2.log (26.9 KB)
camera_app.log (3.0 KB)

13

hello konstantin.sobolev2,

BTW, do you have debug rce-fw applied? where did you obtain that binary file, may I know you’re using the correct version.
besides, can this repo’ed with default JP-5.1 release version?

14

hello @JerryChang
we work with Jetpack5.0.2 and we took the debug rce-fw from your answer in this discussion:

we also check and made the same procedure with the 5.0.2 rce-fw without the debug and the issue persisted.

Thanks,
Ori

15

hello @JerryChang
We are testing JP-5.1.1 and has same error.
We record video to usb storage for five minutes and stop recording and then execute sync command,
finally, unplug the USB device, then nvargus-daemon report erros.
there is no errors in dmesg log.

below is nvargs log details:

SCF: Error InvalidState:  (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 600)
SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
SCF: Error InvalidState: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function startCaptureInternal(), line 816)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCapture(), line 530)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 477)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1291)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1122)
SCF: Error ResourceAlreadyInUse:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
SCF: Error ResourceAlreadyInUse: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 644)
SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 426)