Argus timeout when capturing frames followed by unrecoverable state

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
,
25

@ShaneCCC ,

As promised, here is the complete daemon log captured with your exact command (journalctl -u nvargus-daemon -f teeing to disk), covering the daemon from a fresh start, through ~3.7 hours of healthy operation, into the wedge, and through the restart that recovers it.

Setup for this capture: the production-like pipeline (3x nvarguscamerasrc -> nvv4l2h264enc -> matroskamux -> filesink, 1080p120, 5 s recordings per cycle, ~24 s cycle cadence), stock r36.4.3 camera stack, 25 W power mode, fresh boot and fresh daemon.

Timeline in the attached log:

  • 15:19:21: daemon started (systemd), harness begins record cycles.
  • 15:19 to 19:01: 556 healthy cycles. Worth noting: 373 SCF: Error InvalidState lines during this healthy span (viCsiHw isn't closed for source, propagating through CaptureService closeSource → Session shutdown → deleteSession), all during teardowns that nevertheless completed. Dirty teardown is routine background behavior; roughly once per ~450 cycles it is fatal instead.
  • 19:01:17, onset (cycle 557): a session teardown jams. SCF: Error Timeout: waitForIdle() timed out (Session.cpp, waitForIdleLocked(), line 969), then at 19:01:27 SCF: Error InvalidState: 1 buffers still pending during EGLStreamProducer destruction, then waitForIdleLocked remaining request 718 frozen at that value (19:01:32, 19:01:37). Our process telemetry shows the daemon fd count jumping 19 → 78 at this exact cycle, our 11-for-11 leading indicator.
  • 19:01 to 19:05: the next two record cycles connect successfully but receive zero frames (321-byte header-only output files on all three cameras); the wedged session never drains.
  • 19:05:15: the field workaround (systemctl restart nvargus-daemon); the next cycle at 19:06 is fully healthy (fd back to 19, daemon RSS reset from ~1.2 GB to ~200 MB).

One detail relevant to root-causing: there is no SCF_AutocontrolACSync line in this event. Across our 11 captured wedges, some show the ACSync timeout at onset and some do not; the invariant is the teardown jam itself (frozen undrained request queue, exactly one stranded EGLStream buffer, eternal waitForIdle), not ACSync. Kernel dmesg is silent throughout, as always.

Since all the jam-side error lines originate in libnvscf.so, our earlier questions stand: does the JetPack 6.2.2 SCF capture-service restructuring (CaptureDeviceSingleThread) address this teardown race, is it present in 6.2.1, and is a patched libnvscf.so for r36.4.3 available?

best,
Andres Campos

connect_program

26

If this happened without an intentional app abort, I would suspect the camera stopped delivering a frame or Argus could not complete request 718, and the rest of the errors are cascade failures during cleanup.

Could you verify by argus_camera?

27

@ShaneCCC,

Thanks for getting back so quickly.

The app abort is intentional.
Our use case requires us to start and stop video capture constantly.

That is why our test harness is built on running hours of intended restarts.

What tests could we run on our side to better help you debug this issue?

best,
Andres Campos

connect_program

28

Rsstart the daemon for each cycle to try.

29

@ShaneCCC,

Restarting the daemon does fix the issue.

But do you mean restart on each cycle just to see if it ends up on the bad state either way ?

best,
Andres Campos

30

Yes, restart the daemon before launch the camera every time.

Thanks

31

Is there any update or follow up on this?

restart the daemon before launch the camera every time.

does not seem like a practical solution in production because:

  1. Not everyone uses the daemon, nvidia supports a non-daemon version of the Argus library, which is what we use for example
  2. In case of application wedged on a waitForIdle call in a production environment, there’s no user that can access a device and restart a daemon. One would have to implement some sort of “watchdog” do detect these cases and tell systemd to restart the daemon? Seems like something nvidia should provide, not written at application layer
32

Here’s libs for long run problem.

libnvfusacap.so.r36.5_None_notify_block (232.0 KB)

33

Is this a drop-in replacement for libnvfusacap.so ?

34

Yes, replace the libnvfusacap.so and restart the nvargus-daemon if using daemon.

35

Thank you,

Do you have a r36.4.3 version of this patch? or can I use the same?

36

Attached the 36.4 and 36.5 libs here.

libnvfusacap.so.r36.4.none_notify_block (231.9 KB)

libnvfusacap.so.r36.5_None_notify_block (232.0 KB)

37

Thank you,

I will test today and let you know

38

I tried the r36.4 patch and it hung as described in Argus timeout when capturing frames still occurs in JetPack 6.2.2 (r36.5.0) on the very first run

39

On every first run?

That’s weird do you run by using daemon?

40

Yes, I am using nvargus-daemon and the argus_syncsensor example. Every time I run it, it hangs at the end.

However I now checked the previous version of the library: it seems that the argus_syncsensor example behaves exactly the same, reaching the end and hanging.

I have not looked inside the argus_syncsensor example source code supplied by Nvidia, I might need to review it to ensure there isn’t any issue with he sample itself, I’ll let you know once I’ve done that.

41

It seems that the systematic hang at the end is caused by the argus_syncsensor example requiring a display, while my setup is headless. Changing EGLDisplayHolder(bool disableWindow = false); to true in the example fixes it.

With this out of the way, I can now test the original repeated start/stop use case with the new library as intended.

I’ll let you know the results of the tests when I’m finished.

42

I repeated the start/stop experiment for several batches of 1000 repetitions.

There seem to be a different issue now, affecting “start” rather than “stop” (waitForIdle) phase.

Example of failure log from argus_syncsensor (e.g. 59 of 1000 runs):

# Runs 0 to 58 succeeded

[2026-08-10 12:47:17] Run 59/10000: -t 5

Executing Argus Sample: argus_syncsensor
Argus Version: 0.99.3.3 (multi-process)
PRODUCER: Creating left stream.
PRODUCER: Creating right stream.
PRODUCER: Launching disparity checking consumer
Initializing CUDA
CONSUMER: Connecting CUDA consumer to left stream
CONSUMER: Connecting CUDA consumer to right stream
CONSUMER: Waiting for Argus producer to connect to left stream.
PRODUCER: Starting repeat capture requests.
CONSUMER: Waiting for Argus producer to connect to right stream.
CONSUMER: Streams connected, processing frames.
PRODUCER: Captures complete, disconnecting producer.

# My script kills process if hangs for more than a minute
./repeat.sh: line 22: 173667 Killed                  timeout --signal=KILL "${timeout_s}" ${program} "$@"
[2026-08-10 12:48:17] TIMEOUT on run 59/10000 after 60: -t 5

And corresponding failure in nvargus-daemon for the failed run:

=== argus_syncsensor[173668]: Connection established (FFFF83FAB840)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
LSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
LSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
(NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
(NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 111)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
NvPclOpen: PCL Open Failed. Error: 0xf
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 725)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 455)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 382)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 554)
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
(NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
(NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 111)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclStateControllerOpen: Failed ImagerGUID 1. (error 0xA000E)
NvPclOpen: PCL Open Failed. Error: 0xf
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 725)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 455)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 382)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 554)

=== argus_syncsensor[173668]: CameraProvider initialized (0xffff7db8d830)LSC: LSC surface is not based on full res!
SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
SCF_AutocontrolACSync failed to wait for an earlier frame to complete.

SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 145)
SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
SCF: Error Timeout: Sending critical error event for Session 4 
 (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 1069)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1164)
SCF: Error InvalidState: Sensor 2 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 0 draining session frameEnd events 3
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 0 draining session frameStart events 3
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
SCF: Error InvalidState: Sensor 2 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 1 draining session frameEnd events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error InvalidState: Sensor 2 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 2 draining session frameEnd events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error InvalidState: Sensor GUID 2 is in error state. Skipping requests, capture sequence ID = 1 continue draining session frameStart events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 2 draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
=== argus_syncsensor[173668]: Connection closed (FFFF83FAB840)=== argus_syncsensor[173668]: WARNING: CameraProvider was not destroyed before client connection terminated.=== argus_syncsensor[173668]:          The client may have abnormally terminated. Destroying CameraProvider...=== argus_syncsensor[173668]: CameraProvider destroyed (0xffff7db8d830)=== argus_syncsensor[173668]: WARNING: Cleaning up 1 outstanding requests...=== argus_syncsensor[173668]: WARNING: Cleaning up 1 outstanding stream settings...=== argus_syncsensor[173668]: WARNING: Cleaning up 1 outstanding sessions...PowerServiceCore:handleRequests: timePassed = 56041
=== argus_syncsensor[173668]: WARNING: Cleaning up 2 outstanding streams...(Argus) Error InvalidState: Unknown stream deleted. (in src/api/CaptureSessionImpl.cpp, function outputStreamDeleted(), line 1106)
(Argus) Error InvalidState: Unknown stream deleted. (in src/api/CaptureSessionImpl.cpp, function outputStreamDeleted(), line 1106)
=== argus_syncsensor[173668]: NOTE: Destroy all libargus objects before destroying the CameraProvider to avoid these warnings.=== argus_syncsensor[173668]: Connection cleaned up (FFFF83FAB840)
43

Try infinite mode.

sudo env enableCamInfiniteTimeout=1 /usr/sbin/nvargus-daemon
44

I tried and e.g. run 222 out of 10000 failed to start with:

Executing Argus Sample: argus_syncsensor
Argus Version: 0.99.3.3 (multi-process)
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 277)
(Argus) Error EndOfFile: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 350)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 379)
(Argus) Error EndOfFile: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error EndOfFile:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
Error generated. /usr/src/jetson_multimedia_api/argus/samples/syncSensor/main.cpp, execute:357 Failed to get capture session interface
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
Error generated. /usr/src/jetson_multimedia_api/argus/samples/utils/EGLGlobal.cpp, cleanup:150 eglTerminate failed (error 0x3008)

And on the daemon side

=== argus_syncsensor[985333]: CameraProvider initialized (0xffffb5676a80)CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
Aborted

I think it segfaulted