Hello,
I am on JetPack 5.1.2, and have a Jetson Orin AGX connected to two compatible Framos sensors. I am running into an issue where I get a TIMEOUT error from nvarguscamerasrc after ~4 hours of runtime (sometimes less). My gstreamer pipeline takes the video from each sensor and sends it to a udpsink. I have also seen the same issue if I output each sensor stream to the Jetson monitor using nv3dsink.
The error looks like this:
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: TIMEOUT
Additional debug info:
Argus Error Status
EOS on shutdown enabled -- waiting for EOS after Error
Waiting for EOS...
CONSUMER: ERROR OCCURRED
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc1: TIMEOUT
Additional debug info:
Argus Error Status
Got EOS from element "pipeline0".
EOS received - stopping pipeline...
Execution ended after 2:46:58.564004408
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
GST_ARGUS: Cleaning up
Freeing pipeline ...
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
The relevant dmesg output looks like this:
[10547.365907] Call trace:
[10547.365909] tegra_capture_ivc_notify_chan_id+0x18c/0x1b0
[10547.365914] isp_capture_setup+0x314/0x5c0
[10547.365916] isp_channel_ioctl+0x248/0x590
[10547.365921] __arm64_sys_ioctl+0xac/0xf0
[10547.365925] el0_svc_common.constprop.0+0x80/0x1d0
[10547.365927] do_el0_svc+0x38/0xb0
[10547.365930] el0_svc+0x1c/0x30
[10547.365932] el0_sync_handler+0xa8/0xb0
[10547.365933] el0_sync+0x16c/0x180
[10547.365935] ---[ end trace 73fbb883ffe28fc8 ]---
[10547.365954] tegra194-isp5 14800000.isp: failed to update control callback
[10548.409576] tegra194-isp5 14800000.isp: isp capture control message timed out
[10548.416965] tegra194-isp5 14800000.isp: isp capture setup failed
[10549.433360] tegra194-isp5 14800000.isp: isp capture control message timed out
[10550.457349] tegra194-isp5 14800000.isp: isp capture control message timed out
Restarting the nvargus-daemon does not resolve the issue. Attempting to restart the gstreamer pipeline fails with this error:
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 277)
(Argus) Error EndOfFile: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 350)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 379)
(Argus) Error EndOfFile: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error EndOfFile: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Sometimes the Jetson even freezes up after attempting to restart the pipeline after failure.
I tried to run nvargus-daemon with this flag enableCamInfiniteTimeout=1 , but it still failed after a similar time.
This is my gstreamer pipeline for reference:
gst-launch-1.0 -v -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),framerate=30/1,width=1920,height=1920' ! nvv4l2h265enc ! h265parse config-interval=-1 ! rtph265pay ! udpsink host=192.168.1.162 port=5004 sync=false nvarguscamerasrc sensor-id=1 sensor-mode=0 ! 'video/x-raw(memory:NVMM),framerate=30/1,width=1920,height=1920' ! nvv4l2h265enc ! h265parse config-interval=-1 ! rtph265pay ! udpsink host=192.168.1.162 port=5005 sync=false
I’ve attached more detailed logs for review. The only way to clear the issue is to power cycle the AGX Orin, which is not practical for my usecase.
Any help is appreciated.
Thanks!
crash-logs.zip (22.2 KB)