Infinite timeout support with GStreamer R36.4.4

Robotics & Edge Computing Jetson Systems Jetson Orin NX
1

Hey,

I tried to enable enableCamInfiniteTimeout to support indefinitely stopped streaming. Unfortunately, this does not work as intended.

Tested with gstreamer: gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=11 ! 'video/x-raw(memory:NVMM),width=(int)1296, height=(int)972, framerate=1/20' ! nvjpegenc ! fakesink

After stopping the image trigger signals, gstreamer produces the logs:

(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
...

After 7min in total without any trigger / image, the pipeline crashes with a SIGSEGV:

Caught SIGSEGV
#0  0x0000ffffa7759f24 in syscall () from /usr/lib/libc.so.6
#1  0x0000ffffa797d52c in g_cond_wait () from /usr/lib/libglib-2.0.so.0
#2  0x0000ffffa74a7f2c [PAC] in ?? () from /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so
#3  0x0000ffffa59625f8 [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#4  0x0000ffffa5967fbc [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#5  0x0000ffffa7ac6df0 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#6  0x0000ffffa7ac7978 [PAC] in gst_pad_set_active () from /usr/lib/libgstreamer-1.0.so.0
#7  0x0000ffffa7a9bf2c [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#8  0x0000ffffa7ab4a94 [PAC] in gst_iterator_fold () from /usr/lib/libgstreamer-1.0.so.0
#9  0x0000ffffa7a9c844 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#10 0x0000ffffa7a9ec3c [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#11 0x0000ffffa7a9ef48 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#12 0x0000ffffa5965908 [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#13 0x0000ffffa7aa147c [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#14 0x0000ffffa7aa1aa8 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#15 0x0000ffffa7a79f88 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#16 0x0000ffffa7aa147c [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#17 0x0000ffffa7aa1504 [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#18 0x0000ffffa7aa1aa8 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#19 0x0000aaaad02a5e00 [PAC] in ?? ()
#20 0x0000ffffa769868c [PAC] in ?? () from /usr/lib/libc.so.6
#21 0x0000ffffa7698760 [PAC] in __libc_start_main () from /usr/lib/libc.so.6
#22 0x0000aaaad02a3930 [PAC] in ?? ()

Generated logs:

gstreamer output 
root@ovp93x-a2-ff-7f (productive):~# gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=11 ! 'video/x-raw(memory:NVMM),width=(int)1296, height=(int)972, framerate=1/20' ! nvjpegenc ! fakesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 11 
   Output Stream W = 1296 H = 972 
   seconds to Run    = 0 
   Frame Rate = 20.000000 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
CONSUMER: Producer has connected; continuing.
NvMMLiteBlockCreate : Block : BlockType = 1 
Redistribute latency...
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
CONSUMER: Done Success
Got EOS from element "pipeline0".
Execution ended after 0:07:37.965437696
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
Caught SIGSEGV
#0  0x0000ffffa7759f24 in syscall () from /usr/lib/libc.so.6
#1  0x0000ffffa797d52c in g_cond_wait () from /usr/lib/libglib-2.0.so.0
#2  0x0000ffffa74a7f2c [PAC] in ?? () from /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so
#3  0x0000ffffa59625f8 [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#4  0x0000ffffa5967fbc [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#5  0x0000ffffa7ac6df0 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#6  0x0000ffffa7ac7978 [PAC] in gst_pad_set_active () from /usr/lib/libgstreamer-1.0.so.0
#7  0x0000ffffa7a9bf2c [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#8  0x0000ffffa7ab4a94 [PAC] in gst_iterator_fold () from /usr/lib/libgstreamer-1.0.so.0
#9  0x0000ffffa7a9c844 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#10 0x0000ffffa7a9ec3c [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#11 0x0000ffffa7a9ef48 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#12 0x0000ffffa5965908 [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#13 0x0000ffffa7aa147c [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#14 0x0000ffffa7aa1aa8 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#15 0x0000ffffa7a79f88 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#16 0x0000ffffa7aa147c [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#17 0x0000ffffa7aa1504 [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#18 0x0000ffffa7aa1aa8 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#19 0x0000aaaad02a5e00 [PAC] in ?? ()
#20 0x0000ffffa769868c [PAC] in ?? () from /usr/lib/libc.so.6
#21 0x0000ffffa7698760 [PAC] in __libc_start_main () from /usr/lib/libc.so.6
#22 0x0000aaaad02a3930 [PAC] in ?? ()
Spinning.  Please run 'gdb gst-launch-1.0 1834' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
nvargus-daemon 
Jan 01 05:32:26 ovp93x-a2-ff-7f systemd[1]: Started NVIDIA Argus daemon.
Jan 01 05:32:54 ovp93x-a2-ff-7f nvargus-daemon[1822]: === NVIDIA Libargus Camera Service (0.99.33)=== Listening for connections...=== gst-launch-1.0[1834]: Connection established (FFFFA0C6B8C0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Jan 01 05:32:54 ovp93x-a2-ff-7f nvargus-daemon[1822]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jan 01 05:32:54 ovp93x-a2-ff-7f nvargus-daemon[1822]: NvPclHwGetModuleList: No module data found
Jan 01 05:32:55 ovp93x-a2-ff-7f nvargus-daemon[1822]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Jan 01 05:32:55 ovp93x-a2-ff-7f nvargus-daemon[1822]: ---- imager: Found override file [/var/nvidia/nvcam/settings/ar0544_centerright_ifmar0544.isp]. ----
Jan 01 05:32:58 ovp93x-a2-ff-7f nvargus-daemon[1822]: CAM: serial no file already exists, skips storing again=== gst-launch-1.0[1834]: CameraProvider initialized (0xffff9cb9f840)CAM: serial no file already exists, skips storing againPowerServiceCore:handleRequests: timePassed = 1055

Specs:

  • Orin NX 8GB (P3767-0001)
  • L4T 36.4.4
  • gstreamer plugin version gstreamer1.0-plugins-nvarguscamerasrc_1.0.0-r36.4.4
5

hello florian.glaser,

may I know what’s the actual use-case? it’s quite long without camera frames on CSI channel.
although there’s infinite-timeout configuration, there’s socket wait on the server side, which has 15-sec timeout settings.

please give enableCamInfiniteTimeout another try with single-process mode.
as you can see in ~/jetson_multimedia_api/argus/README.TXT.
there’s an option to build the Libargus application in single-process mode, it’ll ignore the socket wait.
for instance, cmake -DDISABLE_MULTIPROCESS=ON ..

6

Hey JerryChang,

thanks for the reply and suggestion!

may I know what’s the actual use-case?

One use case, for example, would be a situation where certain positions must be reached first, and then a trigger signal is received. If images were captured continuously, that would increase power consumption. Turning the camera on only after the position is reached increases the delay. At the end, we also do not know how long our customers would need between each trigger signals, so we assume “indefinitely”

please give enableCamInfiniteTimeout another try with single-process mode.

I build the libgstnvarguscamerasrc.so linked to the single process lib libnvargus.so and got a crash after some seconds.

# gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=11 ! 'video/x-raw(memory:NVMM),width=(int)1296, height=(int)972, framerate=1/20' ! nvjpegenc ! fakesink
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/ar0544_centerright_ifmar0544.isp]. ----
CAM: serial no file already exists, skips storing againSetting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
CAM: serial no file already exists, skips storing againGST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
CONSUMER: Producer has connected; continuing.
NvMMLiteBlockCreate : Block : BlockType = 1 
Redistribute latency...
CONSUMER: Done Success
Got EOS from element "pipeline0".
Execution ended after 0:00:25.201819040
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 283 draining session frameStart events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
SCF: Error Timeout: Sending critical error event for Session 2 
 (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
SCF: Error InvalidState: Sensor GUID 2 is in error state. Skipping requests, capture sequence ID = 284 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
SCF: Error InvalidState: Sensor 2 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 282 draining session frameEnd events 3
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
(Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function waitForIdle(), line 729)
SCF: Error InvalidState: 2 buffers still pending during EGLStreamProducer destruction (in src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
GST_ARGUS: Done Success
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 1069)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1164)
(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/gl/EGLStreamProducer.cpp:494) (in Mutex.cpp, function lock(), line 79)
Caught SIGSEGV
#0  0x0000ffff82a328c4 in ?? () from /usr/lib/libc.so.6
#1  0x0000ffff82a37730 [PAC] in ?? () from /usr/lib/libc.so.6
#2  0x0000ffff82cbd424 [PAC] in ?? () from /usr/lib/libglib-2.0.so.0
#3  0x0000ffff82c8d394 [PAC] in g_thread_join () from /usr/lib/libglib-2.0.so.0
#4  0x0000ffff827e7fc4 [PAC] in gst_nv_argus_camera_stop (src_base=0xaaab112603b0) at gstnvarguscamerasrc.cpp:2001
#5  0x0000ffff80d425f8 [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#6  0x0000ffff80d47fbc [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#7  0x0000ffff82e06df0 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#8  0x0000ffff82e07978 [PAC] in gst_pad_set_active () from /usr/lib/libgstreamer-1.0.so.0
#9  0x0000ffff82ddbf2c [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#10 0x0000ffff82df4a94 [PAC] in gst_iterator_fold () from /usr/lib/libgstreamer-1.0.so.0
#11 0x0000ffff82ddc844 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#12 0x0000ffff82ddec3c [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#13 0x0000ffff82ddef48 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#14 0x0000ffff80d45908 [PAC] in ?? () from /usr/lib/libgstbase-1.0.so.0
#15 0x0000ffff82de147c [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#16 0x0000ffff82de1aa8 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#17 0x0000ffff82db9f88 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#18 0x0000ffff82de147c [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#19 0x0000ffff82de1504 [PAC] in gst_element_change_state () from /usr/lib/libgstreamer-1.0.so.0
#20 0x0000ffff82de1aa8 [PAC] in ?? () from /usr/lib/libgstreamer-1.0.so.0
#21 0x0000aaaae2615e00 [PAC] in ?? ()
#22 0x0000ffff829d868c [PAC] in ?? () from /usr/lib/libc.so.6
#23 0x0000ffff829d8760 [PAC] in __libc_start_main () from /usr/lib/libc.so.6
#24 0x0000aaaae2613930 [PAC] in ?? ()
Spinning.  Please run 'gdb gst-launch-1.0 1603' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
waitForIdleLocked remaining request 383 
waitForIdleLocked remaining request 382 
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 969)
waitForIdleLocked remaining request 383 
waitForIdleLocked remaining request 382 
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 969)
waitForIdleLocked remaining request 383 
waitForIdleLocked remaining request 382 
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 969)
waitForIdleLocked remaining request 383 
waitForIdleLocked remaining request 382 
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 969)
^C^C^Z
7

hello florian.glaser,

it looks strange to me that SCF report timeout even with enableCamInfiniteTimeout.
let’s collect nvargus-daemon logs for more details.
for instance, $ sudo journalctl -b -u nvargus-daemon

BTW,
is it possible for moving forward to the latest JP-6 public release version?

8

Hey JerryChang,

the nvargus-daemon does not produce any logs if DISABLE_MULTIPROCESS=ON is set

systemd[1]: Started NVIDIA Argus daemon.

Just to make sure, I checked the linked lib’s linked to libgstnvarguscamerasrc

readelf -d /tmp/libgstnvarguscamerasrc.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libnvbufsurface.so.1.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libnvbufsurftransform.so.1.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libnvdsbufferpool.so.1.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libnvargus.so]   # <---- here
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgstvideo-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgstbase-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgstallocators-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgstreamer-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libjson-glib-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgio-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgobject-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libglib-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-aarch64.so.1]

is it possible for moving forward to the latest JP-6 public release version?

This is part of our long-term plan. We are currently evaluating whether we can update to JetPack 6.2.3/L4T R36.5.2 (latest version at OE4T meta-tegra)

9

hello florian.glaser,

since you’re running with single-process mode, you should have the environment variable attached directly to gst-launch pipeline.
for instance,
$ sudo env enableCamInfiniteTimeout=1 gst-launch-1.0 -v nvarguscamerasrc sensor-id=0 sensor-mode=11 ! 'video/x-raw(memory:NVMM),width=(int)1296,height=(int)972,framerate=(fraction)20/1' ! fakesink sync=false 2>&1 | tee /tmp/topic383887-debug.log

11

Hey JerryChang,
thank you for pointing that out!

I tested it and now we crash after 1h. So it is getting better.
Here are the logs (nvargus-daemon did not produce any logs):

/tmp/topic383887-debug.log 
# env enableCamInfiniteTimeout=1 gst-launch-1.0 -v nvarguscamerasrc sensor-id=0 sensor-mode=11 ! 'video/x-raw(memory:NVMM),width=(int)1296,height=(int)972,framerate=(fraction)20/1' ! fakesink sync=false 2>&1 | tee /tmp/topic383887-debug.log
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/ar0544_centerright_ifmar0544.isp]. ----
CAM: serial no file already exists, skips storing againOFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/ar0544_centerright_ifmar0544.isp]. ----
CAM: serial no file already exists, skips storing againSetting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1296, height=(int)972, format=(string)NV12, framerate=(fraction)20/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1296, height=(int)972, format=(string)NV12, framerate=(fraction)20/1
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1296, height=(int)972, format=(string)NV12, framerate=(fraction)20/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1296, height=(int)972, format=(string)NV12, framerate=(fraction)20/1
CAM: serial no file already exists, skips storing againGST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 11 
   Output Stream W = 1296 H = 972 
   seconds to Run    = 0 
   Frame Rate = 20.000000 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
CONSUMER: Producer has connected; continuing.
Redistribute latency...


SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 2406 draining session frameEnd events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error Timeout: Sending critical error event for Session 2 
 (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
SCF: Error InvalidState: Sensor 2 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 2407 draining session frameEnd events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error InvalidState: Sensor 2 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 2406 draining session frameStart events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
SCF: Error InvalidState: Sensor GUID 2 is in error state. Skipping requests, capture sequence ID = 2407 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: TIMEOUT
Additional debug info:
Argus Error Status
CONSUMER: ERROR OCCURRED
Execution ended after 1:13:48.432135552
Setting pipeline to NULL ...
PowerServiceCore:handleRequests: timePassed = 4307353
GST_ARGUS: Cleaning up
Freeing pipeline ...

But to be honest, I don’t see much hope in the single-process approach, since we have a setup with multiple cameras. If the single-process approach were to work, we’d have to check whether it’s even feasible for us.

That’s why we’d prefer a solution using the daemon. Do you have any other ideas?

14

hello florian.glaser,

just an FYI, we’ve verified with the latest JP-6 public release version. (JP-6.2.3/r36.5.2)
for instance,
we’ve enableCamInfiniteTimeout enabled, launching camera stream via gst pipeline with nvarguscamerasrc plugin, using software simulated approach to stop-stream for 20-min.
there’re Error Timeout reported, which is expected, but there’s no crashes with a SIGSEGV.

15

Hey CherryChang,

thanks for the testing.
Would it be possible to port nvargus r36.5.2 back to the r36.4.4 version? Or apply a patch that would fix the SEGV?

16

hello florian.glaser,

unfortunately, we usually not back-port the fixes.
please moving forward to the latest JP-6 to have verification.