Argus Error OverFlow

• Hardware Platform (Jetson / GPU) Jetson NX
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only) JetPack 5.1.1 – L4T r35.3.1

Problem Description:

I’ve encountered significant issues with Argus since updating to JetPack (JP) 5.1.1, as I previously described in this post. Initially, the problems seemed to be resolved by replacing libnvargus.so with the version provided in this post. My pipeline has been functioning relatively smoothly since then.

However, after making some updates to my pipeline, specifically adding an nvtracker plugin, I’m facing new challenges. Since the addition of this plugin, I’m experiencing random problems that are difficult to diagnose due to a lack of informative error messages.

After several minutes of running, at a seemingly random interval with no clear pattern, my pipeline comes to a complete halt. In the Argus logs, I observe the following:

ene 15 15:41:33 wbNX511 nvargus-daemon[102512]: LSC: LSC surface is not based on full res!
ene 15 15:41:33 wbNX511 nvargus-daemon[102512]: LSC: LSC surface is not based on full res!
ene 15 15:41:34 wbNX511 nvargus-daemon[102512]: LSC: LSC surface is not based on full res!
ene 15 15:41:34 wbNX511 nvargus-daemon[102512]: LSC: LSC surface is not based on full res!
ene 15 15:50:21 wbNX511 nvargus-daemon[102512]: Module_id 30 Severity 2 : (fusa) Error: InvalidState Invalid Status no longer available in:/capture/src/fusaViHandler.cpp 710
ene 15 15:50:21 wbNX511 nvargus-daemon[102512]: SCF: Error InvalidState:  Corr Error Received for sensor 4 .. Continuing!
ene 15 15:50:21 wbNX511 nvargus-daemon[102512]:  (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 643)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
ene 15 15:55:54 wbNX511 nvargus-daemon[102512]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)

I would greatly appreciate any insights or suggestions. Thank you in advance for your support and guidance.

Best regards,
Ángel.

hello angel.corrales,

may I know how many camera streams you’re running, and, do you had other background applications (especially the high loading services) has executed simultaneously?

please also share your test pipelines, for example, how to recreate this issue on the develoepr kits.

Hi Jerry, thank you very much for your reply.

I am using a pipeline with 4 cameras in which I use elements such as nvarguscamerasrc, nvvideoconvert, nvinfer and nvtracker among others. Nothing else is running on the Jetson simultaneously, only the video pipeline is running.

I have monitored the resource usage and at no time has it exceeded more than 70% CPU and 40% GPU. Temperatures are stable and moderate all the time.

Most of the time the pipeline stops completely due to lack of flow (I guess the problem comes from Argus). Sometimes I have seen the cameras shut down one by one at random time intervals, until, due to the death of one of them, the program stops with the aforementioned error:

(Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)

Thank you very much in advance.

hello angel.corrales,

let’s further narrow down by putting 4-cam streaming only.
for example,
here’s gst pipeline to launch camera stream with printing frame-rate only.
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

please try initial 4 terminals, and launching all your cameras to check the stream stability.

Hello JerryChang

I have performed the tests you suggested launching the pipeline on 4 terminals and the result is as follows:

  • Sensor_2 has stopped working after 6:29:24
  • Sensor_1 has stopped working after 8:14:22
  • Sensor_3 has stopped working after 9:48:08
  • Sensor_0 has stopped working after 9:48:24

I attach the logs of each sensor on their terminals, as well as the Argus log and a jtop log during the execution of the experiment (the jtop log was partially interrupted for a while).

sensor_0.txt (8.6 MB)
sensor_1.txt (7.2 MB)
sensor_2.txt (5.7 MB)
sensor_3.txt (8.6 MB)
argus.txt (26.3 KB)
jtop_log_full.csv.txt (9.2 MB)

Hi,
Please apply the libs to try.

libnvfusacap.so.35.3.1 (193.6 KB)
libnvargus.so.35.3.1 (1.2 MB)

Hi,

After replacing the libraries with the ones provided, I have repeated the test proposed by @JerryChang with very good results. The four cameras have been running for 50 hours and 24 minutes at an average of 28fps.
sensor_0.txt (44.8 MB)
sensor_1.txt (44.8 MB)
sensor_2.txt (44.7 MB)
sensor_3.txt (44.8 MB)

I am going to perform additional tests with more complex pipelines, where I initially identified the problems.

thanks for sharing your test results,
FYI, those pre-built update in comment #8 actually the potential bug fixes for long-run scenario.

Hello @JerryChang and @ShaneCCC ,

I have tested my complete pipleine, which contains nvinfer in the 4 video sources, a later nvmultistreamtiler and an nvtracker among other elements.

The first test was finished after about 44 minutes and I could see the following error in terminal:

0:40:36.309184943 7127 0xfffeb00188c0 ERROR nvarguscamerasrc gstnvarguscamerasrc.cpp:1699:consumer_thread:<src2> NvBufSurfTransform Failed.
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
0:40:36.317609847 7127 0xffff854695e0 ERROR nvarguscamerasrc gstnvarguscamerasrc.cpp:1699:consumer_thread:<src0> NvBufSurfTransform Failed
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
0:40:36.323613259 7127 0xfffe9c013640 ERROR nvarguscamerasrc gstnvarguscamerasrc.cpp:1699:consumer_thread:<src1> NvBufSurfTransform Failed
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
0:40:36.327267050 7127 0xfffea8010b60 ERROR nvarguscamerasrc gstnvarguscamerasrc.cpp:1699:consumer_thread:<src3> NvBufSurfTransform Failed

In the Argus log the well known Overflow error:

nvargus-daemon[7142]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)

After this I used export DBG_NVBUFSURFTRANSFORM=1 and repeated the tests. The second test died after about 44 minutes with the following logs:

/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:823: taskTrim=> Time out for cleanup
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe9c019020
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffea80293a0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe9c018c10
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffea8028770
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffea8028b80
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffea8028f90
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe94014000
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe9c014300
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe980150c0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe94016c40
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe94017050
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe980148d0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe98017420
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe94016830
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe98017830
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffe9c0153a0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:448: deleteCuResData=> Destroy tex array

To make sure that there was some correlation in the pipeline stopping at 44 minutes, I repeated a third test. This third test stopped at 4h 28 minutes:

/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:823: taskTrim=> Time out for cleanup
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:851: taskTrim=> Egl map 0 size 16
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:823: taskTrim=> Time out for cleanup
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffec0015830
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffec0015420
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffeb80148c0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffebc028510
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffeb8017cc0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffebc0278e0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffeb80174a0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffebc027cf0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffeb80178b0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffebc028100
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffeb4019e70
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffec00125f0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffeb40164e0
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffeb401a280
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffeb401a690
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:838: operator()=> Remove egl res 0xfffec000b150
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:448: deleteCuResData=> Destroy tex array

Hi @JerryChang and @ShaneCCC,

I’m completely unable to resolve the issue described in #Post 11.

Additionally, after further testing, we have discovered that very simple functionalities that we had implemented in JP 4.6.1 and extensively tested over time do not work in this JP5.1.1 . For example, we have a ROS2 node that publishes images from our previously mentioned cameras on ROS2 topics, similar to the official gscam package. To achieve this, we use an extremely simple pipeline:

nvarguscamerasrc -> nvvidconv ! video/x-raw(memory:NVMM) -> videoconvert ! video/x-raw,format=(string)BGR -> appsink

After about 20 minutes of running (In the previous JP, it worked smoothly for days), the programme ends. Specifically, it fails to perform GstSample * sample = gst_app_sink_pull_sample(GST_APP_SINK(sink_)) (code line).

[video360_publisher-2] [ERROR] [1706617959.907345911] [sr.video360_publisher_1]: Could not get gstreamer sample.
[video360_publisher-2] [INFO] [1706617959.907749437] [sr.video360_publisher_1]: Stopping gstreamer pipeline...
[video360_publisher-1] [ERROR] [1706617959.998386280] [sr.video360_publisher_0]: Could not get gstreamer sample.
[video360_publisher-1] [INFO] [1706617959.998615819] [sr.video360_publisher_0]: Stopping gstreamer pipeline...
[video360_publisher-3] [ERROR] [1706617960.009455789] [sr.video360_publisher_2]: Could not get gstreamer sample.
[video360_publisher-3] [INFO] [1706617960.009699633] [sr.video360_publisher_2]: Stopping gstreamer pipeline...
[video360_publisher-4] [ERROR] [1706617960.021480961] [sr.video360_publisher_3]: Could not get gstreamer sample.
[video360_publisher-4] [INFO] [1706617960.021715076] [sr.video360_publisher_3]: Stopping gstreamer pipeline...
[video360_publisher-2] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
[video360_publisher-2] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
[video360_publisher-2] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
[video360_publisher-2] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
[video360_publisher-1] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
[video360_publisher-1] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
[video360_publisher-1] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
[video360_publisher-1] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
[video360_publisher-3] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
[video360_publisher-3] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
[video360_publisher-3] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
[video360_publisher-3] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
[video360_publisher-4] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
[video360_publisher-4] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
[video360_publisher-4] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
[video360_publisher-4] (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)

In the argus log:

Jan 30 13:32:39 nvargus-daemon[1280]: Module_id 30 Severity 2 : (fusa) Error: ResourceAlreadyInUse All captures are already pending, no idle captures available in:/capture/src/fusaViHandler.cpp 627
Jan 30 13:32:39 nvargus-daemon[1280]: Module_id 30 Severity 2 : (fusa) Error: ResourceAlreadyInUse  propagating from:/capture/src/fusaViHandler.cpp 469
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function startCaptureInternal(), line 816)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCapture(), line 530)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 477)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1291)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1122)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error ResourceAlreadyInUse:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error ResourceAlreadyInUse: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 644)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 426)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 906)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/SensorCaptureStage.cpp, function doHandleRequest(), line 86)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 906)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/SensorCaptureStage.cpp, function doHandleRequest(), line 86)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 906)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequest(), line 143)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 906)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequest(), line 143)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 906)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequest(), line 143)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 906)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/SensorCaptureStage.cpp, function doHandleRequest(), line 86)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 906)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/SensorCaptureStage.cpp, function doHandleRequest(), line 86)
Jan 30 13:32:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Jan 30 13:32:48 nvargus-daemon[1280]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Jan 30 13:32:48 nvargus-daemon[1280]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Jan 30 13:32:48 nvargus-daemon[1280]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Jan 30 13:32:48 nvargus-daemon[1280]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Jan 30 13:32:48 nvargus-daemon[1280]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Jan 30 13:32:48 nvargus-daemon[1280]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Jan 30 13:32:48 nvargus-daemon[1280]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Jan 30 13:32:48 nvargus-daemon[1280]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Jan 30 13:33:39 nvargus-daemon[1280]: SCF: Error InvalidState: 3 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
Jan 30 13:33:39 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 49)
Jan 30 13:33:40 nvargus-daemon[1280]: SCF: Error InvalidState: 4 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
Jan 30 13:33:40 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 49)
Jan 30 13:33:40 nvargus-daemon[1280]: SCF: Error InvalidState: 4 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
Jan 30 13:33:40 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 49)
Jan 30 13:33:40 nvargus-daemon[1280]: SCF: Error InvalidState: 4 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
Jan 30 13:33:40 nvargus-daemon[1280]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 49)

It’s very frustrating; it’s completely impossible to make anything work in this JP, at least with these cameras. I cannot upgrade to a higher JP since my carrier board manufacturer only supports up to JP5.1.1.

I would appreciate any kind of suggestion. Thank you in advance,
Angel

hello angel.corrales,

please note that we usually don’t support back-porting with previous release version.

anyways,
I’ve based-on JP-5.1.1/r35.3.1 to apply some known bug fixes, (see-also Topic 243051, Topic 268519)
I’ve also do some timeout values extension since you’ve lots of socket timeout errors reported by Argus.

please give it a try with this attachment, Topic278999_Jan31.7z (2.3 MB)
you may overwrite those pre-built binaries, and reboot the system for confirmation,
thanks

Hi JerryChang

Thank you for your reply. After applying the changes it seems that argus does not even detect the cameras.

Running a simple pipeline: gst-launch-1.0 nvarguscamerasrc sensor-id=1 ! nvvidconv ! xvimagesink:

Setting pipeline to PAUSED ...
0:00:01.061669563  4780 0xaaab17d7f800 FIXME                default gstutils.c:3980:gst_pad_create_stream_id_internal:<nvarguscamerasrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:751 No cameras available
0:00:01.072504523  4780 0xaaab17d82a40 WARN                basesink gstbasesink.c:1209:gst_base_sink_query_latency:<xvimagesink0> warning: Pipeline construction is invalid, please add queues.
0:00:01.072592268  4780 0xaaab17d82a40 WARN                basesink gstbasesink.c:1209:gst_base_sink_query_latency:<xvimagesink0> warning: Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Pipeline construction is invalid, please add queues.
Additional debug info:
gstbasesink.c(1209): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.
Got EOS from element "pipeline0".
Execution ended after 0:00:00.011253173
Setting pipeline to NULL ...
Freeing pipeline ...

Argus log:

Jan 31 08:30:04 nvargus-daemon[4777]: === NVIDIA Libargus Camera Service (0.99.33)=== Listening for connections...=== gst-launch-1.0[4780]: Connection established (FFFFA7E12900)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Jan 31 08:30:04 nvargus-daemon[4777]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Jan 31 08:30:04 nvargus-daemon[4777]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
Jan 31 08:30:04 nvargus-daemon[4777]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
Jan 31 08:30:04 nvargus-daemon[4777]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jan 31 08:30:04 nvargus-daemon[4777]: NvPclHwGetModuleList: No module data found
Jan 31 08:30:04 nvargus-daemon[4777]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jan 31 08:30:04 nvargus-daemon[4777]: NvPclHwGetModuleList: No module data found
Jan 31 08:30:04 nvargus-daemon[4777]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jan 31 08:30:04 nvargus-daemon[4777]: NvPclHwGetModuleList: No module data found
Jan 31 08:30:04 nvargus-daemon[4777]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jan 31 08:30:04 nvargus-daemon[4777]: NvPclHwGetModuleList: No module data found
Jan 31 08:30:04 nvargus-daemon[4777]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Jan 31 08:30:04 nvargus-daemon[4777]: ---- imager: No override file found. ----
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter: Unknown sensor pixel type (in src/common/Utils.cpp, function translateColorFormat(), line 432)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceSensor.cpp, function open(), line 216)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 717)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 453)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 333)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 505)
Jan 31 08:30:04 nvargus-daemon[4777]: ---- imager: No override file found. ----
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter: Unknown sensor pixel type (in src/common/Utils.cpp, function translateColorFormat(), line 432)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceSensor.cpp, function open(), line 216)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 717)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 453)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 333)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 505)
Jan 31 08:30:04 nvargus-daemon[4777]: ---- imager: No override file found. ----
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter: Unknown sensor pixel type (in src/common/Utils.cpp, function translateColorFormat(), line 432)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceSensor.cpp, function open(), line 216)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 717)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 453)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 333)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 505)
Jan 31 08:30:04 nvargus-daemon[4777]: ---- imager: No override file found. ----
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter: Unknown sensor pixel type (in src/common/Utils.cpp, function translateColorFormat(), line 432)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceSensor.cpp, function open(), line 216)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 717)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 453)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 333)
Jan 31 08:30:04 nvargus-daemon[4777]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 505)

I’m afraid we are going to continue developing on JP4.6.1 and we definitely discard the upgrade, we have been too long at this point. It’s a pity not to be able to take advantage of all the new features of Deepstream and a more updated l4t, but we have no choice.

hello angel.corrales,

there’re some dependency, this failure might due to my unclean code-base, I’ve check-out entire tree to r35.3.1 for builting those binary files,
please kindly test again for verification. for instance, Topic278999_Jan31_v2.7z (2.3 MB)

Hi JerryChang.

I have just checked that with this new version the cameras are detected. We are going to do some tests during the day, I will get back to you when we have some results.

Thank you very much!

Hello JerryChang

We have been able to verify that the problem described in #Post 14 has apparently been solved. Gscam node has been running without problem for at least 3h until we stopped it manually (previously it was running for about 10/20 minutes).

Regarding the problem described originally in this thread, we are still the same. When we perform more complex pipelines involving elements such as nvinfer, nvtracker and several nvvideoconvert (among others) we get the mentioned (Argus) Error OverFlow: Too many pending events, ignoring new events.

We are not able to stably run our production pipelines that were running smoothly in JP4.6.1, so we cannot develop on this new JP.

Thank you very much for all your support and help, we appreciate it.

Best regards,
Ángel.

I see, thanks for sharing your test results.
let me arrange resources to check what we can improve in the future release version.

1 Like

some questions…

  1. is there any network difference between JP-4 and JP-5? since you mentioned the use-case were running smoothly in JP-4.6.1, it might be performance bottleneck/regression of nvinfer.
  2. is your pipeline to mux several camera frames as nvinfer input? (only single nvinfer were used)
    if yes, please give it a try for adding this setting, batched-push-timeout=30000 to configure a larger timeout to wait after the first buffer is available.
  3. is it possible for testing with lower camera fps?

I understand you might not develop on JP-5 at the moment,
please file another new discussion thread if you need supports in the future.

Hi Jerry,

  1. The only difference from the nvinfer is that the custom-lib-path has been compiled for CUDA 11.2 instead of 10.4. The same custom .onnx model is being used.

  2. Yes, different cameras are muxed for a batch 4 nvinfer.

  3. Yes, we could do the tests, but currently we don’t have any NXs upgraded to JP 5.1.1 (we have flashed them back to the previous version), when we get one back we will do it.

update

let me have summarization here.
here’re patches (pre-built updates) in comment #17 to resolve multi-cam Argus timeout for l4t-r35.3.1
the test result (#19) shows it could have 4-cam preview stream works normally for 3 hours.

however,
there’s an issue with more complex pipelines which has involving elements such as nvinfer…etc
my suggestion is closing current discussion thread, and let’s have new topic for following up.
thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.