Nvargus-daemon: Too many pending events error?

seerich-edge · April 25, 2025, 8:19pm

I am using Jetpack 4.6.6 [L4T 32.7.6] and the gstreamer element nvarguscamersrc. My current setup has as custom carrier board with 2 CSI IMX219 cameras connected.

Most of the time, things work great, but every once in a while, the video feed for one of the cameras will lock up and and I see hundreds of the following error in the journalctl logs:

nvargus-daemon[27554]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)

This error seems to happen randomly but when it does occur, it is usually shortly after starting the gstreamer pipeline (i.e. setting it to playing state).

I have a couple of questions:

What is causing this error and how can I further debug it?
What is nvargus-daemon and what does it do?
Is the source code for this application public? I couldn’t find any sources in the BSP download.
Do I need this at all if only a single process is interacting with the cameras? If not, what alternatives are available? I am open to writing my own capture source if necessary but I would still need the frames to flow downstream to other gstreamer elements using (memory:NVMM).

JerryChang · April 28, 2025, 2:57am

hello seerich-edge,

is it tested with customize application? can you reproduce this on Jetson Nano developer kit?
is it only happened with dual camera use-case?
for example, please try below simple gst pipeline to launch single camera to reproduce the failure,
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

seerich-edge · April 29, 2025, 7:35pm

Hi @JerryChang! Thank you for the help!

I took your pipeline and wrote a quick and dirty bash script around it allowing me to run a pipeline in a loop. My goal is to be able to reproduce it on my carrier board and then test on dev kit. However, I still can’t reproduce it on my hardware reliably and instead am facing other errors/issues.

#!/bin/bash

# Validate arguments
if [ "$#" -ne 2 ]; then
  echo "Usage: $0 <number_of_iterations> <time_per_iteration_in_seconds>"
  exit 1
fi

# Assign arguments to variables
num_iterations=$1
time_limit=$2

# Validate arguments are integers
if ! [[ "$num_iterations" =~ ^[0-9]+$ ]] || ! [[ "$time_limit" =~ ^[0-9]+$ ]]; then
  echo "Error: Both arguments must be positive integers."
  exit 1
fi

# Loop for the specified number of iterations
for ((i=1; i<=num_iterations; i++)); do
    echo "Starting iteration $i..."

    # Run the GStreamer pipeline in the background
    # PIPELINE 1 - Simple w/ Single stream - Works 10x60sec
    # gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v &
    # Pipeline 2 - Dual stream recording to file
    gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 name=cam1 exposuretimerange='5000000 20000000' ! 'video/x-raw(memory:NVMM),width=(int)1632,height=(int)1232,format=(string)NV12,framerate=(fraction)30/1' ! videorate ! 'video/x-raw(memory:NVMM),framerate=30/1' ! queue ! nvvidconv flip-method=02 ! "video/x-raw(memory:NVMM),format=(string)NV12" ! queue ! nvv4l2h264enc maxperf-enable=true bitrate=20000000 ! h264parse ! queue ! qtmux ! filesink location=test1.mp4 nvarguscamerasrc sensor-id=1 name=cam2 exposuretimerange='5000000 20000000' ! 'video/x-raw(memory:NVMM),width=(int)1632,height=(int)1232,format=(string)NV12,framerate=(fraction)30/1' ! videorate ! 'video/x-raw(memory:NVMM),framerate=30/1' ! queue ! nvvidconv flip-method=02 ! "video/x-raw(memory:NVMM),format=(string)NV12" ! queue ! nvv4l2h264enc maxperf-enable=true bitrate=20000000 ! h264parse ! queue ! qtmux ! filesink location=test2.mp4 &
    gst_pid=$!

    # Sleep for the specified time
    for ((j=1; j <= time_limit; j++)); do
        sleep 1
        echo "Progress: $j/$time_limit"
    done

    # Kill the GStreamer process
    echo "Stopping iteration $i..."
    kill -SIGINT "$gst_pid" 2> /dev/null

    # Wait for the process to exit
    wait "$gst_pid"
done

echo "All iterations completed."

The stream you sent over is working fine. However, the second stream in the example above (dual cameras recording to file) is producing the following errors:

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:751 Failed to create CaptureSession
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:751 Failed to create CaptureSession
Mar 02 20:34:32 edge nvargus-daemon[5131]: === gst-launch-1.0[29962]: CameraProvider destroyed (0x7f9e05b670)=== gst-launch-1.0[29962]: Connection closed (7EF9FFB1D0)=== gst-launch-1.0[29962]: Connection cleaned up (7EF9FFB1D0)=== gst-launch-1.0[14309]: Connection established (7EF9FFB1D0)=== gst-launch-1.0[14309]: CameraProvider initialized (0x29e788c0)(Argus) Error AlreadyAllocated: Device 0 (of 1) is in use (in src/api/CameraProviderImpl.cpp, function createCaptureSessionInternal(), line 274)
Mar 02 20:34:32 edge nvargus-daemon[5131]: (Argus) Error AlreadyAllocated: Device 0 (of 1) is in use (in src/api/CameraProviderImpl.cpp, function createCaptureSessionInternal(), line 274)

Any idea what could be causing this?

JerryChang · April 30, 2025, 2:02am

hello seerich-edge,

were those cameras work individually?
please refer to developer guide, Applications Using V4L2 IOCTL Directly.
please try V4L2 IOCTL to fetch these two camera streams to verify basic functionality.

seerich-edge · May 2, 2025, 1:55pm

@JerryChang thanks again for the support. I haven’t had the time to dig deep into this issue yet but I’m hoping to next week (please don’t close the issue yet).

These cameras work fine 95% of the time when using nvarguscamerasrc and having both camera streams share the same gstreamer pipeline. That’s why I was trying to keep the scripted pipeline as close to my actual use case as possible. However, my main application doesn’t receive the errors from the previous comment (I will spend time on this next week).

I glanced at the V4L2 IOCTL link you provided and I will test that; however, it does bring up a question:
Is there a way to bypass nvargus-daemon? and why does nvarguscamerasrc have to interface with the daemon at all? What purpose does it provide?

Thank you!

JerryChang · May 5, 2025, 2:17am

hello seerich-edge,

it’s due to multi-process support, by using libnvargus_socketclient.so to link the applications for multi-process use-case.
however, you may test with single-process Argus implementation, which does not require nvargus-daemon service. please try to re-build Argus with.. $ cmake -DDISABLE_MULTIPROCESS=ON

seerich-edge · May 5, 2025, 7:11pm

@JerryChang, hi!

Ah, ok, that makes more sense now. Could you point me to the “Argus” source so I can attempt to rebuild it?

Thank you!

JerryChang · May 6, 2025, 2:33am

hello seerich-edge,

you may download MMAPI package with following, $ sudo apt install nvidia-l4t-jetson-multimedia-api
please see-also /usr/src/jetson_multimedia_api/argus/README.TXT for the build instructions.

seerich-edge · May 12, 2025, 8:31pm

Hi @JerryChang, there is a wealth of information in the directory you linked, thank you!

I downloaded the source code for nvarguscamerasrc plugin and noticed that it was linking nvargus_socketclient and also, noticed that the cmake flag -DDISABLE_MULTIPROCESS from the multimedia sources simply changes the library that is linked from nvargus_socketclient to nvargus.

Should nvarguscamerasrc compile if I change the Makefile to link nvargus instead of nvargus_socketclient?

I tried this but go the following error:

g++ -c gstnvarguscamerasrc.cpp -fPIC `pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0` -I./ -I../ -I/usr/src/jetson_multimedia_api/include/ -I/usr/src/jetson_multimedia_api/argus/samples/utils/ -o gstnvarguscamerasrc.o
gstnvarguscamerasrc.cpp: In member function ‘virtual bool ArgusCamera::StreamConsumer::threadExecute(GstNvArgusCameraSrc*)’:
gstnvarguscamerasrc.cpp:396:27: warning: format not a string literal and no format arguments [-Wformat-security]
  396 |       error = g_error_new (domain, argusStatus, getStatusString(argusStatus));
      |               ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gstnvarguscamerasrc.cpp:416:29: warning: format not a string literal and no format arguments [-Wformat-security]
  416 |         error = g_error_new (domain, frame_status, getStatusString(frame_status));
      |                 ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gstnvarguscamerasrc.cpp:646:22: error: cannot convert ‘NvBufferColorFormat’ to ‘NvBufSurfaceColorFormat’
  646 |                 src->cap_pix_fmt,
      |                 ~~~~~^~~~~~~~~~~
      |                      |
      |                      NvBufferColorFormat
In file included from gstnvarguscamerasrc.cpp:49:
/usr/src/jetson_multimedia_api/include/EGLStream/NV/ImageNativeBuffer.h:90:56: note:   initializing argument 2 of ‘virtual int EGLStream::NV::IImageNativeBuffer::createNvBuffer(Argus::Size2D<unsigned int>, NvBufSurfaceColorFormat, NvBufSurfaceLayout, EGLStream::NV::Rotation, Argus::Status*) const’
   90 |                                NvBufSurfaceColorFormat format,
      |                                ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
make: *** [Makefile:63: gstnvarguscamerasrc.o] Error 1

JerryChang · May 13, 2025, 2:35am

hello seerich-edge,

I don’t see such build error, please refer to gst-nvarguscamera/README.txt to install gstreamer related packages on target.
you may revise Makefile to replace the libnvargus_socketclient.so with libnvargus.so.
for instance,

diff --git a/gst-nvarguscamera/Makefile b/gst-nvarguscamera/Makefile
@@ -31,7 +31,8 @@ CC := g++
 CFLAGS:=
-LIBS:= -lnvbufsurface -lnvbufsurftransform -lnvdsbufferpool -lnvargus_socketclient -lpthread
+LIBS:= -lnvbufsurface -lnvbufsurftransform -lnvdsbufferpool -lnvargus -lpthread

then.. to build the gstnvarguscamerasrc

$ make
g++ -c gstnvarguscamerasrc.cpp -fPIC `pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0` -I./ -I../ -I/usr/src/jetson_multimedia_api/include/ -I/usr/src/jetson_multimedia_api/argus/samples/utils/ -o gstnvarguscamerasrc.o
g++ -c gstnvarguscamera_utils.cpp -fPIC `pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0` -I./ -I../ -I/usr/src/jetson_multimedia_api/include/ -I/usr/src/jetson_multimedia_api/argus/samples/utils/ -o gstnvarguscamera_utils.o
g++ -shared -o libgstnvarguscamerasrc.so gstnvarguscamerasrc.o gstnvarguscamera_utils.o -lnvbufsurface -lnvbufsurftransform -lnvdsbufferpool -lnvargus -lpthread `pkg-config --libs gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0` -Wl,--no-undefined -L/usr/lib/aarch64-linux-gnu/tegra/ -Wl,-rpath,/usr/lib/aarch64-linux-gnu/tegra/

and, updating the library.

$ sudo make install
cp -vp libgstnvarguscamerasrc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/
'libgstnvarguscamerasrc.so' -> '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so'

you may try to disable nvargus-daemon service for testing it’s running without nvargus_socketclient,
it should works normally with nvargus-daemon disabled.
for example,

$ pkill nvargus-daemon
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v
...
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 17, dropped: 0, current: 32.39, average: 32.39
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 33, dropped: 0, current: 30.04, average: 31.21

seerich-edge · May 20, 2025, 7:21pm

@JerryChang

Thank you for this, I was able to get nvargus-daemon compiled and linked against libnvargus. Unfortunately, I still see the same error.

I’m taking a deeper dive into this week and I’m working on creating a minimal reproducable example. I did have another question though. Is there any environment variable or setting that I can use to tell me which “new” event is being ignored in EventProviderImpl.cpp? Knowing this might help point me in the correct direction.

Thank you!

JerryChang · May 21, 2025, 1:53am

hello seerich-edge,

is this happened with your test application only?
can you reproduce this with gst pipeline with fakesink (preview disabled), or nv3dsink (preview rendering) elements?

seerich-edge · May 21, 2025, 11:56am

Hi @JerryChang, thanks again for the support!

Unfortunately no, this is only happening inside my “production” application. I have been unable to reproduce it reliably with a “test” application or CLI based gstreamer pipeline.

I am currently working on extracting the relevant portion of the application into a standalone python script to see if I can replicate it there.

I hope to get back to you soon with something you can test with. I was asking about the environment variables so I could collect more information from the “production” instances of the application and help guide my search.

JerryChang · May 22, 2025, 8:49am

hello seerich-edge,

all right, please debug into your apps.

there’re event types, EventTypeError, EventTypeCaptureStarted, and EventTypeCaptureComplete.
please see-also Argus::EventType Class Reference.
Too many pending events usually caused by the event queue has reached its max size, it has reported to ignore new events.

seerich-edge · May 28, 2025, 2:14pm

Hi @JerryChang,

Just wanted to provide an update. I found the issue and it was a bug in the application code. Basically, there was a race condition where I would set the pipeline to the playing state and at the same time try to link a new filesink leg to the pipeline. Some how, this worked most of the time but every so often, it would block the buffers flowing through and cause this error message.

Thank you for the support!

JerryChang · May 29, 2025, 3:03am

hello seerich-edge,

thanks for status update, are we able to close this discussion thread?

seerich-edge · May 30, 2025, 9:50am

Yes, we can close this.

system · June 13, 2025, 9:50am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Tx2-4g R32.3.1 nvargus-daemon does not restart 100% of the time Jetson TX2 camera , gstreamer	46	4959	October 18, 2021
JP5.1 nvarguscamera doesn't recover from single NVCSI failure Jetson AGX Xavier camera , nvbugs	51	4595	July 18, 2023
Nvargus-daemon hanging - Jetson Nano 32.7.3 Jetson Nano camera , gstreamer	17	245	August 11, 2024
Nvargus-daemon crashing during recording Jetson TX2 camera , gstreamer	18	241	December 17, 2024
How to make Argus in Jetson 35.2.1 recover after a corrupted frame? Jetson AGX Orin camera , nvbugs	30	2494	July 7, 2023
Video stream crashes in jetpack 5.1.1 xavier nx Jetson Xavier NX camera	15	921	December 25, 2023
Some basic problems with nvarguscamerasrc plugin for gstreamer Jetson AGX Xavier camera	15	1992	January 26, 2024
Argus errors on some boots Jetson TX2 camera , gstreamer	8	971	March 28, 2023
Nvarguscamerasrc error when using nvdewarper plugin DeepStream SDK	5	930	October 12, 2021
R32.3.1 tx2-4g gstreamer app fails to start Jetson TX2 gstreamer	15	1363	October 18, 2021

Nvargus-daemon: Too many pending events error?

Related topics