"ARGUS" timeout problems when shutting down a gstreamer pipeline with more than one camera using nvarguscamerasrc

mtrank · January 15, 2021, 9:24pm

I have an application using Gstreamer and “D3 Engineering’s” camera modules on a Jetson TX2 development board. Everything was working nicely with Jetpack 4.3 and D3’s BSP v 3.0.1.

But we now have a need for some features that D3 has introduced into their version 4.0 BSP, which works with Jetpack 4.4. After upgrading the board, I now have problems when shutting down the GStreamer pipeline, and it is evident that the problem occurs when Gstreamer tells argus to shut down more than one camera. (This problem does not occur when only one camera is in the pipeline.)

When this occurs, I need to do a “systemctl restart nvargus-daemon” in order to get the cameras to work again.

The D3 guys tell me that this problem is located in something that changed between JetPack 4.3 and JetPack 4.4, rather than anything that changed betwen their old BSP and their new one.

The problem that I am having also exists and map be reproduced with a gstreamer “gst-launch-1.0” command line pipeline that invoke two cameras, like this:

mtrank@JetsonTx2:~$ gst-launch-1.0 nvarguscamerasrc sensor-mode=0 sensor-id=1 ! ‘video/x-raw(memory:NVMM),width=(int)1280,height=(int)1080,format=(string)NV12,framerate=(fraction)30/1’ ! fakesink nvarguscamerasrc sensor-mode=0 sensor-id=0 ! ‘video/x-raw(memory:NVMM),width=(int)1280,height=(int)1080,format=(string)NV12,framerate=(fraction)30/1’ ! nvoverlaysink -e

Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1280 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 8.000000; Exposure Range min 20000, max 33698000;

CONSUMER: Waiting until producer is connected…
GST_ARGUS: 1280 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 8.000000; Exposure Range min 640000, max 32640000;

GST_ARGUS: Running with following settings:
Camera index = 1
Camera mode = 0
Output Stream W = 1280 H = 1080
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1280 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 8.000000; Exposure Range min 20000, max 33698000;

GST_ARGUS: 1280 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 8.000000; Exposure Range min 640000, max 32640000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 1280 H = 1080
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
^Chandling interrupt.
Interrupt: Stopping pipeline …
EOS on shutdown enabled – Forcing EOS on the pipeline
Waiting for EOS…
Got EOS from element “pipeline0”.
EOS received - stopping pipeline…
Execution ended after 0:00:06.767941519
Setting pipeline to PAUSED …
Setting pipeline to READY …
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
GST_ARGUS: Cleaning up
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
CONSUMER: Done Success
GST_ARGUS: Done Success
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Setting pipeline to NULL …
Freeing pipeline …

Help!

D3_growe · January 20, 2021, 12:11am

Hi Michael,

What I think we need to do here is back the truck up. ;) I’m kidding, mostly.

As you know there have been numerous emails exchanged between you and D3. There are many details that have become unclear. The best path to a resolution is to return a known starting point. Therefore, I request that you start with a freshly flashed system with the standard D3 configuration. From there we can verify that the image quality is at the level we expect for the rugged ov10640 camera module. From there we can start looking at your more specific issue.

Please do the following:

Follow the instructions in the release notes to install D3 BSP 4.0.0 which is, roughly the following:

Install Jetpack 4.4 (not 4.4.1)
Install the D3 BSP version 4.0.0-pre.1
Install the D3 BSP utils version 4.0.0-pre.1
Run d3-install-dtb to install the correct DTB for your system
Edit /boot/extlinux/extlinux.conf to add active_overlays=ov10640_0,ov10640_1,ov10640_2,ov10640_3 to the APPEND line. Please reference the documentation in the release notes.

After re-installing the BSP, if you are still having trouble, would you please (1) provide video or still captures, and (2) the exact commands you ran to get those captures?

Thanks,
Greg

Radar343 · January 21, 2021, 3:53pm

I am experiencing the same same problem. I have just reflashed to jetpack 4.4 and now scripts that used to work no longer work. I am also using two cameras and I am getting the same errors reported. I am using IMX265 cameras from Leopard imaging though. Theis is the output I am seeing:-
user@XavierCamera:/media/SSD/DualCameraTest$ ./DualCameraCapture265.sh
nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED …
Opening in BLOCKING MODE
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
Redistribute latency…
NvMMLiteOpen : Block : BlockType = 8
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8
nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED …
Opening in BLOCKING MODE
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 8
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8
Redistribute latency…
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 2048 x 1554 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 48.000000; Exposure Range min 30000, max 660000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 2048 H = 1554
seconds to Run = 900
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 900 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
NVMEDIA: H265 : Profile : 1
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 2048 x 1554 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 48.000000; Exposure Range min 30000, max 660000000;

GST_ARGUS: Running with following settings:
Camera index = 1
Camera mode = 0
Output Stream W = 2048 H = 1554
seconds to Run = 900
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 900 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
NVMEDIA: H265 : Profile : 1
GST_ARGUS: Cleaning up
GST_ARGUS: Cleaning up
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
CONSUMER: Done Success
GST_ARGUS: Done Success
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
CONSUMER: Done Success
GST_ARGUS: Done Success
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Got EOS from element “pipeline0”.
Execution ended after 0:17:02.657457174
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Got EOS from element “pipeline0”.
Execution ended after 0:17:15.876951467
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error InvalidState: Argus client is exiting with 3 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error InvalidState: Argus client is exiting with 3 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)

D3_growe · February 18, 2021, 2:03pm

To provide an update to the forum: this issue hasn’t been solved yet. Michael and I have been working offline to find the root cause. We will communicate back here when we have better information to report.

terrysu50z · March 5, 2021, 5:06pm

any more info? I am having a similar problem on r32.3.1

Terry

mtrank · March 5, 2021, 5:41pm

Hello terrysu50z:

My problem did not exist when I was using r32.3.1 (which I believe is what gets installed with Jetpack 4.3, right?)

The problem was introduced when I upgraded to Jetpack 4.4, r32.4.3.

You may want to try out the pipeline in my message in order to see if you are really seeing the same problem.

Topic		Replies	Views
Argus errors when restarting pipeline - Jetson AGX Jetson AGX Xavier camera	10	57	February 7, 2025
Argus stability on Orin NX Jetson Orin NX camera , gstreamer	28	1354	August 28, 2023
Gstreamer pipleine is not working second time for imx219 Jetson Orin Nano camera , gstreamer	50	179	February 6, 2025
Argus API is not stable in JetPack 5 and gstreamer doesnt work Jetson AGX Xavier camera , gstreamer	11	1000	May 30, 2023
Some basic problems with nvarguscamerasrc plugin for gstreamer Jetson AGX Xavier camera	15	1882	January 26, 2024
Deepstream crashes with camera source, but works fine with nvgstcapture DeepStream SDK	6	720	October 12, 2021
JetPack4.5 using gst to open raw camera "nvarguscamerasrc0 reported: TIMEOUT " Jetson AGX Xavier gstreamer	8	1564	October 18, 2021
Errors starting 7 nvarguscamersrc pipeline in R32.7.1(Jetpack 4.6.1) Jetson AGX Xavier gstreamer , sony-imx390 , deepstream	31	926	August 17, 2023
Gstreamer command failed after long run Jetson Orin Nano camera	17	112	October 18, 2024
(Argus) Error Timeout: gstreamer Jetson AGX Xavier camera , gstreamer	2	874	May 2, 2022

"ARGUS" timeout problems when shutting down a gstreamer pipeline with more than one camera using nvarguscamerasrc

Related topics