Gstreamer command failed after long run

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

I have 2 gstreamer commands in 2 system services that works fine at the beginning for serveral hours, but it always crash loopback after 4-7 hours, my commands are:

gst-launch-1.0 nvarguscamerasrc sensor-id="$vis" ! \
  'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=30/1' ! \
  nvvidconv flip-method=0 ! \
  'video/x-raw, width=1280, height=720, format=BGRx' ! \
  queue ! videoconvert ! \
  'video/x-raw, format=BGR' ! \
  tee name=t ! \
  queue ! v4l2sink device=/dev/video6 t. ! \
  queue ! v4l2sink device=/dev/video7

gst-launch-1.0 v4l2src device="/dev/video$ir" ! videoconvert ! tee name=t ! queue ! v4l2sink device=/dev/video8 t. ! queue ! v4l2sink device=/dev/video9

/dev/video6,7,8,9 are virtual videos created by v4l2loopback, errors are:

For first service:

Oct 15 21:35:43 tegra-ubuntu nvargus-daemon[1103]: === gst-launch-1.0[91674]: CameraProvider destroyed (0xffff50000df0)=== gst-launch-1.0[91674]: Connection closed (FFFF551EC900)=== gst-launch-1.0[91674]: Connection cleaned up (FFFF551EC900)=== gst-launch-1.0[91696]: Connection established (FFFF551EC900)=== gst-launch-1.0[91696]: CameraProvider initialized (0xffff50000c20)(Argus) Error AlreadyAllocated: Device 0 (of 1) is in use (in src/api/CameraProviderImpl.cpp, function createCaptureSessionInternal(), line 272)
Oct 15 21:35:43 tegra-ubuntu start-virtual-video[91696]: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:762 Failed to create CaptureSession

For second service:

163]: Setting pipeline to PAUSED ...
163]: Pipeline is live and does not need PREROLL ...
163]: Setting pipeline to PLAYING ...
163]: New clock: GstSystemClock
163]: ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
163]: Additional debug info:
163]: gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
163]: streaming stopped, reason not-negotiated (-4)
163]: Execution ended after 0:00:00.000900954
163]: Setting pipeline to NULL ...
163]: Freeing pipeline ...
service: Main process exited, code=exited, status=1/FAILURE

FYI, I use those 4 virtual devices in docker container with privileged mode, and those commands works fine in Jetson Nano

3

What’s the BSP version?

cat /etc/nv_tegra_release
4

Hi it’s # R35 (release), REVISION: 3.1, GCID: 32827747, BOARD: t186ref, EABI: aarch64, DATE: Sun Mar 19 15:19:21 UTC 2023

5

Apply the patch for argus from below link.

https://elinux.org/Jetson/L4T/r35.3.x_patches

6

Okay I will apply this patch first

7

Hi it still happens, the full gstreamer trace log is too big, I will collect debug log of 2 commands and upload it later.

8

We found some suspicious log

Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473161116 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin90>#033[00m g_dir_open(/root/.lv2) failed: Error opening directory <E2><80><9C>/root/.lv2<E2><80><9D>: No such file or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473190877 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin90>#033[00m g_dir_open(/usr/lib/lv2) failed: Error opening directory <E2><80><9C>/usr/lib/lv2<E2><80><9D>: No such file or director
y
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473206909 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin90>#033[00m g_dir_open(/usr/lib64/lv2) failed: Error opening directory <E2><80><9C>/usr/lib64/lv2<E2><80><9D>: No such file or dire
ctory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473221885 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin90>#033[00m g_dir_open(/usr/local/lib/lv2) failed: Error opening directory <E2><80><9C>/usr/local/lib/lv2<E2><80><9D>: No such file
 or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473236254 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin90>#033[00m g_dir_open(/usr/local/lib64/lv2) failed: Error opening directory <E2><80><9C>/usr/local/lib64/lv2<E2><80><9D>: No such 
file or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473251134 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin90>#033[00m g_dir_open(/usr/lib/aarch64-linux-gnu/lv2) failed: Error opening directory <E2><80><9C>/usr/lib/aarch64-linux-gnu/lv2
<E2><80><9D>: No such file or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473415107 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin119>#033[00m g_dir_open(/root/.frei0r-1/lib) failed: Error opening directory <E2><80><9C>/root/.frei0r-1/lib<E2><80><9D>: No such f
ile or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473431428 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin119>#033[00m g_dir_open(/usr/lib/aarch64-linux-gnu/frei0r-1) failed: Error opening directory <E2><80><9C>/usr/lib/aarch64-linux-gnu
/frei0r-1<E2><80><9D>: No such file or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473444196 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin119>#033[00m g_dir_open(/usr/lib/frei0r-1) failed: Error opening directory <E2><80><9C>/usr/lib/frei0r-1<E2><80><9D>: No such file 
or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473456997 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin119>#033[00m g_dir_open(/usr/local/lib/frei0r-1) failed: Error opening directory <E2><80><9C>/usr/local/lib/frei0r-1<E2><80><9D>: No such file or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473469477 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin119>#033[00m g_dir_open(/usr/lib32/frei0r-1) failed: Error opening directory <E2><80><9C>/usr/lib32/frei0r-1<E2><80><9D>: No such file or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473480965 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin119>#033[00m g_dir_open(/usr/local/lib32/frei0r-1) failed: Error opening directory <E2><80><9C>/usr/local/lib32/frei0r-1<E2><80><9D<9D>: No such file or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473492550 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin119>#033[00m g_dir_open(/usr/lib64/frei0r-1) failed: Error opening directory <E2><80><9C>/usr/lib64/frei0r-1<E2><80><9D>: No such file or directory
Oct 17 06:28:47 tegra-ubuntu start-virtual-video[145304]: 0:00:00.473503910 #033[333m145304#033[00m 0xaaaae118fe00 #033[37mDEBUG  #033[00m #033[00;01;36m  GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin119>#033[00m g_dir_open(/usr/local/lib64/frei0r-1) failed: Error opening directory <E2><80><9C>/usr/local/lib64/frei0r-1<E2><80><9D<9D>: No such file or directory

the key part is GST_PLUGIN_LOADING gstplugin.c:1611:gst_plugin_ext_dep_scan_dir_and_match_names:<plugin259> g_dir_open(/dev/v4l2) failed: Error opening directory /dev/v4l2: No such file or directory

9

Full trace log:
errors.log (1.9 MB)

10

These message looks like in gstreamer framework. I don’t have idea for this error.
Please check the daemon log if any error.

11

Hi,
Sadly nvargus daemon log is mixed in the syslog, but we do find that we are very similar to this problem Gstreamer randomly dies when running with v4l2loopback - Jetson & Embedded Systems / Jetson Nano - NVIDIA Developer Forums

12

Hi,
I greped the nvargus log.
it’s only one line
Oct 17 06:28:47 tegra-ubuntu nvargus-daemon[1094]: === gst-launch-1.0[145190]: CameraProvider destroyed (0xffff68000c20)=== gst-launch-1.0[145190]: Connection closed (FFFF711EC900)=== gst-launch-1.0[145190]: Connection cleaned up (FFFF711EC900)=== gst-launch-1.0[145304]: Connection established (FFFF711EC900)=== gst-launch-1.0[145304]: CameraProvider initialized (0xffff68000df0)(Argus) Error AlreadyAllocated: Device 0 (of 1) is in use (in src/api/CameraProviderImpl.cpp, function createCaptureSessionInternal(), line 272)

13

You can run the argus-daemon manually and check the log in the console running the argus-daemon.

sudo su
pkill nvargus-daemon
/usr/sbin/nvargus-daemon
lanuch camera APP from another console
14

Hi,
Following is some more argus log

Oct 18 02:00:05 tegra-ubuntu systemd[1]: Started Oiler vis virtual video services.
Oct 18 02:00:05 tegra-ubuntu start-virtual-video[490065]: Found vis device index 0 with imx keyword
Oct 18 02:00:05 tegra-ubuntu start-virtual-video[490075]: Setting pipeline to PAUSED ...
Oct 18 02:00:05 tegra-ubuntu start-virtual-video[490075]: Pipeline is live and does not need PREROLL ...Oct 18 02:00:05 tegra-ubuntu start-virtual-video[490075]: Setting pipeline to PLAYING ...
Oct 18 02:00:05 tegra-ubuntu start-virtual-video[490075]: New clock: GstSystemClock
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: TIMEOUT
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: Additional debug info:
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: Argus Error Status
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: Creating output stream
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: CONSUMER: Waiting until producer is connected...
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: Available Sensor modes :
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: 3280 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: 3280 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: Running with following settings:
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]:    Camera index = 0
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]:    Camera mode  = 4
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]:    Output Stream W = 1280 H = 720
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]:    seconds to Run    = 0
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]:    Frame Rate = 59.999999
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: Setup Complete, Starting captures for 0 seconds
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: Starting repeat capture requests.
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: CONSUMER: Producer has connected; continuing.
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: CONSUMER: ERROR OCCURRED
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: Execution ended after 2:03:36.898977394
Oct 18 04:03:42 tegra-ubuntu start-virtual-video[490075]: Setting pipeline to NULL ...
Oct 18 04:04:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
Oct 18 04:04:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Oct 18 04:04:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
Oct 18 04:04:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Oct 18 04:05:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
Oct 18 04:05:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Oct 18 04:05:42 tegra-ubuntu start-virtual-video[490075]: GST_ARGUS: Cleaning up
Oct 18 04:05:42 tegra-ubuntu start-virtual-video[490075]: Freeing pipeline ...
Oct 18 04:06:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
Oct 18 04:06:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Oct 18 04:06:42 tegra-ubuntu start-virtual-video[490075]: (Argus) Error InvalidState: Argus client is exiting with 4 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
Oct 18 04:06:42 tegra-ubuntu systemd[1]: virtual-vis.service: Main process exited, code=exited, status=1/FAILURE
Oct 18 04:06:42 tegra-ubuntu systemd[1]: virtual-vis.service: Failed with result 'exit-code'.
Oct 18 04:06:44 tegra-ubuntu systemd[1]: virtual-vis.service: Scheduled restart job, restart counter is at 1.
15

Have you apply below?

[Argus] Extend timeout values to avoid socket timeout errors reported by Argus

16

No I haven’t, the error above may due to a gstreamer pipeline restart, but what I am worried about is that it randomly dies after long run, and it may not be this reason, I will collect more accurate logs.

17

Hi,
Following log appears after long run with gstreamer and v4l2loopback

012]: Found ir device index 1 with AT600 keyword
023]: Setting pipeline to PAUSED ...
023]: Pipeline is live and does not need PREROLL ...
023]: Setting pipeline to PLAYING ...
023]: New clock: GstSystemClock
023]: ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
023]: Additional debug info:
023]: gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
023]: streaming stopped, reason not-negotiated (-4)
023]: Execution ended after 0:00:00.000830497
023]: Setting pipeline to NULL ...
023]: Freeing pipeline ...
service: Main process exited, code=exited, status=1/FAILURE

I think this is the key issue, would you mind helping me figuring it out?

18

I think you need to check the nvargus-daemon log to figure out the root cause.

Thanks

20

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.