Gstreamer randomly dies when running with v4l2loopback

Hi All,

I’m using gstreamer to duplicate my camera feed so I can use it in multiple applications (with the help of v4l2loopback).
The idea is to create v4l2 devices on /dev/video1, /dev/video2, etc.

Right now I have 5 applications, so I have 5 /dev/video devices.

Here’s my v4l2loopback configuration:
options v4l2loopback video_nr=1,2,3,4,5 max_buffers=2

And here is my video duplicator pipeline:
gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=3264, height=1848, format=NV12, framerate=(fraction)28/1' ! nvvidconv ! 'video/x-raw, width=2560, height=1440' ! videoconvert ! tee name=t ! queue ! v4l2sink device=/dev/video1 t. ! queue ! v4l2sink device=/dev/video2 t. ! queue ! v4l2sink device=/dev/video3 t. ! queue ! v4l2sink device=/dev/video4 t. ! queue ! v4l2sink device=/dev/video5

As you can see, it uses my IMX219 to record and then downsizes it to 1440p, then passes the video to multiple v4l2sink's, so that different applciations can use it

This system works, but my issue now is that Gstreamer randomly dies, causing all the applications that are using the /dev/video devices to crash.

Here is the output from my gstreamer script:

nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 1 
   Output Stream W = 3264 H = 1848 
   seconds to Run    = 0 
   Frame Rate = 28.000001 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
Execution ended after 0:02:05.247421442
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
client_loop: send disconnect: Broken pipe

As you can see, the error occurs here:
CONSUMER: ERROR OCCURRED ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED

But it never happens immediately, and only seems to happen after my other applications have started using the /dev/videoX devices.

How can I debug these (seemingly) random gstreamer crashes? Is there a better way to duplicate my camera feed into multiple v4l2 devices so that multiple applications can use it?

Here’s some more output from the gstreamer pipeline script:


WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink4: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink4:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink1: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink2: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink2:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink3: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink3:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink1: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink3: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink3:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink2: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink2:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink4: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink4:
There may be a timestamping problem, or this computer is too slow.
(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
CONSUMER: Done Success
Got EOS from element "pipeline0".
Execution ended after 0:03:57.033124880
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Caught SIGSEGV
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)

By looking at the journalctl for nvargus-daemon, I’ve noticed this error before it crashes:

Oct 14 13:51:58 jetson nvargus-daemon[8178]: === gst-launch-1.0[8191]: CameraProvider initialized (0x7f8c91fe60)LSC: LSC surface is not based on full res!
Oct 14 13:53:14 jetson nvargus-daemon[8178]: PowerServiceCore:handleRequests: timePassed = 2798
Oct 14 13:53:18 jetson nvargus-daemon[8178]: SCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 566)
Oct 14 13:53:18 jetson nvargus-daemon[8178]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 997)
Oct 14 13:53:18 jetson nvargus-daemon[8178]: PowerServiceCore:handleRequests: timePassed = 1544
Oct 14 13:53:41 jetson systemd[1]: nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
Oct 14 13:53:41 jetson systemd[1]: nvargus-daemon.service: Failed with result 'signal'.
Oct 14 13:53:41 jetson systemd[1]: nvargus-daemon.service: Service hold-off time over, scheduling restart.
Oct 14 13:53:41 jetson systemd[1]: nvargus-daemon.service: Scheduled restart job, restart counter is at 1.
Oct 14 13:53:41 jetson systemd[1]: Stopped Argus daemon.

Does anyone know what could cause nvargus-daemon to crash with these errors?

Every time I restart the processes, I seem to get different errors from nvargus-daemon.

Here is the latest and greatest in nvargus-daemon errors:

Oct 14 14:40:13 jetson nvargus-daemon[4106]: === gst-launch-1.0[5294]: CameraProvider initialized (0x7f957303e0)LSC: LSC surface is not based on full res!

Oct 14 14:46:14 jetson nvargus-daemon[4106]: W/ libnvphs:socket: Warning (NvPHSSendThroughputHints): too many throughput hints given: buffer full, hints not sent!

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3068769987, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3071129318, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3073423114, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3075913518, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3078141777, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3080435573, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3082729369, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3085088701, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=3087448032, timeout_ms=1000]: queue_or_send() failed

Oct 14 14:46:14 jetson nvargus-daemon[4106]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

Hi,
Please refer to this and give it a try:
Gstreamer v4l2loopback hw mjpeg stream - #5 by Honey_Patouceul

Hi @DaneLLL,

I have done this, and am now running v4l2loopback version 0.12.5, but I still get the same error.

I’ve narrowed the issue down a little bit though. nvargus-daemon runs for some time and then randomly throws the following error:

gst-launch-1.0[5548]: CameraProvider initialized (0x7ef8000be0)(Argus) Error AlreadyAllocated: Device 0 (of 1) is in use (in src/api/CameraProviderImpl.cpp, function createCaptureSessionInternal(), line 274)
Oct 14 15:34:15 jetson nvargus-daemon[5112]: W/ libnvphs:socket: Warning (NvPHSSendThroughputHints): too many throughput hints given: buffer full, hints not sent!
Oct 14 15:34:15 jetson nvargus-daemon[5112]: === gst-launch-1.0[5548]: CameraProvider destroyed

Do you know what could cause this error to occur?