I’m working on an application that utilizes GStreamer pipelines to handle multiple camera sources. The current setup involves four camera inputs, which are muxed with audio into 4 separate file sinks, while also being streamed via UDP for live viewing. While everything works smoothly for short recordings, I run into issues when attempting longer recordings (typically 4 hours or more, but this has also occurred with shorter durations). The nvargus-daemon seems to crash during these sessions.
Here’s a brief overview of the pipelines I’m using:
Primary pipeline for capturing
Gst.parse_launch(f"""
nvarguscamerasrc sensor_id=1 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t1
t1. ! queue ! nvv4l2h264enc ! h264parse ! queue ! mux1.
t1. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5000
nvarguscamerasrc sensor_id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,interlace-mode=interlaced ! tee name=t2
t2. ! queue ! nvv4l2h264enc ! h264parse ! queue ! mux2.
t2. ! queue max-size-buffers=2 ! nvvidconv ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5001
nvarguscamerasrc sensor_id=2 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t3
t3. ! queue ! nvv4l2h264enc ! h264parse ! queue ! mux3.
t3. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5002
nvarguscamerasrc sensor_id=3 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t4
t4. ! queue ! nvv4l2h264enc ! h264parse ! queue ! mux4.
t4. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5003
alsasrc device=hw:2,0 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! volume name=volume1 ! mix1.
alsasrc device=hw:3,0 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! volume name=volume2 ! mix1.
audiomixer name=mix1 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! tee name=t
t. ! queue ! mux1.
t. ! queue ! mux2.
t. ! queue ! mux3.
t. ! queue ! mux4.
qtmux name=mux1 ! filesink location=/media/{CAM1_output_file}
qtmux name=mux2 ! filesink location=/media/{CAM2_output_file}
qtmux name=mux3 ! filesink location=/media/{CAM3_output_file}
qtmux name=mux4 ! filesink location=/media/{CAM4_output_file}
""")
Pipeline for live view on Display 0
Gst.parse_launch(f"""
udpsrc port={current_port_cam1} caps="application/x-rtp, media=video" name=udpsrc1 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! nvoverlaysink display-id=0 name=overlay1 overlay=1 overlay-depth=1 sync=false
""")
Pipeline for live view on Display 1
Gst.parse_launch(f"""
udpsrc port={current_port_cam2} caps="application/x-rtp, media=video" name=udpsrc2 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! nvoverlaysink display-id=1 name=overlay2 overlay=1 overlay-depth=1 sync=false
""")
Gstreamer does run and capture frames into an mp4 but eventually leads to the following response. The program will hang until ctrl+C.
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
NvMMLiteBlockCreate : Block : BlockType = 4
NvMMLiteBlockCreate : Block : BlockType = 4
NvMMLiteBlockCreate : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
Opening in BLOCKING MODE
Recording started. Press 'Ctrl+C' to stop.
Enter command: NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: Running with following settings:
Camera index = 1
Camera mode = 4
Output Stream W = 1280 H = 720
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: Running with following settings:
Camera index = 3
Camera mode = 4
Output Stream W = 1280 H = 720
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: Running with following settings:
Camera index = 2
Camera mode = 4
Output Stream W = 1280 H = 720
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 2
Output Stream W = 1920 H = 1080
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
a
Switched Camera 1 to port 5002.
Live Display Swapped
Enter command: areference in DPB was never decoded
a
Invalid command or not in the correct mode.
Enter command: a
Switched Camera 1 to port 5003.
Live Display Swapped
Enter command: reference in DPB was never decoded
CONSUMER: ERROR OCCURRED
CONSUMER: ERROR OCCURRED
CONSUMER: ERROR OCCURRED
(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
CONSUMER: Done Success
^CStopping the pipeline...
Error received: NvArgusCameraSrc: DISCONNECTED (8), Argus Error Status
GST_ARGUS: Cleaning up
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Segmentation fault (core dumped)
I believe this error is only caused from nvargus-daemon stopping which has been my main way of reproducing the error above. What is stopping nvargus-daemon I am unsure of. To investigate further, I checked the logs using journalctl, syslog, and dmesg for any relevant messages and at best :
nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
nvargus-daemon.service: Failed with result 'signal'.
nvargus-daemon.service: Service hold-off time over, scheduling restart.
nvargus-daemon.service: Scheduled restart job, restart counter is at 6.
Stopped Argus daemon.
Started Argus daemon.
My goal is to achieve stable recordings lasting at least 8+ hours without issues. I am unsure why or where else to look for errors. I would hope to be able to stop nvargus-daemon from crashing or if it does crash, it can restart and continue the previously opened pipeline. I have two Jetson Tx2 which this occurs on both.