Nvargus-daemon crashing during recording

I’m working on an application that utilizes GStreamer pipelines to handle multiple camera sources. The current setup involves four camera inputs, which are muxed with audio into 4 separate file sinks, while also being streamed via UDP for live viewing. While everything works smoothly for short recordings, I run into issues when attempting longer recordings (typically 4 hours or more, but this has also occurred with shorter durations). The nvargus-daemon seems to crash during these sessions.

Here’s a brief overview of the pipelines I’m using:

Primary pipeline for capturing

Gst.parse_launch(f"""
        nvarguscamerasrc sensor_id=1 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t1
        t1. ! queue ! nvv4l2h264enc ! h264parse ! queue ! mux1.
        t1. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5000 
        nvarguscamerasrc sensor_id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,interlace-mode=interlaced ! tee name=t2
        t2. ! queue ! nvv4l2h264enc ! h264parse ! queue ! mux2.
        t2. ! queue max-size-buffers=2 ! nvvidconv ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5001 
        nvarguscamerasrc sensor_id=2 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1  ! tee name=t3
        t3. ! queue ! nvv4l2h264enc ! h264parse ! queue ! mux3.
        t3. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5002 
        nvarguscamerasrc sensor_id=3 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t4
        t4. ! queue ! nvv4l2h264enc ! h264parse ! queue ! mux4.
        t4. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5003
        alsasrc device=hw:2,0 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! volume name=volume1 ! mix1.
        alsasrc device=hw:3,0 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! volume name=volume2 ! mix1.
        audiomixer name=mix1 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! tee name=t
        t. ! queue ! mux1.
        t. ! queue ! mux2.
        t. ! queue ! mux3.
        t. ! queue ! mux4.
        qtmux name=mux1 ! filesink location=/media/{CAM1_output_file}
        qtmux name=mux2 ! filesink location=/media/{CAM2_output_file}
        qtmux name=mux3 ! filesink location=/media/{CAM3_output_file}
        qtmux name=mux4 ! filesink location=/media/{CAM4_output_file}
    """)

Pipeline for live view on Display 0

Gst.parse_launch(f"""
        udpsrc port={current_port_cam1} caps="application/x-rtp, media=video" name=udpsrc1 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! nvoverlaysink display-id=0 name=overlay1 overlay=1 overlay-depth=1 sync=false
    """)

Pipeline for live view on Display 1

Gst.parse_launch(f"""
        udpsrc port={current_port_cam2} caps="application/x-rtp, media=video" name=udpsrc2 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! nvoverlaysink display-id=1 name=overlay2 overlay=1 overlay-depth=1 sync=false
    """)

Gstreamer does run and capture frames into an mp4 but eventually leads to the following response. The program will hang until ctrl+C.


Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
NvMMLiteBlockCreate : Block : BlockType = 4
NvMMLiteBlockCreate : Block : BlockType = 4
NvMMLiteBlockCreate : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
Opening in BLOCKING MODE
Recording started. Press 'Ctrl+C' to stop.
Enter command: NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 1
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

CONSUMER: Producer has connected; continuing.
GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 3
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 2
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0
   Camera mode  = 2
   Output Stream W = 1920 H = 1080
   seconds to Run    = 0
   Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
H264: Profile = 66, Level = 0
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
a
Switched Camera 1 to port 5002.
Live Display Swapped
Enter command: areference in DPB was never decoded
a
Invalid command or not in the correct mode.
Enter command: a
Switched Camera 1 to port 5003.
Live Display Swapped
Enter command: reference in DPB was never decoded
CONSUMER: ERROR OCCURRED
CONSUMER: ERROR OCCURRED
CONSUMER: ERROR OCCURRED
(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
CONSUMER: Done Success
^CStopping the pipeline...
Error received: NvArgusCameraSrc: DISCONNECTED (8), Argus Error Status
GST_ARGUS: Cleaning up
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Segmentation fault (core dumped)

I believe this error is only caused from nvargus-daemon stopping which has been my main way of reproducing the error above. What is stopping nvargus-daemon I am unsure of. To investigate further, I checked the logs using journalctl, syslog, and dmesg for any relevant messages and at best :

nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
nvargus-daemon.service: Failed with result 'signal'.
nvargus-daemon.service: Service hold-off time over, scheduling restart.
nvargus-daemon.service: Scheduled restart job, restart counter is at 6.
Stopped Argus daemon.
Started Argus daemon.

My goal is to achieve stable recordings lasting at least 8+ hours without issues. I am unsure why or where else to look for errors. I would hope to be able to stop nvargus-daemon from crashing or if it does crash, it can restart and continue the previously opened pipeline. I have two Jetson Tx2 which this occurs on both.

Please get the daemon log by below command.

sudo journalctl -u nvargus-daemon -f
Oct 29 13:22:15 tx2-desktop nvargus-daemon[8416]: === python3[8503]: CameraProvider initialized (0x7f94892510)LSC: LSC surface is not based on full res!
Oct 29 13:22:15 tx2-desktop nvargus-daemon[8416]: LSC: LSC surface is not based on full res!
Oct 29 13:22:15 tx2-desktop nvargus-daemon[8416]: LSC: LSC surface is not based on full res!
Oct 29 13:22:15 tx2-desktop nvargus-daemon[8416]: LSC: LSC surface is not based on full res!
Oct 29 18:32:36 tx2-desktop systemd[1]: nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
Oct 29 18:32:36 tx2-desktop systemd[1]: nvargus-daemon.service: Failed with result 'signal'.
Oct 29 18:32:36 tx2-desktop systemd[1]: nvargus-daemon.service: Service hold-off time over, scheduling restart.
Oct 29 18:32:36 tx2-desktop systemd[1]: nvargus-daemon.service: Scheduled restart job, restart counter is at 1.
Oct 29 18:32:36 tx2-desktop systemd[1]: Stopped Argus daemon.
Oct 29 18:32:36 tx2-desktop systemd[1]: Started Argus daemon.

I haven’t been able to figure out what specifically is happening when these logs occur.

I have been testing further and it seems when only recording with 1 camera I have been able to get a 12+ hr session to work.

[EDIT]: Ran again this morning and grabbed the log from the run.

Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: === NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== python3[12063]: Connection established (7FAF5631D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module4
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: NvPclHwGetModuleList: No module data found
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: NvPclHwGetModuleList: No module data found
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: ---- imager: No override file found. ----
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: (NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: (NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: (NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 107)
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: NvPclStartPlatformDrivers: Failed to start module drivers
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: NvPclOpen: PCL Open Failed. Error: 0xf
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 593)
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 437)
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 305)
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 471)
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: Acquiring SCF Camera device source via index 0 has failed. ---- imager: No override file found. ----
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: LSC: LSC surface is not based on full res!
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: ---- imager: No override file found. ----
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: LSC: LSC surface is not based on full res!
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: ---- imager: No override file found. ----
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: LSC: LSC surface is not based on full res!
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: ---- imager: No override file found. ----
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: LSC: LSC surface is not based on full res!
Oct 30 07:47:29 tx2-desktop nvargus-daemon[9502]: === python3[12063]: CameraProvider initialized (0x7fa8434e60)LSC: LSC surface is not based on full res!
Oct 30 07:47:30 tx2-desktop nvargus-daemon[9502]: LSC: LSC surface is not based on full res!
Oct 30 07:47:30 tx2-desktop nvargus-daemon[9502]: LSC: LSC surface is not based on full res!
Oct 30 07:47:30 tx2-desktop nvargus-daemon[9502]: LSC: LSC surface is not based on full res!
Oct 30 09:15:59 tx2-desktop systemd[1]: nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
Oct 30 09:15:59 tx2-desktop systemd[1]: nvargus-daemon.service: Failed with result 'signal'.
Oct 30 09:15:59 tx2-desktop systemd[1]: nvargus-daemon.service: Service hold-off time over, scheduling restart.
Oct 30 09:15:59 tx2-desktop systemd[1]: nvargus-daemon.service: Scheduled restart job, restart counter is at 2.
Oct 30 09:15:59 tx2-desktop systemd[1]: Stopped Argus daemon.
Oct 30 09:15:59 tx2-desktop systemd[1]: Started Argus daemon.

The errors concerning NvCamV4l2 and SCF seem to occur on every program start up but do not inhibit the ability to create the mp4 if closed before crash.

Continued on with more tests,
I cut back on the main pipeline to record and live view with two cameras which has worked for 12+hrs before I cut it, will attempt to record 3 view 2 for further testing.

I also tested to see if I could just view for long periods of time, removing the file sink and just having the pipeline stream the source with the two other pipelines viewing them. This seemed to also break with the same error. Below are the pipelines used as well as the daemon log from sudo journalctl -u nvargus-daemon -f.

Streaming Pipeline

Gst.parse_launch(f"""
        nvarguscamerasrc sensor_id=1 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t1
        t1. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5000 
        nvarguscamerasrc sensor_id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,interlace-mode=interlaced ! tee name=t2
        t2. ! queue max-size-buffers=2 ! nvvidconv ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5001 
        nvarguscamerasrc sensor_id=2 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1  ! tee name=t3
        t3. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5002 
        nvarguscamerasrc sensor_id=3 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t4
        t4. ! queue max-size-buffers=2 ! nvv4l2h264enc preset-level=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5003
    """)

Pipeline for live view on Display 0

Gst.parse_launch(f"""
        udpsrc port={currentPortCam1} caps="application/x-rtp, media=video" name=udpsrc1 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! nvoverlaysink display-id=0 name=overlay1 overlay=1 overlay-depth=1 sync=false
    """)

Pipeline for live view on Display 1

Gst.parse_launch(f"""
        udpsrc port={currentPortCam2} caps="application/x-rtp, media=video" name=udpsrc2 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! nvoverlaysink display-id=1 name=overlay2 overlay=1 overlay-depth=1 sync=false
    """)

sudo journalctl -u nvargus-daemon -f

Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: === NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== python3[16454]: Connection established (7FA67F31D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module4
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: NvPclHwGetModuleList: No module data found
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: NvPclHwGetModuleList: No module data found
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: ---- imager: No override file found. ----
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: (NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: (NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: (NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 107)
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: NvPclStartPlatformDrivers: Failed to start module drivers
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: NvPclOpen: PCL Open Failed. Error: 0xf
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 593)
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 437)
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 305)
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 471)
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: Acquiring SCF Camera device source via index 0 has failed. ---- imager: No override file found. ----
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: LSC: LSC surface is not based on full res!
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: ---- imager: No override file found. ----
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: LSC: LSC surface is not based on full res!
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: ---- imager: No override file found. ----
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: LSC: LSC surface is not based on full res!
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: ---- imager: No override file found. ----
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: LSC: LSC surface is not based on full res!
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: === python3[16454]: CameraProvider initialized (0x7fa099ec50)LSC: LSC surface is not based on full res!
Oct 30 15:46:44 tx2-desktop nvargus-daemon[15539]: LSC: LSC surface is not based on full res!
Oct 30 15:46:45 tx2-desktop nvargus-daemon[15539]: LSC: LSC surface is not based on full res!
Oct 30 15:46:45 tx2-desktop nvargus-daemon[15539]: LSC: LSC surface is not based on full res!
Oct 30 20:38:55 tx2-desktop systemd[1]: nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
Oct 30 20:38:55 tx2-desktop systemd[1]: nvargus-daemon.service: Failed with result 'signal'.
Oct 30 20:38:55 tx2-desktop systemd[1]: nvargus-daemon.service: Service hold-off time over, scheduling restart.
Oct 30 20:38:55 tx2-desktop systemd[1]: nvargus-daemon.service: Scheduled restart job, restart counter is at 2.
Oct 30 20:38:55 tx2-desktop systemd[1]: Stopped Argus daemon.
Oct 30 20:38:55 tx2-desktop systemd[1]: Started Argus daemon.

All of the failed tell the APP terminal unnormally.
Please remove the encoder/decoder by using fake sink to narrow down the problem.

I ran the following pipeline using fake sinks and I still was able to get the error occurring with no real change on both machines. I will provide the pipelines used, the gstreamer that ran and the logs from journalctl.

Pipeline used with fakesinks instead of udpsink and filesink. Viewing pipelines were disabled for the test. One Machine ran pipeline without the audio sources as well.

        nvarguscamerasrc sensor_id=1 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t1
        t1. ! queue ! fakesink
        t1. ! queue max-size-buffers=2 ! fakesink

        nvarguscamerasrc sensor_id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,interlace-mode=interlaced ! tee name=t2
        t2. ! queue ! fakesink
        t2. ! queue max-size-buffers=2 ! nvvidconv ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! queue ! fakesink

        nvarguscamerasrc sensor_id=2 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t3
        t3. ! queue ! fakesink
        t3. ! queue max-size-buffers=2 ! fakesink

        nvarguscamerasrc sensor_id=3 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1 ! tee name=t4
        t4. ! queue ! fakesink
        t4. ! queue max-size-buffers=2 ! fakesink

        alsasrc device=hw:2,0 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! volume name=volume1 ! mix1.
        alsasrc device=hw:3,0 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! volume name=volume2 ! mix1.
        audiomixer name=mix1 ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! tee name=t
        t. ! queue ! fakesink
        t. ! queue ! fakesink
        t. ! queue ! fakesink
        t. ! queue ! fakesink

Gstreamer Logs

Running pipelines...
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

CONSUMER: Waiting until producer is connected...
GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 1
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59.999999
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0
   Camera mode  = 2
   Output Stream W = 1920 H = 1080
   seconds to Run    = 0
   Frame Rate = 29.999999
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 3
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

CONSUMER: Producer has connected; continuing.
GST_ARGUS: Running with following settings:
   Camera index = 2
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59.999999
CONSUMER: Producer has connected; continuing.
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
CONSUMER: Producer has connected; continuing.
CONSUMER: ERROR OCCURRED
CONSUMER: ERROR OCCURRED
(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 2 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 2 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
CONSUMER: Done Success
CONSUMER: Done Success
^CError received: NvArgusCameraSrc: DISCONNECTED (8), Argus Error Status
GST_ARGUS: Cleaning up
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Segmentation fault (core dumped)

Logs from sudo journalctl -u nvargus-daemon -f

-- Logs begin at Mon 2024-11-04 07:05:46 EST. --
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: ---- imager: No override file found. ----
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: LSC: LSC surface is not based on full res!
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: ---- imager: No override file found. ----
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: LSC: LSC surface is not based on full res!
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: ---- imager: No override file found. ----
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: LSC: LSC surface is not based on full res!
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: === python3[7276]: CameraProvider initialized (0x7f8d9cf8a0)LSC: LSC surface is not based on full res!
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: LSC: LSC surface is not based on full res!
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: LSC: LSC surface is not based on full res!
Nov 04 07:09:17 tx2-desktop nvargus-daemon[4688]: LSC: LSC surface is not based on full res!
Nov 04 09:29:37 tx2-desktop systemd[1]: nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
Nov 04 09:29:37 tx2-desktop systemd[1]: nvargus-daemon.service: Failed with result 'signal'.
Nov 04 09:29:37 tx2-desktop systemd[1]: nvargus-daemon.service: Service hold-off time over, scheduling restart.
Nov 04 09:29:37 tx2-desktop systemd[1]: nvargus-daemon.service: Scheduled restart job, restart counter is at 1.
Nov 04 09:29:37 tx2-desktop systemd[1]: Stopped Argus daemon.
Nov 04 09:29:37 tx2-desktop systemd[1]: Started Argus daemon.

[EDIT]
I have additionally ran all 4 of them using separate instances of nvgstcapture-1.0. The same error occurred on 3 of them while 1 crashed and had a different error that is below. They each ran and viewed fine until this error. This has occurred once so I will attempt to recreate it.

GST_ARGUS: Running with following settings:
   Camera index = 2
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
CONSUMER: ERROR OCCURRED
ERROR on bus: by /GstPipeline:capture_native_pipeline/GstBin:cap_bin/GstNvArgusCameraSrc:nvarguscamerasrc0: DISCONNECTED
debug info:
Argus Error Status
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Segmentation fault (core dumped)

What’s the version?

cat /etc/nv_tegra_release

cat /etc/nv_tegra_release
R32 (release), REVISION: 7.5, GCID: 36557527, BOARD: t186ref, EABI: aarch64, DATE: Tue Jun 11 23:18:04 UTC 2024