Critical failure in nvargus-daemon

Oystein1 · February 29, 2020, 1:31am

Hello. I am having problems with streaming video over RTSP on Jetson TX2 with GStreamer using an IMX274 MIPI CSI camera from Leopard imaging.

This is an example pipeline (which I pass to gst_rtsp_media_factory_set_launch):

nvarguscamerasrc maxperf=1 do-timestamp=true ! video/x-raw(memory:NVMM), width=(int)640, height=(int)480, format=(string)NV12, framerate=(fraction)30/1 ! queue ! omxh264enc control-rate=3 bitrate=1000000 peak-bitrate=3000000 qos=true vbv-size=30 ! rtph264pay name=pay0 pt=96

When I connect to this stream it runs for a few seconds before I get the following error:

===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 40 
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3840 x 2160 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 44.400002; Exposure Range min 44000, max 478696000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 3840 H = 2160 
   seconds to Run    = 0 
   Frame Rate = 59.999999 
GST_ARGUS: PowerService: requested_clock_Hz=1024000000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
CONSUMER: Done Success
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
packet_write_wait: Connection to 10.10.1.104 port 22: Broken pipe

This is the output from journalctl -u nvargus-daemon:

jan. 28 16:58:24 scout-onboard-1 systemd[1]: Started Argus daemon.
jan. 28 16:58:36 scout-onboard-1 nvargus-daemon[6808]: === NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...=== scm[6812]: Connection established (7F86E2E1D0)NvPclHwGetModuleList: No module data found
jan. 28 16:58:36 scout-onboard-1 nvargus-daemon[6808]: NvPclHwGetModuleList: No module data found
jan. 28 16:58:36 scout-onboard-1 nvargus-daemon[6808]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
jan. 28 16:58:36 scout-onboard-1 nvargus-daemon[6808]: ---- imager: No override file found. ----
jan. 28 16:58:36 scout-onboard-1 nvargus-daemon[6808]: ---- imager: No override file found. ----
jan. 28 16:58:36 scout-onboard-1 nvargus-daemon[6808]: ---- imager: No override file found. ----
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: === scm[6812]: CameraProvider initialized (0x7f80a2aee0)Error: waitCsiFrameStart timeout guid 1
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: VI Stream Id = 2 Virtual Channel = 0
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: ************VI Debug Registers**********
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: VI_CSIMUX_STAT_FRAME_8         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: VI_CSIMUX_FRAME_STATUS_0         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: VI_CFG_INTERRUPT_STATUS_0         = 0x3f000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: VI_ISPBUFA_ERROR_0         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: VI_FMLITE_ERROR_0         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: VI_NOTIFY_ERROR_0         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: *****************************************
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: CSI Stream Id = 2 Brick Id = 1
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: ************CSI Debug Registers**********
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: CILA_INTR_STATUS_CILA[0x20400]         = 0x00000089
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: CILB_INTR_STATUS_CILB[0x20c00]         = 0x00000088
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: INTR_STATUS[0x200a4]         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: ERR_INTR_STATUS[0x200ac]         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: ERROR_STATUS2VI_VC0[0x20094]         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: ERROR_STATUS2VI_VC1[0x20098]         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: ERROR_STATUS2VI_VC2[0x2009c]         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: ERROR_STATUS2VI_VC3[0x200a0]         = 0x00000000
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: *****************************************
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: Error: waitCsiFrameStart Something went wrong with waiting on frame start
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
jan. 28 16:59:58 scout-onboard-1 nvargus-daemon[6808]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)

I am no longer able to get the camera working in a stable way. What could be the reason for this error and how do I resolve it?

KevinGong · February 1, 2020, 9:10am

Hi. Oystein.

Could you please share what driver do you use? What carrier board do you run this camera?

As I know. The driver may support resolution below:

3840x2160@60fps
1920x1080@60fps
1280x720@60fps
1280x540@240fps
3840x2160@30fps DOL-HDR
1920x1080@60fps DOL-HDR.

In your command, you define the video resolution at 640 x 480. But resolution 640 x 480 is not listed in the driver. So this resolution is not supported. I believe this is the reason.

Thank you.

Oystein1 · May 6, 2020, 5:56pm

Dear Kevin,

Thank you for your answer. We are using a Connect Tech Spacely carrier board. We are using JetPack 4.2 along with the Board Support Package from Connect Tech.

I have actually experimented with different resolutions. We have the same problem when choosing 3840x2160@60fps or 1920x1080@60fps.

My application has actually worked fine for several weeks, but suddenly it became unstable for both of our cameras.

I suspect that the camera may have entered a “bad state” after I was playing with some of the settings available for the nvarguscamerasrc element in GStreamer (such as exposuretimerange, gainrange, ispdigitalgainrange and aelock). But even after removing these parameters to nvarguscamerasrc, I am no longer able to get a stable videostream.

KevinGong · February 1, 2020, 9:10am

Hi. Oystein.

Can you confirm the cameras work well when you use them on Nvidia Devkit or our LI-TX1-CB with correct driver? Actually when you use this camera with Connect Tech carrier board. You need to use the driver from them since the driver for different carrier boards are different. I believe the driver is not correct in your case.

Thank you.

chenxuancui · March 16, 2021, 9:57pm

Hi, Im having the same problem too, any solutions? Im using LI-IMX185-GMSL2 from Leopard Imaging,

kayccc · April 8, 2021, 2:49am

Hi chenxuancui,

Please contact with Leopard Imaging for the support.

Thanks