NVargus crashes when streaming with 2 simultaneous IMX390 cameras

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
#1

Hi all,

I am running 2 simple GStreamer pipelines (just capture with nvarguscamerasrc and display) and both pipelines crash after ~ 40 minutes. I tested several times and the crash happens after 40 minutes.

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! nvvidconv ! queue ! xvimagesink sync=false

I tested with 3 cameras and the issue happens after ~5 minutes, always.

My hardware setup:

  • Xavier AGX running JetPack 4.2.0
  • 2x IMX390RCM cameras using FPD Link III
  • 2x Octo camera interface board from D3

I tested NVarguscamerasrc capture to fakesink and the pipelines still fail.

I captured frames using V4L2 Utils from both cameras and the crash did not appear, so It must be related only to NVargus.

Check log from NVargus-daemon when the issue happens:

=== gst-launch-1.0[7337]: CameraProvider initialized (0x7fa8add240)LSC: LSC surface is not based on full res!
=== gst-launch-1.0[7414]: Connection established (7F637FE1D0)=== gst-launch-1.0[7414]: CameraProvider initialized (0x7f00000be0)LSC: LSC surface is not based on full res!
PowerServiceCore:handleRequests: timePassed = 2432
NvCaptureStatusErrorDecode Stream 0.0 failed: sof_ts 0 eof_ts 1041307540672 frame 0 error 3 data 0x00000002
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_STREAM (0x00000003)
CsimuxStreamError : 0x00000002
    FIFO overflow stream 0      [ 1]: 1
        
(NvCapture) Error InvalidState: Channel is in error state, reset required (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureRequestGetAttribute(), line 1863)
SCF: Error InvalidState:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 904)
SCF: Error InvalidState: Capture error with status 0 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 913)
SCF: Error InvalidState: Sequence order error (52642 received, 0 expected, channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 920)
(NvCapture) Error InvalidState: Channel in error state, reset required (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureReleaseRequest(), line 644)
SCF: Error InvalidState:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 930)
SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 972)
NvCaptureStatusErrorDecode Stream 4.0 failed: sof_ts 0 eof_ts 1041507332160 frame 0 error 3 data 0x00020000
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_STREAM (0x00000003)
CsimuxStreamError : 0x00020000
    FIFO overflow stream 4      [17]: 1
        
(NvCapture) Error InvalidState: Channel is in error state, reset required (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureRequestGetAttribute(), line 1863)
SCF: Error InvalidState:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 904)
SCF: Error InvalidState: Capture error with status 0 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 913)
SCF: Error InvalidState: Sequence order error (51159 received, 0 expected, channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 920)
(NvCapture) Error InvalidState: Channel in error state, reset required (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureReleaseRequest(), line 644)
SCF: Error InvalidState:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 930)
SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 972)

Is there any hint to avoid the crash after 40 minutes?

I know that some parameters can be set with nvargus-daemon but I am not sure if this issue can be avoided by setting a special configuration to nvargus. I already tested enableCamInfiniteTimeout=1 but pipelines still crash.

Any help with this issue will be appreciated.

Greivin F.

#2

Could you repo this to D3 we have some issues working with them.
It’s better to have them to help on this.

#3

hello greivin.fallas,

since IMX390 had standard output formats and also PWL sensor mode for high dynamic range.
may I know which sensor mode you’re running with to reproduce the issue.
thanks

#4

Hi JerryChang,

I tested 2 modes: HDR and Linear common.

Running HDR mode (60 FPS), the issue appears after ~40min.
Using Linear common mode (30 FPS), the issue happens after ~80min.

Regards,
Greivin F.

#5

hello greivin.fallas,

it looks like the same issue reported by D3 team.
we had some changes deliver to D3 and confirmed two imx390s running in linear mode without failures.
please consult with D3 team for the updates.

regarding to HDR mode, we had setup an environment locally for dual PWL sensors preview long run testing.
verified that dual PWL camera preview frame works normally, thanks

#6