Libargus crash when streaming synchronized cameras

ryan5k4rz · March 12, 2020, 5:23pm

Overview

We are setting up a system to stream 6 synchronized camera streams using Leopard Imaging camera sensors and their carrier board. Cam 0 is configured as the Master with all others set to Slave.

It is expected that all cameras would function when Master is running. Though, Cam 3 does not.

We initially had 400mm MIPI cables which the supplier then informed were beyond spec and 300mm should work. When swapping to 300mm, the issue seemed to happen less but still persists.

Has anyone experienced similar issues when streaming multiple cameras with libargus? What are possible reasons that the driver would crash?

More generally, what is the best method to monitor and debug libargus flows?

Hardware

(6x) LI-IMX264-MIPI v1.1
(6x) FAW-1233-03 (300 mm)
(1x) LI-TX1-CB-6CAM
(1x) TX2, Model number: 900-83310-0001-000

Software

L4T R32.1’ (modified with Leopard camera drivers)
JAX-TX2-Tegra_Multimedia_API_R32.1.0
Argus Camera v0.97.3

Steps to reproduce

In one terminal, run argus_camera and in a second terminal run argus_camera --device=3.

Expected Result

Both cameras would stream uninterrupted.

Actual Result

Cam 0 (master, Port A) streams fine. Launching Cam 3 (slave, Port D) crashes both Cam 3 and Cam 0 (as shown in visual proof below).

Visual Proof (screenshots, videos, text)

Video capture of argus_camera crash

Additional Notes

If Cam 0, 1, 2, 4, and 5 are launched then Cam 3 is launched, it will sometimes work.
Occasionally, a camera will crash after a period of streaming (e.g., 5 min - 20 min).

greivin.fallas · March 12, 2020, 11:54pm

Hi ryan5k4rz,

Is there a reason to use ‘argus_camera’ app?

I think that application works fine with one camera but I haven’t tested simultaneous cameras in that way.

Does the issue still happen with GStreamer?
I have run several cameras using GStreamer, I remember a similar issue with 4 synchronized cameras when last camera sometimes worked. When the camera failed I saw a error message related to a timeout on the CSI channel.

I found that NvArgus has a timeout for the incoming data in the CSI channels, I had to stop nvargus-daemon then restart nvargus-daemon using the parameter ‘enableCamInfiniteTimeout=1’.

This solved the issue at my side, I hope this helps at yours.

sudo service nvargus-daemon stop 
nvargus-daemon enableCamInfiniteTimeout=1
 
launch camera from another console

Greivin F.

pvechersky · March 13, 2020, 9:28pm

Hi greivin.fallas,

I work together with ryan5k4rz, the OP, and I am also working on trying to solve this issue.

Thank you for your quick and informative response!

Regarding the use of the ‘argus_camera’ - there is no particular reason for using it other than it is an easy-to-use tool for testing the cameras, with the UI interface allowing for changing certain camera parameters at runtime, and because it’s one of the tools mentioned in the Leopard’s driver installation instructions that should work out of the box.

I tried with GStreamer, first without re-starting the nvargus-daemon. I start Camera 0 (Master) in one console and the live stream works as expected. Streaming Camera 1, for example, in another console works as expected too. However, streaming Camera 3 in another console fails - the GStreamer window freezes before a single frame is displayed. Note that the other streams (Cameras 0 and 1 in this example) continue to stream just fine.

For reference, here is the GStreamer command that I use to launch the stream (with the sensor-id changed for different cameras, of course):

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)2448, height=(int)2058, framerate=20/1' ! nvvidconv flip-method=0 ! 'video/x-raw, format=(string)I420' ! xvimagesink -e

I then followed your suggestion to restart the nvargus-daemon and to relaunch it with the enableCamInfiniteTimeout=1 argument. Unfortunately I am observing the same behavior as described previously - Camera 3 stream window freezes without displaying a single frame. I do see some error messages in the console that is running the nvargus-daemon. Here is the console output:

=== NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...=== gst-launch-1.0[8648]: Connection established (7F8347E1D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module4
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module4
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module5
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module5
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
=== gst-launch-1.0[8648]: CameraProvider initialized (0x7f7ca98180)CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
=== gst-launch-1.0[8838]: Connection established (7F2F7FE1D0)=== gst-launch-1.0[8838]: CameraProvider initialized (0x7edc000be0)CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
PowerServiceCore:handleRequests: timePassed = 840
=== gst-launch-1.0[8938]: Connection established (7EBAFFD1D0)=== gst-launch-1.0[8938]: CameraProvider initialized (0x7e94000be0)CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
PowerServiceCore:handleRequests: timePassed = 827
Error: waitCsiFrameStart timeout guid 3
VI Stream Id = 3 Virtual Channel = 0
************VI Debug Registers**********
VI_CSIMUX_STAT_FRAME_12	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000000
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 3 Brick Id = 1
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x20400]	 = 0x00000000
CILB_INTR_STATUS_CILB[0x20c00]	 = 0x00000000
INTR_STATUS[0x208a4]	 = 0x00000000
INTR_STATUS[0x208a4]	 = 0x00000000
ERR_INTR_STATUS[0x208ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x20894]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x20898]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x2089c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x208a0]	 = 0x00000000
*****************************************
Error: waitCsiFrameEnd timeout guid 3
VI Stream Id = 3 Virtual Channel = 0
************VI Debug Registers**********
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
VI_CSIMUX_STAT_FRAME_12	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000011
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 3 Brick Id = 1
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x20400]	 = 0x00000000
CILB_INTR_STATUS_CILB[0x20c00]	 = 0x00000000
INTR_STATUS[0x208a4]	 = 0x00000000
INTR_STATUS[0x208a4]	 = 0x00000000
ERR_INTR_STATUS[0x208ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x20894]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x20898]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x2089c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x208a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc

Is there anything here that can point us in the right direction? Or any other logs that would be useful to provide?

Thank you!

greivin.fallas · March 17, 2020, 1:30am

Hi,

I would suggest some tests using V4L2-utils instead of nvargus, just to isolate the issue and test if CSI channels also suffer the timeout.
With v4l2-ctl you can specify the video device to extract buffers, so you can test different launch order on cameras to check if camera3 still fails.

Regards.

SimonZhu · March 17, 2020, 8:45am

Hi ryan5k4rz,

Sorry for the trouble. Have you tried other “good” slave cameras on portD? If other cameras on portD have the same issue, I think the problem is caused by the LI-TX1-CB-6CAM board (probably portD interface). You can contact support@leopardimaging.com for return and further investigation.

waisskharni.sm · March 19, 2020, 6:30am

HI Ryan5k4rz and Pvechersky,
For making my understanding more clearer, there are 6 cameras connected with Cam 0 acting as master and the remaining 5 cameras running in slave mode. I believe the master-slave configuration is done using a custom enumerated control.
May I know if you are configuring all the remaining cameras to slave mode before starting CAM 3?
I have worked in a similar product where we synced 4 cameras using master-slave.

SimonZhu · March 20, 2020, 8:01am

Yes, we enabled one camera (video0) as master and other cameras as slave in driver. We don’t see the CAM3 issue on our side, that’s why I suspect that there is an issue on the CAM3 port (portD).