Argus (Tegra Multimedia API) in multi-process mode does not terminate

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hello,

We have an application that uses libArgus version 0.97.3 to capture cameras on TX2 (L4T-32.2). This application was first developed and tested using the “single-process” version of Argus (linking against libnvargus.so). We tried recently to use the “multi-process” version (linking against libnvargus_socketclient.so) instead and ran into a cleaning issue :

There is always at least 2 processes “/usr/sbin/nvargus-daemon” already running at startup.
Running with the multi-process version sometimes results in additional nvargus-daemon processes being instantiated that are never terminated after exiting the application. With “htop”, we saw that those additional processed also keep holding RAM memory even after application stopped. This can lead to attempts to restart the app failing (either cannot start or cannot stop capture requests). We have to manually stop (kill) those processes in order to re-run properly the application.

In fact, using multi-process mode, our app sometimes randomly hangs at calls of Argus::ICaptureSession::waitForIdle() for seconds, until timeout is reached. In that case, our app cannot exit after reached end of main(). We think that it may be caused by other threads that we could not join therefore holding the app process alive. We observe numerous non-terminated nvargus-daemon in the specific case where waiting for idle failed.

After a while when being blocked, we get :

(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)

We observe this behavior with the argus_camera sample.

How to properly clean Argus processes using the multi-process version?

Note : we use the following configuration

  • Platform = Jetson TX2i
  • Carrier board = ConnectTech Spacely
    • DIP Switches = TX2 ; CSIx2
  • Cameras : 6x Leopard Imaging IMX274 csi
  • OS : L4T-32.2 (JetPack-4.2.1)
  • BSP ConnectTech : V126 (support for L4T-32.2)
    • support for Spacely
    • support for TX2i
    • support for 6x IMX274 (CSI 2-lanes)
2

Any kernel timeout message while error?
Could you attached the binary to try?

3

Unfortunately, I cannot attach our own binary, but you can try with argus_camera since we observe the same behavior. The only problem is that you will not be able to diagnose that it hangs at waitForIdle().

To reproduce the problem :

  1. run argus_camera with 6 CSI cameras
  2. go into Multi Session mode (you should see 6 displays)
  3. exit the app
  4. repeat steps above until either app shows less displays than 6 (usually 1-2 restarts are enough) or crash
  5. after the app crashed, you should see a lot of "nvargus-daemon" running (top or htop)
  6. sometimes, running sudo kill -9 $(pidof nvargus-daemon && sudo service nvargus-daemon restart is sufficient to restore to normal, but the best way to do it is reboot
4

I am providing attached the extracted syslog from one time when I have the problem.

At 10:03, I am running argus_camera without problem. At 10:04, I am trying to re-run argus_camera (after exiting previously), and got only the cameras of index 0-1-2-4 displayed (displays of cameras #3 and #5 remain black). Trying to exit results in “This window is not responding” and multiple nvargus_daemons processes not being cleaned.

Note : I am sorry I am unable to precisely locate into syslog what exactly matches my calls of “argus_camera” and the associated crashes, so I tried the best I could to extract from the file the lines linked to the case I describe above. It is very difficult to locate something into a file long of tens of thousands of lines. May I suggest not logging Argus errors and warnings into syslog but in a dedicated log file for future releases?
argus_crash_log.txt (305 KB)

1 Like
5

Could you try the single process by cmake -DMULTIPROCESS=OFF

NEW FEATURES:

  • Multi-process support. This adds an nvargus-daemon system service and
    corresponding client-side library that applications link against in order to
    enable multi-process support. Besides changing the Argus library being linked
    against, no other application changes are required for multi-process support.

    To link applications for multi-process use, replace the libnvargus.so library
    name with libnvargus_socketclient.so. Alternatively, if using the CMake build
    system with the sample applications, provide the optional MULTIPROCESS flag
    during the makefile generation step:
    ‘cmake -DMULTIPROCESS=ON …’

6

(Note : the cmake option MULTIPROCESS does not exist anymore in the sample cmake files. While digging into it, I found a DISABLE_MULTIPROCESS option. Turning it on did the trick.)

When we encounter the problem, we got the following outputs:

During run, repeated very frequently:

<!-- Flow -->
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
	Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
	parsing aborted at line 6:53: 
    <node name="lowpass" file="filters/box3x3.xml"/>
    <node name="ltmFilter" file="filters/ltm42
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
	Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
	parsing aborted at line 6:49: 

    <!-- Parameters -->
    <export name="Quality" from="ltm" attribute="quality" default="3" min=
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
	Out of memory
SCF: Error InsufficientMemory:  (propagating from src/components/stages/HdfxStage.cpp, function doAsyncInitialize(), line 147)
SCF: Error InsufficientMemory: HdfxStage initialization failed (in src/components/stages/HdfxStage.cpp, function doAsyncInitialize(), line 252)
PowerServiceCore:handleRequests: timePassed = 718

While exiting argus_camera just before having the problem:

...
(NvOdmDevice) Error FileOperationFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/HotplugMonitor.cpp, function close(), line 153)
(NvOdmDevice) Error FileOperationFailed: Failure in HotplugMonitor::close() (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/HotplugMonitor.cpp, function removeHotplugCallback(), line 830)

While trying to run again the app:

...
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 205)
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 205)
Error: waitCsiFrameEnd timeout guid 3
VI Stream Id = 4 Virtual Channel = 0
************VI Debug Registers**********
VI_CSIMUX_STAT_FRAME_16	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00001111
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 4 Brick Id = 2
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x30400]	 = 0x00000089
CILB_INTR_STATUS_CILB[0x30c00]	 = 0x00000000
INTR_STATUS[0x300a4]	 = 0x00000000
INTR_STATUS[0x300a4]	 = 0x00000000
ERR_INTR_STATUS[0x300ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x30094]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x30098]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x3009c]	 = 0x00000000
Error: waitCsiFrameStart timeout guid 3
VI Stream Id = 4 Virtual Channel = 0
************VI Debug Registers**********
ERROR_STATUS2VI_VC3[0x300a0]	 = 0x00000000
*****************************************
VI_CSIMUX_STAT_FRAME_16	 = 0x00000000
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
VI_CSIMUX_FRAME_STATUS_0	 = 0x00001111
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 4 Brick Id = 2
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x30400]	 = 0x00000089
CILB_INTR_STATUS_CILB[0x30c00]	 = 0x00000000
INTR_STATUS[0x300a4]	 = 0x00000000
INTR_STATUS[0x300a4]	 = 0x00000000
ERR_INTR_STATUS[0x300ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x30094]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x30098]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x3009c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x300a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Error: waitCsiFrameEnd timeout guid 0
VI Stream Id = 5 Virtual Channel = 0
************VI Debug Registers**********
VI_CSIMUX_STAT_FRAME_20	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000101
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 5 Brick Id = 2
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x30400]	 = 0x00000089
CILB_INTR_STATUS_CILB[0x30c00]	 = 0x00000000
INTR_STATUS[0x308a4]	 = 0x00000000
INTR_STATUS[0x308a4]	 = 0x00000000
ERR_INTR_STATUS[0x308ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x30894]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x30898]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x3089c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x308a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
Error: waitCsiFrameStart timeout guid 0
VI Stream Id = 5 Virtual Channel = 0
************VI Debug Registers**********
SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
VI_CSIMUX_STAT_FRAME_20	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00001111
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 5 Brick Id = 2
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x30400]	 = 0x00000089
CILB_INTR_STATUS_CILB[0x30c00]	 = 0x00000000
INTR_STATUS[0x308a4]	 = 0x00000000
INTR_STATUS[0x308a4]	 = 0x00000000
ERR_INTR_STATUS[0x308ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x30894]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x30898]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x3089c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x308a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Killed
7

Looks like it’s MIPI timing case the NVCSI/VI didn’t get validate data from MIPI bus cause timeout.
Does this error while run multiple sensors?
Could you try to boost the clock to try. run the sudo jetson_clocks also boost below clock.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate
echo ${max_rate} > /sys/kernel/debug/bpmp/debug/clk/vi/rate
echo ${max_rate} > /sys/kernel/debug/bpmp/debug/clk/isp/rate
echo ${max_rate} > /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
8

Yes it is when trying to capture and display from 6 IMX274 CSI cameras (“MultiSession” in the argus_camera GUI).

After rebooting + applying your instructions, displaying 6 cameras resulted in only 3 cameras being displayed and the following output:

<!-- Parameters -->
    <export name="Quality" from="ltm" attribute="quality" default="3" min=
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
	Out of memory
SCF: Error InsufficientMemory:  (propagating from src/components/stages/HdfxStage.cpp, function doAsyncInitialize(), line 147)
SCF: Error InsufficientMemory: HdfxStage initialization failed (in src/components/stages/HdfxStage.cpp, function doAsyncInitialize(), line 252)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 115
	Invalid default value of type 'vec4'
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
	parsing aborted at line 13:96: 

    <!-- Flow -->
    <flow operation="execute"/>
    <flow operation="detach" operand0="inputImag
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
	(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
	(propagating)
SCF: Error InsufficientMemory:  (propagating from src/components/stages/SharpenStage.cpp, function doAsyncInitialize(), line 159)
SCF: Error InsufficientMemory: SharpenStage initialization failed (in src/components/stages/SharpenStage.cpp, function doAsyncInitialize(), line 182)
PowerServiceCore:handleRequests: timePassed = 707
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 1856)
updateOutputSettings: Set Control failed. Use cached values
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 1856)
updateOutputSettings: Set Control failed. Use cached values
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 205)
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 1856)
updateOutputSettings: Set Control failed. Use cached values
Error: waitCsiFrameEnd timeout guid 2
VI Stream Id = 3 Virtual Channel = 0
************VI Debug Registers**********
Error: waitCsiFrameStart timeout guid 2
VI_CSIMUX_STAT_FRAME_12	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000111
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 3 Brick Id = 1
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x20400]	 = 0x00000000
CILB_INTR_STATUS_CILB[0x20c00]	 = 0x00000089
INTR_STATUS[0x208a4]	 = 0x00000000
INTR_STATUS[0x208a4]	 = 0x00000000
ERR_INTR_STATUS[0x208ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x20894]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x20898]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x2089c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x208a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
VI Stream Id = 3 Virtual Channel = 0
************VI Debug Registers**********
VI_CSIMUX_STAT_FRAME_12	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000111
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 3 Brick Id = 1
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x20400]	 = 0x00000000
CILB_INTR_STATUS_CILB[0x20c00]	 = 0x00000089
INTR_STATUS[0x208a4]	 = 0x00000000
INTR_STATUS[0x208a4]	 = 0x00000000
ERR_INTR_STATUS[0x208ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x20894]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x20898]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x2089c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x208a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 1856)
updateOutputSettings: Set Control failed. Use cached values
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 205)
PowerServiceCore:handleRequests: timePassed = 10309
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 1856)
updateOutputSettings: Set Control failed. Use cached values
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 1856)
updateOutputSettings: Set Control failed. Use cached values
(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 205)
launchCC abort cc 107 session 4
launchCC abort cc 106 session 1
SCF: Error Timeout:  (propagating from src/api/Session.cpp, function capture(), line 834)
SCF: Error Timeout:  (propagating from src/api/Session.cpp, function capture(), line 834)
(Argus) Error Timeout:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
(Argus) Error InvalidState: MetadataResult callback for unknown capture. (in src/api/CaptureSessionImpl.cpp, function metadataResult(), line 906)
(Argus) Error Timeout:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
(Argus) Error InvalidState: MetadataResult callback for unknown capture. (in src/api/CaptureSessionImpl.cpp, function metadataResult(), line 906)
launchCC abort cc 107 session 5
SCF: Error Timeout:  (propagating from src/api/Session.cpp, function capture(), line 834)
(Argus) Error Timeout:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
(Argus) Error InvalidState: MetadataResult callback for unknown capture. (in src/api/CaptureSessionImpl.cpp, function metadataResult(), line 906)
Error: waitCsiFrameEnd timeout guid 3
VI Stream Id = 4 Virtual Channel = 0
************VI Debug Registers**********
VI_CSIMUX_STAT_FRAME_16	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000111
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 4 Brick Id = 2
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x30400]	 = 0x00000000
CILB_INTR_STATUS_CILB[0x30c00]	 = 0x00000000
INTR_STATUS[0x300a4]	 = 0x00000000
INTR_STATUS[0x300a4]	 = 0x00000000
ERR_INTR_STATUS[0x300ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x30094]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x30098]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x3009c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x300a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
Error: waitCsiFrameStart timeout guid 3
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
VI Stream Id = 4 Virtual Channel = 0
************VI Debug Registers**********
SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
VI_CSIMUX_STAT_FRAME_16	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000111
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 4 Brick Id = 2
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x30400]	 = 0x00000000
CILB_INTR_STATUS_CILB[0x30c00]	 = 0x00000000
INTR_STATUS[0x300a4]	 = 0x00000000
INTR_STATUS[0x300a4]	 = 0x00000000
ERR_INTR_STATUS[0x300ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x30094]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x30098]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x3009c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x300a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Error: waitCsiFrameStart timeout guid 0
VI Stream Id = 5 Virtual Channel = 0
************VI Debug Registers**********
Error: waitCsiFrameEnd timeout guid 0
VI_CSIMUX_STAT_FRAME_20	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000111
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 5 Brick Id = 2
************CSI Debug Registers**********
VI Stream Id = 5 Virtual Channel = 0
************VI Debug Registers**********
CILA_INTR_STATUS_CILA[0x30400]	 = 0x00000000
VI_CSIMUX_STAT_FRAME_20	 = 0x00000000
VI_CSIMUX_FRAME_STATUS_0	 = 0x00000111
VI_CFG_INTERRUPT_STATUS_0	 = 0x3f000000
CILB_INTR_STATUS_CILB[0x30c00]	 = 0x00000000
VI_ISPBUFA_ERROR_0	 = 0x00000000
VI_FMLITE_ERROR_0	 = 0x00000000
VI_NOTIFY_ERROR_0	 = 0x00000000
*****************************************
CSI Stream Id = 5 Brick Id = 2
************CSI Debug Registers**********
CILA_INTR_STATUS_CILA[0x30400]	 = 0x00000000
CILB_INTR_STATUS_CILB[0x30c00]	 = 0x00000000
INTR_STATUS[0x308a4]	 = 0x00000000
INTR_STATUS[0x308a4]	 = 0x00000000
INTR_STATUS[0x308a4]	 = 0x00000000
INTR_STATUS[0x308a4]	 = 0x00000000
ERR_INTR_STATUS[0x308ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x30894]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x30898]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x3089c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x308a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
ERR_INTR_STATUS[0x308ac]	 = 0x00000000
ERROR_STATUS2VI_VC0[0x30894]	 = 0x00000000
ERROR_STATUS2VI_VC1[0x30898]	 = 0x00000000
ERROR_STATUS2VI_VC2[0x3089c]	 = 0x00000000
ERROR_STATUS2VI_VC3[0x308a0]	 = 0x00000000
*****************************************
SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Killed
9

It’s looks like the sensor timing or somethings cause the NVCSI/VI didn’t get validate data from the MIPI bus.
Could you have a narrow if run only one or two sensor will the problem happen?

10

Indeed, the problem appears when using more than 3 cameras. We have been running our application for months with 3 cameras without having this problem (except we were in “single-process” mode). The probability to encounter the problem seems to be kind of proportional to the number of cameras when using more than 3 cameras. That is, using 4 cameras will results in failing less often than using 6 cameras.

We tried with our own application, since we cannot choose the camera number in the argus_camera sample application: it always try to open 6 streams since the platform is 6-camera compatible. We would need to re-flash with the 3-camera configuration profile.

In fact, the problem exposed here is closely connected to the one described in our previous thread (https://devtalk.nvidia.com/default/topic/1065977/jetson-tx2/csi-camera-acquisition-multiple-problems-when-capturing-more-than-libargus-l4t-32-2-jetpack-4-2-1-spacely-6-imx274-/?offset=4#5398450), which is still open.

11

Could you try v4l2-ctl pipeline to run 6 cameras to confirm sensor output.

12

Could you please provide instructions to run this pipeline? I cannot find any command or folder that matches “c4l2-ctl”.

13

You need to install v4l-utils
sudo apt-get install v4l-utils and run below
v4l2-ctl -d /dev/videoX --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000

14

Thank you for the information.

v4l2-ctl seems to be OK : trying to start in parallel 6 times this pipeline with the 6 devices, and adding “–verbose” results in the following output :

(Starting pipeline #0)
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok

(Starting pipeline #1)
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
...

Nothing else is displayed. We performed the experiment several times to make sure it yields the same results.
Of course, trying to start a new pipeline on a device that is already used yields “Device or ressource busy”.

Side note : stopping the “nvargus-daemon” service before running our app does not fix our problem.

15

Did you use Leopards’ imx274 sensor ?
Could you sync with them if any driver update.

16

Hi videlo,

Are you using the LI-IMX274-MIPI + LI-JTX1-MIPI-ADPT-6CAM + Jetson TX2 developer kit? If so, which driver are you using? We have a latest driver based R32.2.1. You can contact support@leopardimaging.com to get it.

17

Hi,

Thank you for the answer.

No. We are using 6x LI-IMX274-MIPI + Jetson TX2i on ConnectTech Spacely carrier board (which has up to 6x MIPI interfaces). Therefore, our TX2i is flashed with ConnectTech’s BSP supporting said configuration (and L4T R32.2.1). We do not know if this BSP includes latest driver.

How do you check which driver is used?

18

Hi videlo,

We don’t have experience on the Spacely carrier board, but we tested the 6 LI-IMX274-MIPI cameras on LI-JTX1-MIPI-ADPT-6CAM via Argus Multi-session before. It seems that we didn’t see the “hangs” issue. Not sure if the issue is caused by the driver or PC end software.
Below are patch files of our IMX274 6CAM driver in case you need it for comparison.