Video input freeze when opening more than 2 cameras at the same time

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hi forum,

I tried to modified camera driver to enable xavier to support 6 cameras using ti 954, ti953 and imx390.

And I can open one camera, from /dev/video0-5, but when I use gstreamer command to open more than 2 cams, it failed.

gst-launch-1.0 \
nvarguscamerasrc sensor-id=0 bufapi-version='TRUE' ! tee name=c0 \
nvarguscamerasrc sensor-id=1 bufapi-version='TRUE' ! tee name=c1 \
c0. ! 'video/x-raw(memory:NVMM), framerate=30/1, format=(string)NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=30/1, format=(string)NV12' ! queue ! m.sink_0 \
c1. ! 'video/x-raw(memory:NVMM), framerate=30/1, format=(string)NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=30/1, format=(string)NV12' ! queue !m.sink_1 \
nvstreammux name=m batch-size=2 width=1920 height=1080 live-source=1 batched-push-timeout=40000 ! queue ! nvmultistreamtiler width=1920 height=540 rows=1 columns=2 ! nvvideoconvert ! nvdsosd ! queue ! tee name=deep \
deep. ! nvegltransform ! nveglglessink sync='FALSE' -e

Any solution for this?

I am using jetson agx xavier and modifying kernel based on jetpack 4.4(l4t 32.4.3)

BR

3

Could you check if launch by v4l2-ctl and have more simple pipeline to clarify the issue.

v4l2-ctl --stream-user --set-ctrl bypass_mode=0 --stream-count=100 -d /dev/video0 && 
v4l2-ctl --stream-user --set-ctrl bypass_mode=0 --stream-count=100 -d /dev/video1
4

Hi Shane.

Because the command you gave returned Invalid Atgument. So I tried
v4l2-ctl --set-fmt-video=width=1920,height=1100,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video0 && v4l2-ctl --set-fmt-video=width=1920,height=1100,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video1 && v4l2-ctl --set-fmt-video=width=1920,height=1100,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video2

And it can work

5

Right should using --stream-mmap instead --stream-user. and please using & instead of && for launch two sensor at the same time.

Also if v4l2-ctl working well by the && Please try to use xvimagesink like below.

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1100,framerate=60/1,format=NV12' ! nvvidconv ! fpsdisplaysink video-sink=xvimagesink sync=false \
& gst-launch-1.0 nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1100,framerate=60/1,format=NV12' ! nvvidconv ! fpsdisplaysink video-sink=xvimagesink sync=false
6

Hi Shane. Thanks for the reply.

Actually using this command will make one or more camera freeze. But using v4l2-ctl will not

And also I got these error from nvargus

(NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
(NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 1856)
7

Looks like argus set some CID failed cause the error. Could you try to disable the CID add the flags to the sensor driver to try.

https://www.kernel.org/doc/html/v4.9/media/uapi/v4l/control.html

8

Hi Shane. Would you mind elaborating how to do this?

And there is a update, I tested v4l2-ctl several times, I will get this error even if I opened one /dev/video node

root@zlh-desktop:/home/astri/driver_testing_script# cat /sys/kernel/debug/tracing/trace | grep ERR
kworker/0:1-818   [000] ....   213.618909: rtcpu_nvcsi_intr: tstamp:7000713521 class:CORRECTABLE_ERR type:PHY_INTR phy:3 cil:0 st:0 vc:0 status:0x00000088
kworker/0:1-818   [000] ....   213.618909: rtcpu_nvcsi_intr: tstamp:7000713521 class:CORRECTABLE_ERR type:PHY_INTR phy:3 cil:1 st:0 vc:0 status:0x00000088
root@zlh-desktop:/home/astri/driver_testing_script# cat /sys/kernel/debug/tracing/trace | grep rtcpu_nvcsi_intr
kworker/0:1-818   [000] ....   213.618907: rtcpu_nvcsi_intr: tstamp:7000676371 class:GLOBAL type:PHY_INTR0 phy:3 cil:0 st:0 vc:0 status:0x00000001
kworker/0:1-818   [000] ....   213.618908: rtcpu_nvcsi_intr: tstamp:7000713521 class:GLOBAL type:PHY_INTR0 phy:3 cil:0 st:0 vc:0 status:0x00000088
 kworker/0:1-818   [000] ....   213.618908: rtcpu_nvcsi_intr: tstamp:7000713521 class:GLOBAL type:PHY_INTR0 phy:3 cil:1 st:0 vc:0 status:0x00000088
 kworker/0:1-818   [000] ....   213.618909: rtcpu_nvcsi_intr: tstamp:7000713521 class:CORRECTABLE_ERR type:PHY_INTR phy:3 cil:0 st:0 vc:0 status:0x00000088
 kworker/0:1-818   [000] ....   213.618909: rtcpu_nvcsi_intr: tstamp:7000713521 class:CORRECTABLE_ERR type:PHY_INTR phy:3 cil:1 st:0 vc:0 status:0x00000088

but I tested opening 1-5 devnode, they can run normally, unlike libargus. And there is a post about this error, the answer is to modify serdes_pixel_clk_hz, which I already set to 833333333.

9

How about try to boost the NVCSI/VI/ISP clocks to check it.

https://elinux.org/Jetson_TX2_Camera_BringUp

10

Hi, I’ve boosted the clock, but got the same result

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace
11

The error tell the data lane control error for debug if not show every capture and can capture successful that could be ignore I think.

12

OK. Sometime I will get this error

NvCaptureStatusErrorDecode Stream 2.0 failed: sof_ts 0 eof_ts 6035105617568 frame 0 error 2 data 0x000000a2
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.

And also, normally there will be, no reply from camera processor, in dmesg

13

I solved this problem.

It’s because camera initialization time out. I set the enableCamInfiniteTimeout=1 or set the set_mode_delay_ms larger, it can work

1 Like
14

Hi Shane,

although I can open up to 8 cameras using fpdlink serdes, but camera input will freeze and the error is

NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0
    
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 855127029120 eof_ts 855148679840 frame 37 error 2 data 0x000000a0

NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0
    
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
 SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 0 eof_ts 855164282464 frame 0 error 2 data 0x000000a0
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0
    
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 0 eof_ts 859288546112 frame 0 error 2 data 0x000000a0
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0
    
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, 
function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 0 eof_ts 859288814240 frame 0 error 2 data 0x000000a0
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0
    
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, 
function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 0 eof_ts 859293412960 frame 0 error 2 data 0x000000a0
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0
    
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, 
function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 0 eof_ts 859296879712 frame 0 error 2 data 0x000000a0
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0
    
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, 
function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 0 eof_ts 859297762912 frame 0 error 2 data 0x000000a0
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0
    
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.0 failed: sof_ts 859286940256 eof_ts 27497774275584 frame 42 error 14 data 0x00020000
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
NvCaptureStatusErrorDecode Stream 0.0 failed: sof_ts 0 eof_ts 859305443584 frame 0 error 2 data 0x000000a0
NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
CsimuxFrameError_Regular : 0x000000a0
Stream ID                [ 2: 0]: 0        
VPR state from fuse block    [ 3]: 0
    
Frame end (FE)              [ 5]: 1
    A frame end has been found on a regular mode stream.
FS_FAULT                    [ 7]: 1
    A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Binary VC number [3:2]   [27:26]: 0
    To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 539)
=== gst-launch-1.0[9411]: Connection closed (7F8339D1D0)=== gst-launch-1.0[9411]: WARNING: CameraProvider was not destroyed before client connection terminated.=== gst-launch-1.0[9411]:          The client may have abnormally terminated. Destroying CameraProvider...=== gst-launch-1.0[9411]: CameraProvider destroyed (0x7f7c810080)=== gst-launch-1.0[9411]: WARNING: Cleaning up 5 outstanding requests...=== gst-launch-1.0[9411]: WARNING: Cleaning up 5 outstanding streams...SCF: Error InvalidState: 6 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 306)
SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
SCF: Error InvalidState: 4 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 306)
SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
SCF: Error InvalidState: 5 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 306)
SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
SCF: Error InvalidState: 6 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 306)
SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
SCF: Error InvalidState: 5 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 306)
SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
=== gst-launch-1.0[9411]: WARNING: Cleaning up 5 outstanding stream settings...
=== gst-launch-1.0[9411]: WARNING: Cleaning up 5 outstanding sessions...Segmentation fault (core dumped)

The ending part is I use control-c to close the gstreamer pipeline and nvargus-daemon just return segmentation fault. And if I boost the clock, gst command can not open camera and nvargus-daemon will freeze directly

15

The error tell NVCSI/VI got FS package in the frame middle.
Could you try single or less cameras to clarify the problem.

16

Hi Shane,

this issue happened when I used even 1 camera

17

It could be the signal cause the problem.
Could you check if can get the data by v4l2-ctl continuously.

v4l2-ctl --set-ctrl bypass_mode=0 --stream-mmap
18

just as I described here, running 5 v4l-ctl command in 5 separate terminals can work, although sometimes one device will freeze and reset

Screenshot from 2021-03-11 16-14-56
Screenshot from 2021-03-11 16-14-561920×1080 1.25 MB

but I set the camera can only stream 30 fps in dts, don’t know why it always stream 53 fps

19

Always the same device freeze or dynamic happened?

20

Hi Shane,

I remember in most cases, cam connected to csi-g tends to freeze first.

But I ran bist of fpdlink, the connection of serdes should be fine

21

Could you make sure of it.
And what do you mean “But I ran bist of fpdlink, the connection of serdes should be fine”