Multi-camera app fails to capture multiple frames on Jetson AGX

ClusterImaging · March 19, 2021, 5:58pm

Hi,

We are trying to capture frames from multiple cameras (AR0234 CCCC sensors) using 13_multi_camera sample app but facing errors in some cases as mentioned below.

no of cam = 1, no of frames = 1 —> works fine
no of cam = 1, no of frames >1 —> works fine
no of cam = 8, no of frames = 1 —> works fine
no of cam = 8, no of frames > 1
In this case, the application is able to capture 1st frame from all 8 sensors. But the 2nd frame acquire fails from all sensors. Attaching the error logs and dmesg logs.
Please help us identify what could be the issue.

We are using the below environment.

NVIDIA AGX board
Connect Tech RogueX board
AR0234 CCCC RAW sensor from Leopard Imaging

We are using Jetpack 4.3 + Connect Tech BSP support package for connect Tech board.

Error log:
CONSUMER: Dump frame 0
CONSUMER: IImage(2D): buffer 0 (1920x1200, 3840 stride), 82 0a c2 0b 02 0a 82 0a c2 0a 82 0a
CONSUMER: Dump frame 1
CONSUMER: IImage(2D): buffer 0 (1920x1200, 3840 stride), 02 0a c2 0a 02 0a 42 0b 82 0a 82 0a
CONSUMER: Dump frame 2
CONSUMER: IImage(2D): buffer 0 (1920x1200, 3840 stride), 02 0a 02 0b c2 0a 82 0a 02 0b 02 0b
CONSUMER: Dump frame 3
CONSUMER: IImage(2D): buffer 0 (1920x1200, 3840 stride), 42 0b 82 0a c2 0a c2 0a 42 0a c2 0a
CONSUMER: Dump frame 4
CONSUMER: IImage(2D): buffer 0 (1920x1200, 3840 stride), c2 08 c2 0a 82 0a 42 0a 82 0a 42 0a
CONSUMER: Dump frame 5
CONSUMER: IImage(2D): buffer 0 (1920x1200, 3840 stride), 82 0b 42 0a 42 0a c2 0a 82 0a 82 0b
CONSUMER: Dump frame 6
CONSUMER: IImage(2D): buffer 0 (1920x1200, 3840 stride), 02 0b c2 0a 82 0a 42 0a 02 0b c2 0a
CONSUMER: Dump frame 7
CONSUMER: IImage(2D): buffer 0 (1920x1200, 3840 stride), c2 0a 42 0a 82 0a 42 0a 82 0a 42 0b
CONSUMER: Render frame 1
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
^C
nvidia@nvidia-desktop:~/Multimedia_JXAV_R32.3.1/jetson_multimedia_api/samples/13_multi_camera$

dmesg_log.txt (200.2 KB) 13_multi_cam_log.txt (1.7 KB)

fabian.solano · March 19, 2021, 6:25pm

Hi @ClusterImaging

Can you provide the output of the debug trace?

1. Enable debug trace

sudo su

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace

exit

2. Capture with multiple cameras.

3. Print the output of the debug trace with the following command.

sudo cat /sys/kernel/debug/tracing/trace

How are you configuring the mode type for the camera? This is configured through the device tree with the following parameters.

csi_pixel_bit_depth =
dynamic_pixel_bit_depth = 
mode_type =

Are you developing a custom driver for the AR0234? It is a grayscale camera and this is not directly supported by the ISP.

Regards,
Fabian
www.ridgerun.com

ClusterImaging · March 20, 2021, 3:42pm

Hi Fabio,
I have attached the outputlog.txt (2.5 MB) of the debug trace below.

Yes. We have developed a custom driver for AR0234. And we have configured the parameters through device tree.
csi_pixel_bit_depth = “10”;
mode_type = “bayer”;
pixel_phase = “grbg”;

Thanks,
Peri Kasthuri

ShaneCCC · March 22, 2021, 2:38am

Could you verify by argus_camera

ClusterImaging · March 22, 2021, 4:14am

Hi Shane,

We have verified using argus_camera and it is working fine. We have verified using oneShot too and it is also working fine.

With multi_camera app, when no of cameras=1, it is also working fine.

We are facing the issue only when we try to capture using multiple cameras simultaneously using multiple streams as implemented in the multi_camera app.

Using argus_camera app, is there a way to capture from multiple cameras simultaneously?

Thanks,
Kasturi

ShaneCCC · March 22, 2021, 6:39am

You can modify it to capture automatically and launch multiple argus_camera -s -d [sensor-id] to verify it.
I just try by manual switch to sensor id and click the capture button.

ClusterImaging · March 22, 2021, 10:55pm

Hi Shane, Thank you for the reply.
Our goal here is to capture frames or video from multiple cameras simultaneously with minimum lag between cameras. So We are planning to start from the multi_camera app provided in the samples. Please suggest how to proceed for that. Thank you.

ShaneCCC · March 23, 2021, 10:31am

Have a check the multiSensor of argus sample code if match your request.

ClusterImaging · March 23, 2021, 3:53pm

Hi Shane,
multi_camera app captures frames only from one sensor at a time. Our goal is to capture frames from all sensors simultaneously with minimum lag. Hence we think the multi_camera app is best suited for our use-case which captures frames from different sensors using multiple output streams simultaneously.

But unfortunately, we are only able to capture 1 frame from all 8 sensors and unable to capture multiple frames from multiple sensors.
I have attached debug logs of the same.
Thank You

ShaneCCC · March 24, 2021, 4:12am

Have a try below command before run the sample APP.

sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon

ClusterImaging · March 25, 2021, 1:11pm

Hi Shane,
Thank you for the suggestion, I see following error messages with suggested commands. Please help. Thanks a lot for help.

sudo enableCamInfiniteTimeout=1 nvargus-daemon
=== NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections…=== multi_camera[7998]: Connection established (7F8872F1D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module4
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module5
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module6
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module7
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
=== multi_camera[7998]: CameraProvider initialized (0x7f80a6de60)LSC: LSC surface is not based on full res!
LSC: LSC surface is not based on full res!
LSC: LSC surface is not based on full res!
LSC: LSC surface is not based on full res!
LSC: LSC surface is not based on full res!
LSC: LSC surface is not based on full res!
LSC: LSC surface is not based on full res!
LSC: LSC surface is not based on full res!
NvCaptureStatusErrorDecode Stream 5.0 failed: sof_ts 413263536192 eof_ts 13224915937280 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 5.0 failed: sof_ts 413467525248 eof_ts 13231443305472 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 5.0 failed: sof_ts 413671514208 eof_ts 13237970953216 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
PowerServiceCore:handleRequests: timePassed = 2441
NvCaptureStatusErrorDecode Stream 5.1 failed: sof_ts 415728486752 eof_ts 13303794016256 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 5.1 failed: sof_ts 415932476832 eof_ts 13310321698816 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 5.1 failed: sof_ts 416136466912 eof_ts 13316849381376 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
PowerServiceCore:handleRequests: timePassed = 1306
NvCaptureStatusErrorDecode Stream 4.0 failed: sof_ts 417048014528 eof_ts 13346018961408 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 4.0 failed: sof_ts 417252004160 eof_ts 13352546556928 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 4.0 failed: sof_ts 417286060608 eof_ts 13353636362240 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 4.1 failed: sof_ts 417471684064 eof_ts 13359576314880 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 4.1 failed: sof_ts 417675674464 eof_ts 13366104034304 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 4.1 failed: sof_ts 417879664960 eof_ts 13372631693312 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 2.0 failed: sof_ts 417904120288 eof_ts 13373414261760 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 2.0 failed: sof_ts 418108109280 eof_ts 13379941911552 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 2.0 failed: sof_ts 418142165536 eof_ts 13381031714816 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 2.1 failed: sof_ts 418340183872 eof_ts 13387368335360 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 2.1 failed: sof_ts 418544173632 eof_ts 13393895977984 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 2.1 failed: sof_ts 418748163424 eof_ts 13400423632896 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 419202511968 eof_ts 13414962781184 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 4.1 failed: sof_ts 419307597760 eof_ts 13418325528576 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 4.0 failed: sof_ts 419325955520 eof_ts 13418912974848 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 2.1 failed: sof_ts 419360132800 eof_ts 13420006648832 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)
NvCaptureStatusErrorDecode Stream 2.0 failed: sof_ts 419366099392 eof_ts 13420197599232 frame 0 error 14 data 0x00000040
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 916)

ShaneCCC · March 25, 2021, 2:36pm

Are you able to attached the binary for test?

ShaneCCC · March 26, 2021, 4:57am

Yes, the APP binary that you modify for multiple camera. I can try on 6 sensors case.

ClusterImaging · March 26, 2021, 4:05pm

Hi Shane,
I have attached the binary and source code of the modified multi_camera and argus_oneshot apps below. thanks.argus_oneshot (280.2 KB) main_multi_cam_1080p_8cam.cpp (13.5 KB) main_oneshot_2cam.cpp (11.4 KB) multi_camera (461.8 KB)

Source code can be built by renaming the file name to main.cpp

ClusterImaging · March 29, 2021, 9:09pm

Hi Shane

Could you use the binary provided? Kindly update and let us know the possible mitigation to avoid this issue. Thank you

ShaneCCC · March 30, 2021, 6:36am

Attached is the oneshot sample for two sensors. This modify working to create two jpeg.
I am working on the J4.4(r32.5) Also attached the binary here.

argus_oneshot (297.8 KB)
main.cpp (10.5 KB)

ClusterImaging · April 1, 2021, 5:19pm

Hi Shane, Thank you for reply,
Unfortunately the issue we are facing is related to capturing more than one frame from multiple cameras simultaneously. We are able to capture single frame from 8 cameras using multi_camera or modified OneShot apps but we see errors after first frame when we enable to capture more than one camera. Note we are able to stream (several frames) if we only streaming from one camera. Please help.

ClusterImaging · April 2, 2021, 4:24pm

Hi Shane,
I modified the oneshot sample app for two sensors you provided to capture multiple frames and it fails capturing first frame from both sensors.

The error seen is the same error seen with multi_capture sample app which we reported in this thread initally.

Topic		Replies	Views
Camera capture works on v4l2-ctl but hangs with nvarguscamerasrc Jetson AGX Xavier camera , kernel , gstreamer	18	2810	October 18, 2021
Errors when switching nvpmodel with active libargus objects on AGX with >4 cameras Jetson AGX Xavier camera	4	534	October 18, 2021
Xavier AGX Synchronized Capture with 4 Cameras Jetson AGX Xavier camera	31	2477	February 5, 2021
Failed to get capture session interface Jetson Xavier NX camera	9	320	May 22, 2024
Nvargus-daemon crashes when 4 camera 4k@60 capture pipeline stops on AGX Xavier JP4.5.1 Jetson Xavier NX camera , gstreamer , nvbugs	11	1428	October 18, 2021
Video input freeze when opening more than 2 cameras at the same time Jetson AGX Xavier camera	40	1014	October 18, 2021
Nvargus-daemon errors when using nvarguscamerasrc at boot Jetson AGX Xavier camera	2	681	January 19, 2022
CSI Camera acquisition multiple problems when capturing more than (libArgus ; L4T-32.2 ; JetPack 4.2 Jetson TX2	14	2012	October 18, 2021
R35.3.1 AGX ORIN bring up raw14 format，Kernel error Jetson AGX Orin camera	54	1712	October 19, 2023
Camera won't start for the firs time Jetson TX2	17	2576	October 18, 2021

Multi-camera app fails to capture multiple frames on Jetson AGX

Related topics