SIPL simultaneously outputs 8 or more video streams for a certain period of time, and then it will stop working

Robotics & Edge Computing Jetson Systems Jetson Thor
1

Platform: Jetson Thor
Software version: 39.2
Currently, when using the SIPL framework to connect to 8 or more cameras and output streams simultaneously, there is a situation where the program will crash after running for a period of time (approximately 2 minutes). There are no issues when running with a single camera or the same group of 4 cameras. Below is the specific log. Please take a look at it and help us identify the problem.

Output
Sensor0_Out0    Frame rate (fps):               29.9947
Sensor1_Out0    Frame rate (fps):               29.9947
Sensor2_Out0    Frame rate (fps):               29.9947
Sensor3_Out0    Frame rate (fps):               29.9947
Sensor4_Out0    Frame rate (fps):               29.9947
Sensor5_Out0    Frame rate (fps):               29.9947
Sensor6_Out0    Frame rate (fps):               29.9947
Sensor7_Out0    Frame rate (fps):               29.9947
Output
Sensor0_Out0    Frame rate (fps):               29.9943
Sensor1_Out0    Frame rate (fps):               29.9943
Sensor2_Out0    Frame rate (fps):               29.9943
Sensor3_Out0    Frame rate (fps):               29.9943
Sensor4_Out0    Frame rate (fps):               29.9943
Sensor5_Out0    Frame rate (fps):               29.9943
Sensor6_Out0    Frame rate (fps):               29.9943
Sensor7_Out0    Frame rate (fps):               29.9943
nvsipl_camera: ERROR: Pipeline: 6, NOTIF_ERROR_ICP_CAPTURE_FAILURE
nvsipl_camera: ERROR: Pipeline: 7, NOTIF_ERROR_ICP_CAPTURE_FAILURE
nvsipl_camera: ERROR: Pipeline: 3, NOTIF_ERROR_ICP_CAPTURE_FAILURE
nvsipl_camera: ERROR: Pipeline: 0, NOTIF_ERROR_ICP_CAPTURE_FAILURE
nvsipl_camera: ERROR: Pipeline: 4, NOTIF_ERROR_ICP_CAPTURE_FAILURE
nvsipl_camera: ERROR: Pipeline: 5, NOTIF_ERROR_ICP_CAPTURE_FAILURE
nvsipl_camera: ERROR: Pipeline: 1, NOTIF_ERROR_ICP_CAPTURE_FAILURE
nvsipl_camera: ERROR: Pipeline: 2, NOTIF_ERROR_ICP_CAPTURE_FAILURE
Output
Sensor0_Out0    Frame rate (fps):               2.4999
Sensor1_Out0    Frame rate (fps):               2.4999
Sensor2_Out0    Frame rate (fps):               2.4999
Sensor3_Out0    Frame rate (fps):               2.4999
Sensor4_Out0    Frame rate (fps):               2.4999
Sensor5_Out0    Frame rate (fps):               2.4999
Sensor6_Out0    Frame rate (fps):               1.99992
Sensor7_Out0    Frame rate (fps):               2.4999


2026-08-24T17:13:05.136375+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.141949+08:00 robot nvsipl_camera: [TID=6771] Module_id 30 Severity 2 : [PIPELINE (1,0)] CNvFusaRawBlock.cpp:2143: DBG GetOutputBuffer: calling getCaptureStatus
2026-08-24T17:13:05.141968+08:00 robot nvsipl_camera: [TID=6771] Module_id 30 Severity 2 : [PIPELINE (1,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.147240+08:00 robot nvsipl_camera: [TID=6772] Module_id 30 Severity 2 : [PIPELINE (2,0)] CNvFusaRawBlock.cpp:2143: DBG GetOutputBuffer: calling getCaptureStatus
2026-08-24T17:13:05.147262+08:00 robot nvsipl_camera: [TID=6772] Module_id 30 Severity 2 : [PIPELINE (2,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.149814+08:00 robot nvsipl_camera: [TID=6775] Module_id 30 Severity 2 : [PIPELINE (5,0)] CNvFusaRawBlock.cpp:2143: DBG GetOutputBuffer: calling getCaptureStatus
2026-08-24T17:13:05.149858+08:00 robot nvsipl_camera: [TID=6775] Module_id 30 Severity 2 : [PIPELINE (5,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.152591+08:00 robot nvsipl_camera: [TID=6773] Module_id 30 Severity 2 : [PIPELINE (3,0)] CNvFusaRawBlock.cpp:2143: DBG GetOutputBuffer: calling getCaptureStatus
2026-08-24T17:13:05.152634+08:00 robot nvsipl_camera: [TID=6773] Module_id 30 Severity 2 : [PIPELINE (3,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.154843+08:00 robot nvsipl_camera: [TID=6776] Module_id 30 Severity 2 : [PIPELINE (6,0)] CNvFusaRawBlock.cpp:2143: DBG GetOutputBuffer: calling getCaptureStatus
2026-08-24T17:13:05.154891+08:00 robot nvsipl_camera: [TID=6776] Module_id 30 Severity 2 : [PIPELINE (6,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.155369+08:00 robot nvsipl_camera: [TID=6774] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFusaRawBlock.cpp:2143: DBG GetOutputBuffer: calling getCaptureStatus
2026-08-24T17:13:05.155408+08:00 robot nvsipl_camera: [TID=6774] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.159752+08:00 robot nvsipl_camera: [TID=6777] Module_id 30 Severity 2 : [PIPELINE (7,0)] CNvFusaRawBlock.cpp:2143: DBG GetOutputBuffer: calling getCaptureStatus
2026-08-24T17:13:05.159797+08:00 robot nvsipl_camera: [TID=6777] Module_id 30 Severity 2 : [PIPELINE (7,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.169669+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvFusaRawBlock.cpp:2143: DBG GetOutputBuffer: calling getCaptureStatus
2026-08-24T17:13:05.169704+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvFusaRawBlock.cpp:2147: DBG GetOutputBuffer: VI path getCaptureStatus
2026-08-24T17:13:05.175260+08:00 robot nvsipl_camera: [TID=6771] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  83
2026-08-24T17:13:05.175292+08:00 robot nvsipl_camera: [TID=6771] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-24T17:13:05.175316+08:00 robot nvsipl_camera: [TID=6771] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-24T17:13:05.175342+08:00 robot nvsipl_camera: [TID=6771] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-24T17:13:05.175365+08:00 robot nvsipl_camera: [TID=6771] Module_id 30 Severity 2 : [PIPELINE (1,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-24T17:13:05.175394+08:00 robot nvsipl_camera: [TID=6771] Module_id 30 Severity 2 : [PIPELINE (1,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed
2026-08-24T17:13:05.180565+08:00 robot nvsipl_camera: [TID=6772] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  83
2026-08-24T17:13:05.180613+08:00 robot nvsipl_camera: [TID=6772] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-24T17:13:05.180650+08:00 robot nvsipl_camera: [TID=6772] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-24T17:13:05.180687+08:00 robot nvsipl_camera: [TID=6772] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-24T17:13:05.180724+08:00 robot nvsipl_camera: [TID=6772] Module_id 30 Severity 2 : [PIPELINE (2,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-24T17:13:05.180763+08:00 robot nvsipl_camera: [TID=6772] Module_id 30 Severity 2 : [PIPELINE (2,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed
2026-08-24T17:13:05.183121+08:00 robot nvsipl_camera: [TID=6775] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  82
2026-08-24T17:13:05.183171+08:00 robot nvsipl_camera: [TID=6775] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-24T17:13:05.183210+08:00 robot nvsipl_camera: [TID=6775] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-24T17:13:05.183248+08:00 robot nvsipl_camera: [TID=6775] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-24T17:13:05.183285+08:00 robot nvsipl_camera: [TID=6775] Module_id 30 Severity 2 : [PIPELINE (5,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-24T17:13:05.183322+08:00 robot nvsipl_camera: [TID=6775] Module_id 30 Severity 2 : [PIPELINE (5,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed
2026-08-24T17:13:05.185916+08:00 robot nvsipl_camera: [TID=6773] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  83
2026-08-24T17:13:05.185965+08:00 robot nvsipl_camera: [TID=6773] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-24T17:13:05.186002+08:00 robot nvsipl_camera: [TID=6773] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-24T17:13:05.186039+08:00 robot nvsipl_camera: [TID=6773] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-24T17:13:05.186075+08:00 robot nvsipl_camera: [TID=6773] Module_id 30 Severity 2 : [PIPELINE (3,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-24T17:13:05.186113+08:00 robot nvsipl_camera: [TID=6773] Module_id 30 Severity 2 : [PIPELINE (3,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed
2026-08-24T17:13:05.188138+08:00 robot nvsipl_camera: [TID=6776] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  83
2026-08-24T17:13:05.188184+08:00 robot nvsipl_camera: [TID=6776] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-24T17:13:05.188221+08:00 robot nvsipl_camera: [TID=6776] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-24T17:13:05.188259+08:00 robot nvsipl_camera: [TID=6776] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-24T17:13:05.188295+08:00 robot nvsipl_camera: [TID=6776] Module_id 30 Severity 2 : [PIPELINE (6,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-24T17:13:05.188331+08:00 robot nvsipl_camera: [TID=6776] Module_id 30 Severity 2 : [PIPELINE (6,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed
2026-08-24T17:13:05.188664+08:00 robot nvsipl_camera: [TID=6774] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  83
2026-08-24T17:13:05.188709+08:00 robot nvsipl_camera: [TID=6774] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-24T17:13:05.188748+08:00 robot nvsipl_camera: [TID=6774] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-24T17:13:05.188785+08:00 robot nvsipl_camera: [TID=6774] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-24T17:13:05.188821+08:00 robot nvsipl_camera: [TID=6774] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-24T17:13:05.188858+08:00 robot nvsipl_camera: [TID=6774] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed
2026-08-24T17:13:05.193060+08:00 robot nvsipl_camera: [TID=6777] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  83
2026-08-24T17:13:05.193104+08:00 robot nvsipl_camera: [TID=6777] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-24T17:13:05.193147+08:00 robot nvsipl_camera: [TID=6777] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-24T17:13:05.193190+08:00 robot nvsipl_camera: [TID=6777] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-24T17:13:05.193241+08:00 robot nvsipl_camera: [TID=6777] Module_id 30 Severity 2 : [PIPELINE (7,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-24T17:13:05.193278+08:00 robot nvsipl_camera: [TID=6777] Module_id 30 Severity 2 : [PIPELINE (7,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed
2026-08-24T17:13:05.203015+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  84
2026-08-24T17:13:05.203062+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-24T17:13:05.203101+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-24T17:13:05.203139+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-24T17:13:05.203178+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-24T17:13:05.203218+08:00 robot nvsipl_camera: [TID=6770] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed
2026-08-24T17:13:05.941528+08:00 robot kernel: arm-smmu-v3 8106000000.iommu: CMD_SYNC timeout at 0x00000c18 [hwprod 0x00000c18, hwcons 0x03000b5d]
2026-08-24T17:13:06.436162+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE 0] Post-reset channel info: hwChanId= 0, chanMask=0x0, vi2Mask=0x800000000
2026-08-24T17:13:06.436233+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : CNvMQueue.hpp 151
2026-08-24T17:13:06.436277+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : Queue is empty
2026-08-24T17:13:06.436323+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvFusaRawBlock.cpp:2194: Getting element from queue failed!
2026-08-24T17:13:06.436361+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvFSensorPipeline.cpp:2474: Capture block get output buffer without capture status failed
2026-08-24T17:13:06.436402+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvFSensorPipeline.cpp:2701: Failed to stop processing
2026-08-24T17:13:06.436442+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (0,0)] CNvMCamera.cpp:5039: Pipeline stop failed for pipeline
2026-08-24T17:13:06.436897+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE 1] Post-reset channel info: hwChanId= 1, chanMask=0x0, vi2Mask=0x400000000
2026-08-24T17:13:06.436953+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : CNvMQueue.hpp 151
2026-08-24T17:13:06.437004+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : Queue is empty
2026-08-24T17:13:06.437058+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (1,0)] CNvFusaRawBlock.cpp:2194: Getting element from queue failed!
2026-08-24T17:13:06.437097+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (1,0)] CNvFSensorPipeline.cpp:2474: Capture block get output buffer without capture status failed
2026-08-24T17:13:06.437135+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (1,0)] CNvFSensorPipeline.cpp:2701: Failed to stop processing
2026-08-24T17:13:06.437172+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (1,0)] CNvMCamera.cpp:5039: Pipeline stop failed for pipeline
2026-08-24T17:13:06.437891+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE 2] Post-reset channel info: hwChanId= 2, chanMask=0x0, vi2Mask=0x200000000
2026-08-24T17:13:06.437946+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : CNvMQueue.hpp 151
2026-08-24T17:13:06.437991+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : Queue is empty
2026-08-24T17:13:06.438043+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (2,0)] CNvFusaRawBlock.cpp:2194: Getting element from queue failed!
2026-08-24T17:13:06.438081+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (2,0)] CNvFSensorPipeline.cpp:2474: Capture block get output buffer without capture status failed
2026-08-24T17:13:06.438118+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (2,0)] CNvFSensorPipeline.cpp:2701: Failed to stop processing
2026-08-24T17:13:06.438160+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (2,0)] CNvMCamera.cpp:5039: Pipeline stop failed for pipeline
2026-08-24T17:13:06.438884+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE 3] Post-reset channel info: hwChanId= 3, chanMask=0x0, vi2Mask=0x100000000
2026-08-24T17:13:06.438937+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : CNvMQueue.hpp 151
2026-08-24T17:13:06.438978+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : Queue is empty
2026-08-24T17:13:06.439014+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (3,0)] CNvFusaRawBlock.cpp:2194: Getting element from queue failed!
2026-08-24T17:13:06.439056+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (3,0)] CNvFSensorPipeline.cpp:2474: Capture block get output buffer without capture status failed
2026-08-24T17:13:06.440274+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (3,0)] CNvFSensorPipeline.cpp:2701: Failed to stop processing
2026-08-24T17:13:06.440317+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (3,0)] CNvMCamera.cpp:5039: Pipeline stop failed for pipeline
2026-08-24T17:13:06.440886+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE 4] Post-reset channel info: hwChanId= 4, chanMask=0x0, vi2Mask=0x80000000
2026-08-24T17:13:06.440934+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : CNvMQueue.hpp 151
2026-08-24T17:13:06.440978+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : Queue is empty
2026-08-24T17:13:06.441030+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFusaRawBlock.cpp:2194: Getting element from queue failed!
2026-08-24T17:13:06.441067+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFSensorPipeline.cpp:2474: Capture block get output buffer without capture status failed
2026-08-24T17:13:06.441104+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFSensorPipeline.cpp:2701: Failed to stop processing
2026-08-24T17:13:06.441141+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvMCamera.cpp:5039: Pipeline stop failed for pipeline
2026-08-24T17:13:06.441887+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE 5] Post-reset channel info: hwChanId= 5, chanMask=0x0, vi2Mask=0x40000000
2026-08-24T17:13:06.441937+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : CNvMQueue.hpp 151
2026-08-24T17:13:06.441982+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : Queue is empty
2026-08-24T17:13:06.442034+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (5,0)] CNvFusaRawBlock.cpp:2194: Getting element from queue failed!
2026-08-24T17:13:06.442071+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (5,0)] CNvFSensorPipeline.cpp:2474: Capture block get output buffer without capture status failed
2026-08-24T17:13:06.442109+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (5,0)] CNvFSensorPipeline.cpp:2701: Failed to stop processing
2026-08-24T17:13:06.442149+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (5,0)] CNvMCamera.cpp:5039: Pipeline stop failed for pipeline
2026-08-24T17:13:06.442884+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE 6] Post-reset channel info: hwChanId= 6, chanMask=0x0, vi2Mask=0x20000000
2026-08-24T17:13:06.442931+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : CNvMQueue.hpp 151
2026-08-24T17:13:06.442975+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : Queue is empty
2026-08-24T17:13:06.443027+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (6,0)] CNvFusaRawBlock.cpp:2194: Getting element from queue failed!
2026-08-24T17:13:06.443063+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (6,0)] CNvFSensorPipeline.cpp:2474: Capture block get output buffer without capture status failed
2026-08-24T17:13:06.443106+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (6,0)] CNvFSensorPipeline.cpp:2701: Failed to stop processing
2026-08-24T17:13:06.443144+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (6,0)] CNvMCamera.cpp:5039: Pipeline stop failed for pipeline
2026-08-24T17:13:06.443882+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE 7] Post-reset channel info: hwChanId= 7, chanMask=0x0, vi2Mask=0x10000000
2026-08-24T17:13:06.443930+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : CNvMQueue.hpp 151
2026-08-24T17:13:06.443966+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : Queue is empty
2026-08-24T17:13:06.444001+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (7,0)] CNvFusaRawBlock.cpp:2194: Getting element from queue failed!
2026-08-24T17:13:06.444038+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (7,0)] CNvFSensorPipeline.cpp:2474: Capture block get output buffer without capture status failed
2026-08-24T17:13:06.444670+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (7,0)] CNvFSensorPipeline.cpp:2701: Failed to stop processing
2026-08-24T17:13:06.444711+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 2 : [PIPELINE (7,0)] CNvMCamera.cpp:5039: Pipeline stop failed for pipeline
2026-08-24T17:13:06.444751+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 5 : Stopping Camera HAL path
2026-08-24T17:13:06.444790+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 6 : [CameraHAL] DBUG: CameraModuleDevice.cpp:221:  Stopping streaming for device Module_3_link_0
2026-08-24T17:13:06.444829+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 5 : [Module_3_link_0] INFO: R0SIM031Module.cpp:103:  R0SIM031: StopStreaming called
2026-08-24T17:13:06.444870+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 5 : [Module_3_link_0] INFO: ISX031.cpp:185:  ISX031 StopStreaming — SKIP ALL REGISTERS
2026-08-24T17:13:06.444910+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 6 : [CameraHAL] DBUG: GmslDeserializerDevice.cpp:361:  Stopping streaming for device Deser_3
2026-08-24T17:13:06.444949+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 6 : [Deser_3] DBUG: MAX967XX.cpp:725:  StopStreaming called
2026-08-24T17:13:06.444988+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 6 : [CameraHAL] DBUG: CameraModuleDevice.cpp:221:  Stopping streaming for device Module_3_link_1
2026-08-24T17:13:06.445028+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 5 : [Module_3_link_1] INFO: R0SIM031Module.cpp:103:  R0SIM031: StopStreaming called
2026-08-24T17:13:06.445064+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 5 : [Module_3_link_1] INFO: ISX031.cpp:185:  ISX031 StopStreaming — SKIP ALL REGISTERS
2026-08-24T17:13:06.445105+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 6 : [CameraHAL] DBUG: GmslDeserializerDevice.cpp:361:  Stopping streaming for device Deser_3
2026-08-24T17:13:06.445141+08:00 robot nvsipl_camera: [TID=6737] Module_id 30 Severity 6 : [Deser_3] DBUG: MAX967XX.cpp:725:  StopStreaming called
3

hello Dio.Brando,

let me have confirmation, were they CoE camera or GMSL camera?

4

GMSL camera

5

hello Dio.Brando,

unfortunately, we don’t have 8-cam long-run test case on Jetson Thor for reference.
please provide the exact nvsipl_camera command, platform JSON, camera module/deserializer topology, enabled outputs, and full kernel/SIPL log from application start.

6

Hi.JerryChang

OK. After multiple tests, it was found that the crash issue doesn’t occur only when using 8 or more lanes. Even when I tested with 4 lanes, the same problem occurred, and it typically appeared after about an hour of testing. I also tested with 3 lanes, but since it didn’t recur after 3 hours of testing, we didn’t continue the test. Below are my test commands, JSON files, and the specific logs after the failure.

cmd:

sudo nvsipl_camera -t ./ISX031_12ch.json -c ISX031_12CH -m "0000 1111 0000" --enable-camera-hal -s -Z -R012

json(The GPIO pins don’t need to be concerned about. I changed the other unused GPIO pins to ensure that it would be convenient for debugging when the power was on.):

{
    "platformTransportSettings": [
        {
            "name": "jetson-thor",
            "description": "Thor - Jetson AGX Devkit (Thor)",
            "boardIdPrefix": "p3840",
            "enableMasks": ["0x1111", "0x1111", "0x1111", "0x1111", "0x1111"],
            "transportSettings": [
                {
                    "name": "transportSettings_jetson-thor_AB",
                    "type": "GMSL",
                    "description": "GMSL2 transport settings for CSI-AB (MAX96712 → Tegra MIPI)",
                    "csiPort": "csi-ab",
                    "phyMode": "dphy",
                    "deserInfo": {
                        "name": "Max96712GmslDeserializer",
                        "i2cAddress": "0x29"
                    },
                    "powerControlInfo": {
                        "deserializerInfo": {
                            "name": "TegraDeserPowerDriver",
			    "gpioPinIndex":65538
                        }
                    },
                    "i2cDevice": 12,
                    "desI2CPort": 0,
                    "groupInitProg": true
                },
                {
                    "name": "transportSettings_jetson-thor_CD",
                    "type": "GMSL",
                    "description": "GMSL2 transport settings for CSI-CD (MAX96712 → Tegra MIPI)",
                    "csiPort": "csi-cd",
                    "phyMode": "dphy",
                    "deserInfo": {
                        "name": "Max96712GmslDeserializer",
                        "i2cAddress": "0x29"
                    },
                    "powerControlInfo": {
                        "deserializerInfo": {
                            "name": "TegraDeserPowerDriver",
			    "gpioPinIndex":131074
                        }
                    },
                    "i2cDevice": 9,
                    "desI2CPort": 0,
                    "groupInitProg": true
                },
                {
                    "name": "transportSettings_jetson-thor_gh",
                    "type": "GMSL",
                    "description": "GMSL2 transport settings for CSI-CD (MAX96712 → Tegra MIPI)",
                    "csiPort": "csi-gh",
                    "phyMode": "dphy",
                    "deserInfo": {
                        "name": "Max96712GmslDeserializer",
                        "i2cAddress": "0x29"
                    },
                    "powerControlInfo": {
                        "deserializerInfo": {
                            "name": "TegraDeserPowerDriver",
			        "gpioPinIndex":196610
                        }
                    },
                    "i2cDevice": 3,
                    "desI2CPort": 0,
                    "groupInitProg": true
                }
            ]
        }
    ],
    "cameraConfigs": [
        {
            "name": "ISX031_12CH",
            "type": "GMSL",
            "description": "SGX-YUV-GMSL 4 cameras: 4x CSI-AB + 4x CSI-CD",
            "cameraModules": [
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 0,
                    "transportIndex": 2
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 1,
                    "transportIndex": 2
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 2,
                    "transportIndex": 2
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 3,
                    "transportIndex": 2
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 0,
                    "transportIndex": 1
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 1,
                    "transportIndex": 1
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 2,
                    "transportIndex": 1
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 3,
                    "transportIndex": 1
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 0,
                    "transportIndex": 0
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 1,
                    "transportIndex": 0
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 2,
                    "transportIndex": 0
                },
                {
                    "name": "R0SIM031S5RU1197NB3",
                    "linkIndex": 3,
                    "transportIndex": 0
                }
            ],
            "transportSettings": [
                {
                    "name": "transportSettings_jetson-thor_AB",
                    "type": "GMSL",
                    "csiPort": "csi-ab",
                    "phyMode": "dphy",
                    "desTxPort": 1,
                    "deserInfo": {
                        "name": "Max96712GmslDeserializer",
                        "i2cAddress": "0x29"
                    },
                    "powerControlInfo": {
                        "deserializerInfo": {
                            "name": "TegraDeserPowerDriver",
                            "gpioPinIndex": 65538
                        }
                    },
                    "i2cDevice": 12,
                    "desI2CPort": 0,
                    "groupInitProg": false
                },
                {
                    "name": "transportSettings_jetson-thor_CD",
                    "type": "GMSL",
                    "csiPort": "csi-cd",
                    "phyMode": "dphy",
                    "desTxPort": 1,
                    "deserInfo": {
                        "name": "Max96712GmslDeserializer",
                        "i2cAddress": "0x29"
                    },
                    "powerControlInfo": {
                        "deserializerInfo": {
                            "name": "TegraDeserPowerDriver",
                            "gpioPinIndex": 131074
                        }
                    },
                    "i2cDevice": 9,
                    "desI2CPort": 0,
                    "groupInitProg": true
                },
                {
                    "name": "transportSettings_jetson-thor_gh",
                    "type": "GMSL",
                    "csiPort": "csi-gh",
                    "phyMode": "dphy",
                    "desTxPort": 1,
                    "deserInfo": {
                        "name": "Max96712GmslDeserializer",
                        "i2cAddress": "0x29"
                    },
                    "powerControlInfo": {
                        "deserializerInfo": {
                            "name": "TegraDeserPowerDriver",
                            "gpioPinIndex": 196610
                        }
                    },
                    "i2cDevice": 3,
                    "desI2CPort": 0,
                    "groupInitProg": true
                }
            ]
        },
        {
            "name": "R0SIM031S5RU1197NB3",
            "type": "GMSL",
            "description": "WHST R0SIM031 — Sony ISX031 (YUV422) + MAX96717F, 1920x1536 @ 30fps, Built-in ISP",

            "moduleDriverName": "R0SIM031S5RU1197NB3",

            "serInfo": {
                "name": "MAX96717F",
                "i2cAddress": "0x40"
            },

            "linkMode": "LINK_MODE_GMSL2_3GBPS",
            "fsyncMode": "external",
	        "linkIndex": 0,

            "mipiSettings": {
                "dphyRate": 2500000,
                "phyMode": "dphy"
            },

            "sensorInfo": [
                {
                    "name": "ISX031",
                    "description": "Sony ISX031-AAQV-W — Built-in ISP, YUV422 output, Sub-pixel HDR",
                    "i2cAddress": "0x1A",
                    "numContext": 1,
                    "virtualChannels": [
                        {
                            "cfa": "uyvy",
                            "inputFormat": "422",
                            "width": 1920,
                            "height": 1536,
                            "fps": 30.0,
                            "embeddedTopLines": 0,
                            "embeddedBottomLines": 0,
                            "isEmbeddedDataTypeEnabled":false 
                        }
                    ]
                }
            ],

            "isEEPROMSupported": false
        }
    ],
    "imageSensors": [
        {
            "name": "ISX031",
            "description": "Sony ISX031-AAQV-W Sensor (Built-in ISP, YUV422 output)",
            "i2cAddress": "0x1A",
            "virtualChannels": [
                {
                    "cfa": "uyvy",
                    "embeddedTopLines": 0,
                    "embeddedBottomLines": 0,
                    "inputFormat": "422",
                    "width": 1920,
                    "height": 1536,
                    "fps": 30.0,
                    "isEmbeddedDataTypeEnabled":true 
                }
            ],
            "isTriggerModeEnabled": true,
            "isTPGEnabled": false,
            "useCDIv2API": true
        }
    ],
    "serializers": [
        {
            "name": "MAX96717F",
            "description": "Maxim MAX96717F GMSL2 Serializer",
            "i2cAddress": "0x40"
        }
    ],

    "deserializers": [
        {
            "name": "Max96712GmslDeserializer",
            "description": "Maxim MAX96712 GMSL2 Deserializer",
            "i2cAddress": "0x29"
        }
    ],
    "cameraModules": [
        {
            "name": "R0SIM031S5RU1197NB3",
	        "linkIndex": 0,
            "description": "Sony ISX031 module - MAX96717F serializer",
            "serializer": {
                "name": "MAX96717F"
            },
            "sensors": [
                {
                    "name": "ISX031",
                    "i2cAddress": "0x1A",
                    "virtualChannels": [
                        {
                            "cfa": "uyvy",
                            "inputFormat": "422",
                            "width": 1920,
                            "height": 1536
                        }
                    ]
                }
            ]
        }
    ]
    
}

log:The log data is too large. I will send it as an attachment.

runlog.txt (322.9 KB)

fail_syslog.txt (168.2 KB)

dmesg_log.txt (129.6 KB)

7

hello Dio.Brando,

thanks for sharing the logs.

2026-08-25T14:40:34.891833+08:00 robot nvsipl_camera: [TID=5780] Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame:  20155
2026-08-25T14:40:34.891873+08:00 robot nvsipl_camera: [TID=5780] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
2026-08-25T14:40:34.891914+08:00 robot nvsipl_camera: [TID=5780] Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
2026-08-25T14:40:34.891954+08:00 robot nvsipl_camera: [TID=5780] Module_id 30 Severity 2 : corr_err: notifyBits: 00000000
2026-08-25T14:40:34.891993+08:00 robot nvsipl_camera: [TID=5780] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFusaRawBlock.cpp:2072: Getting capture status failed!
2026-08-25T14:40:34.892035+08:00 robot nvsipl_camera: [TID=5780] Module_id 30 Severity 2 : [PIPELINE (4,0)] CNvFSensorPipeline.cpp:3016: Capture block get output buffer failed

these logs mean the VI received a frame, but its ATOMP memory-output block could not finish writing the frame to system memory.
according to ATOMP_FRAME_TRUNCATED: Frame not finished, the frame is corrupted/incomplete and SIPL rejects its buffer.
eventually, it’s SensorPipeline propagates the failure as NOTIF_ERROR_ICP_CAPTURE_FAILURE, that’s why it terminates the nvsipl_camera application.

we may need to have issue narrow down,
since ATOMP_FRAME_TRUNCATED may due to insufficient available memory bandwidth.
let’s give it another test by lower the frame-rate (such as 15-fps) to repeat the test.
lower frame rate should reduce average memory bandwidth roughly proportionally and may reduce the probability of ATOMP_FRAME_TRUNCATED.

8

Thank you. After I modified the “performance mode”, it didn’t occur again.

9

hello Dio.Brando,

just for confirmation, did you toggle the Power Modes (Profiles) to MaxN to verify 8-cam at 30-fps?

10

I checked it using the command “sudo nvpmodel -q”, and it was indeed “MaxN”.

11

hello Dio.Brando,

all right, thanks for confirmation, let’s close this discussion thread.
you may start a new forum topic for new issues.