What causes the syncpoint progressing slowly and timeout

liuq1 · July 5, 2023, 2:33pm

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
[√] DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
[√] Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
[√] NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
[√] 1.9.1.10844
other

Host Machine Version
[√] native Ubuntu 18.04
other

We run four cameras in one xavier, sometimes we see the following debug log .
At the same time, it seems that the process in vic or encoder costs more time, even report the timeout error.
What would case this problem?
Thanks.

Jul  4 16:59:51 xavier1 kernel: [ 8870.246912] host1x 13e10000.host1x: camera_node: syncpoint id 12 (camera_node) progressing slowly 510929, timeout=-1
Jul  4 16:59:51 xavier1 kernel: [ 8870.247021] host1x 13e10000.host1x: SIPL_ICP_ISP_4: syncpoint id 12 (camera_node) progressing slowly 510929, timeout=519311586
Jul  4 16:59:52 xavier1 kernel: [ 8870.502918] host1x 13e10000.host1x: SIPL_ICP_ISP_6: syncpoint id 12 (camera_node) progressing slowly 510932, timeout=519311586

Jul  4 16:59:55 xavier1 kernel: [ 8874.343049] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 12 (camera_node) clientid 2, HW thresh 510928, done 510930
Jul  4 16:59:55 xavier1 kernel: [ 8874.343059] ---- mlocks ----
Jul  4 16:59:55 xavier1 kernel: [ 8874.343064]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343068] ---- syncpts ----
Jul  4 16:59:55 xavier1 kernel: [ 8874.343085] id 11 (camera_node) min 255976 max 255976 refs 2 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343118] id 12 (camera_node) min 510928 max 510933 refs 5 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343124] id 13 (154c0000.nvenc_camera_node_0) min 342312 max 342312 refs 1 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343129] id 14 (camera_node) min 511542 max 511542 refs 2 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343135] id 15 (15a80000.nvenc1_camera_node_0) min 342724 max 342724 refs 1 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343141] id 16 (camera_node) min 255676 max 255676 refs 2 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343222] id 17 (camera_node) min 255982 max 255982 refs 2 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343228] id 18 (camera_node) min 255673 max 255673 refs 2 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343235] id 19 (progress) min 767542 max 0 refs 1 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343240] id 20 (progress) min 766653 max 0 refs 1 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343246] id 21 (progress) min 767545 max 0 refs 1 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343251] id 22 (progress) min 766647 max 0 refs 1 (previous client : )
Jul  4 16:59:55 xavier1 kernel: [ 8874.343600]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343603] ---- channels ----
Jul  4 16:59:55 xavier1 kernel: [ 8874.343611]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343611] channel 1 - 15340000.vic
Jul  4 16:59:55 xavier1 kernel: [ 8874.343611]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343614] NvHost basic channel registers:
Jul  4 16:59:55 xavier1 kernel: [ 8874.343620] CMDFIFO_STAT_0:  00002040
Jul  4 16:59:55 xavier1 kernel: [ 8874.343624] CMDFIFO_RDATA_0: 00000041
Jul  4 16:59:55 xavier1 kernel: [ 8874.343629] CMDP_OFFSET_0:   00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343633] CMDP_CLASS_0:    00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343637] CHANNELSTAT_0:   00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343644]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343644] ffffffc703fa7000: JOB, syncpt_id=12, syncpt_val=510930, first_get=000006a8, timeout=10000, num_slots=14
Jul  4 16:59:55 xavier1 kernel: [ 8874.343691]     GATHER at fff90000+64a0, 11 words
Jul  4 16:59:55 xavier1 kernel: [ 8874.343696] 304e0005
Jul  4 16:59:55 xavier1 kernel: [ 8874.343700] 0007cbbc
Jul  4 16:59:55 xavier1 kernel: [ 8874.343703] 0000000c
Jul  4 16:59:55 xavier1 kernel: [ 8874.343707] 304e0005
Jul  4 16:59:55 xavier1 kernel: [ 8874.343711] 0005391b
Jul  4 16:59:55 xavier1 kernel: [ 8874.343714] 0000000d
Jul  4 16:59:55 xavier1 kernel: [ 8874.343718] 304e0005
Jul  4 16:59:55 xavier1 kernel: [ 8874.343722] 000bb2ae
Jul  4 16:59:55 xavier1 kernel: [ 8874.343726] 00000014
Jul  4 16:59:55 xavier1 kernel: [ 8874.343729] 20000001
Jul  4 16:59:55 xavier1 kernel: [ 8874.343733] 0000000c
Jul  4 16:59:55 xavier1 kernel: [ 8874.343736]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343742]     GATHER at fff90000+64cc, 35 words
Jul  4 16:59:55 xavier1 kernel: [ 8874.343746] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343754] 00000080
Jul  4 16:59:55 xavier1 kernel: [ 8874.343758] 00000001
Jul  4 16:59:55 xavier1 kernel: [ 8874.343762] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343766] 00000480
Jul  4 16:59:55 xavier1 kernel: [ 8874.343769] 3fed0000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343773] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343777] 000001c8
Jul  4 16:59:55 xavier1 kernel: [ 8874.343781] bfe20000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343853] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343857] 000001c9
Jul  4 16:59:55 xavier1 kernel: [ 8874.343861] bfe27c00
Jul  4 16:59:55 xavier1 kernel: [ 8874.343864] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343868] 0000020b
Jul  4 16:59:55 xavier1 kernel: [ 8874.343871] 3fff7f70
Jul  4 16:59:55 xavier1 kernel: [ 8874.343875] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343878] 0000020c
Jul  4 16:59:55 xavier1 kernel: [ 8874.343882] 3fff7db0
Jul  4 16:59:55 xavier1 kernel: [ 8874.343886] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343889] 000001cc
Jul  4 16:59:55 xavier1 kernel: [ 8874.343893] 00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343896] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343900] 000001c2
Jul  4 16:59:55 xavier1 kernel: [ 8874.343904] 3fff7c60
Jul  4 16:59:55 xavier1 kernel: [ 8874.343907] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343911] 000001c3
Jul  4 16:59:55 xavier1 kernel: [ 8874.343915] 3fff7f80
Jul  4 16:59:55 xavier1 kernel: [ 8874.343918] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343922] 000001c1
Jul  4 16:59:55 xavier1 kernel: [ 8874.343926] 00b90100
Jul  4 16:59:55 xavier1 kernel: [ 8874.343929] 10100002
Jul  4 16:59:55 xavier1 kernel: [ 8874.343933] 000000c0
Jul  4 16:59:55 xavier1 kernel: [ 8874.343937] 00000100
Jul  4 16:59:55 xavier1 kernel: [ 8874.343940] 20000001
Jul  4 16:59:55 xavier1 kernel: [ 8874.343944] 0000040c
Jul  4 16:59:55 xavier1 kernel: [ 8874.343947]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343951]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343966]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343966] ---- host general irq ----
Jul  4 16:59:55 xavier1 kernel: [ 8874.343966]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343969] sync_intc0mask = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343973] sync_intmask = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343976]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343976] ---- host syncpt irq mask ----
Jul  4 16:59:55 xavier1 kernel: [ 8874.343976]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343979]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343979] ---- host syncpt irq status ----
Jul  4 16:59:55 xavier1 kernel: [ 8874.343979]
Jul  4 16:59:55 xavier1 kernel: [ 8874.343984] syncpt_thresh_cpu0_int_status(0) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343988] syncpt_thresh_cpu0_int_status(1) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343992] syncpt_thresh_cpu0_int_status(2) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.343995] syncpt_thresh_cpu0_int_status(3) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344000] syncpt_thresh_cpu0_int_status(4) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344004] syncpt_thresh_cpu0_int_status(5) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344008] syncpt_thresh_cpu0_int_status(6) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344012] syncpt_thresh_cpu0_int_status(7) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344016] syncpt_thresh_cpu0_int_status(8) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344020] syncpt_thresh_cpu0_int_status(9) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344024] syncpt_thresh_cpu0_int_status(10) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344028] syncpt_thresh_cpu0_int_status(11) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344032] syncpt_thresh_cpu0_int_status(12) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344036] syncpt_thresh_cpu0_int_status(13) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344040] syncpt_thresh_cpu0_int_status(14) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344044] syncpt_thresh_cpu0_int_status(15) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344048] syncpt_thresh_cpu0_int_status(16) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344052] syncpt_thresh_cpu0_int_status(17) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344057] syncpt_thresh_cpu0_int_status(18) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344062] syncpt_thresh_cpu0_int_status(19) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344066] syncpt_thresh_cpu0_int_status(20) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344070] syncpt_thresh_cpu0_int_status(21) = 0x00000000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344189] vhost-client 15340000.vic:     SYNCPT_ID   12
Jul  4 16:59:55 xavier1 kernel: [ 8874.344194] vhost-client 15340000.vic:     SYNCPT_VAL  510930
Jul  4 16:59:55 xavier1 kernel: [ 8874.344198] vhost-client 15340000.vic:     FIRST_GET   0x6a8
Jul  4 16:59:55 xavier1 kernel: [ 8874.344202] vhost-client 15340000.vic:     TIMEOUT     10000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344206] vhost-client 15340000.vic:     NUM_SLOTS   14
Jul  4 16:59:55 xavier1 kernel: [ 8874.344209] vhost-client 15340000.vic:     NUM_HANDLES 7
Jul  4 16:59:55 xavier1 kernel: [ 8874.344216] vhost-client 15340000.vic:     SYNCPT_ID   12
Jul  4 16:59:55 xavier1 kernel: [ 8874.344220] vhost-client 15340000.vic:     SYNCPT_VAL  510933
Jul  4 16:59:55 xavier1 kernel: [ 8874.344224] vhost-client 15340000.vic:     FIRST_GET   0x718
Jul  4 16:59:55 xavier1 kernel: [ 8874.344228] vhost-client 15340000.vic:     TIMEOUT     10000
Jul  4 16:59:55 xavier1 kernel: [ 8874.344231] vhost-client 15340000.vic:     NUM_SLOTS   14
Jul  4 16:59:55 xavier1 kernel: [ 8874.344235] vhost-client 15340000.vic:     NUM_HANDLES 7

VickNV · July 5, 2023, 2:58pm

I haven’t seen any reports from other developers regarding this issue. In order to better understand and address it, could you please try replicating the issue using a camera sample application? Additionally, it would be helpful to know if you are using any specific cameras listed on the ecosystem page: DRIVE AGX Xavier Sensors & Accessories | NVIDIA Developer

liuq1 · August 24, 2023, 1:21am

The error is reported by the function nvhost_syncpt_wait_timeout, do you know what problem will report this error. We are currently unable to stably reproduce this issue, so we want to look for clues from the code.

VickNV · August 24, 2023, 5:02am

Please answer my previous questions below and provide more context:

system · October 9, 2023, 3:34pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Sekonix SF3324: Driveworks sample camera failed DRIVE AGX Xavier General driveworks-cameras	7	536	May 9, 2024
NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR Jetson Xavier NX camera , kernel	4	646	January 19, 2023
IMX390 Rev 6 DRIVE AGX Xavier General camera , sony-imx390 , driveos-nvmedia	16	2689	October 12, 2021
Can't take pictures on some cameras using nvarguscamerasrc Jetson Xavier NX camera	19	910	April 12, 2023
Driveworks sample camera emulation with AR0231 failed DRIVE AGX Xavier General driveworks-cameras	9	1030	October 5, 2023
Xavier AGX reboots sometimes after camera frame capturing error Jetson AGX Xavier camera	17	1617	October 18, 2021
VI performance issue with 6 AR0233 cameras on roscube-x? Jetson AGX Xavier camera	31	2499	October 3, 2021
Recorder application failing DRIVE AGX Xavier General driveworks-recording	6	753	February 24, 2022
Again MIPI camera not working: Failed to create CameraProvider Jetson Nano camera	5	1435	October 15, 2021
Intermittent Camera Error: EngineStatusError (2), Detailed Error Code: 00000200 DRIVE AGX Xavier General driveworks-recording	3	1196	October 12, 2021

What causes the syncpoint progressing slowly and timeout

Related topics