ezaro
September 26, 2018, 6:24pm
1
Hello,
I am experiencing an issue in which an application which is able to stream and capture 3 unsynchronized streams from 3 camera device without any issue. However, when using a hardware synchronization suddenly I start receiving PXL_SOF syncpt timeout errors on streams. It appears almost as if there is some sort of bandwidth limitation although this occurs even at low resolution
I am attempting to stream 3 cameras at 2328x1748 at 45fps.
Once I begin streaming with hardware synchronized cameras I begin seeing the following in dmesg and the application layer begins failing to receive frames intermittently.
270.971785] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
I have recreated the same behavior using the jetpack/samples/v4l2cuda sample program and running multiple instances. In the unsynchronized camera instance the different instances are able to retrieve frames without dropping or errors in dmesg. However, once the synchronization is set the same behavior as noted above begins appearing. I would appreciate any help in diagnosing this issue. Thank you.
You may have a look to this post and post your case’s output.
ezaro
September 26, 2018, 8:41pm
3
Thanks for the response. However, the resolution of that thread revolved around an incorrectly defined size. In my case, I can stream all 3 cameras at the same time with no issues and frame output is as expected.
I only experience these issues when I use the IMX298 XVS synchronization. I have validated using an oscilloscope that all cameras are streaming frames in sync. For some reason the fact that the cameras are synced seems to be causing some new error to occur. Perhaps some type of bandwidth issue due to the fact that synchronizing causes the hardware to receive 3 frames instantaneously rather than at slight phase difference when unsynchronized? But at the frame rate and resolution I am trying to stream at should be well below the maximum rate as defined in the TX2 reference manual.
I was not meaning about the solution of that thread, but about how to enable tracing.
ezaro
September 26, 2018, 10:55pm
5
Oh! Sorry I misunderstood, yes that is a very helpful suggestion. Here is my output from that test below.
root@tegra-ubuntu:/home/ubuntu/zaro-workspace/Lab-App-V2/build/lab-app-v2# echo 1 > /sys/kernel/debug/tracing/tracing_on
root@tegra-ubuntu:/home/ubuntu/zaro-workspace/Lab-App-V2/build/lab-app-v2# echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
root@tegra-ubuntu:/home/ubuntu/zaro-workspace/Lab-App-V2/build/lab-app-v2# echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
root@tegra-ubuntu:/home/ubuntu/zaro-workspace/Lab-App-V2/build/lab-app-v2# echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
root@tegra-ubuntu:/home/ubuntu/zaro-workspace/Lab-App-V2/build/lab-app-v2# echo 2 > /sys/kernel/debug/camrtc/log-level
root@tegra-ubuntu:/home/ubuntu/zaro-workspace/Lab-App-V2/build/lab-app-v2# echo > /sys/kernel/debug/tracing/trace
root@tegra-ubuntu:/home/ubuntu/zaro-workspace/Lab-App-V2/build/lab-app-v2# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 310/310 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
kworker/5:1-2669 [005] ...1 2023.096504: rtcpu_vinotify_handle_msg: tstamp:63523022710 tag:CHANSEL_PXL_EOF channel:0x02 frame:97 vi_tstamp:3393480007 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.096510: rtcpu_vinotify_handle_msg: tstamp:63523023265 tag:CHANSEL_PXL_EOF channel:0x01 frame:97 vi_tstamp:3393480010 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.096511: rtcpu_vinotify_handle_msg: tstamp:63523023380 tag:ATOMP_FE channel:0x02 frame:97 vi_tstamp:3393480566 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.096513: rtcpu_vinotify_handle_msg: tstamp:63523023528 tag:ATOMP_FE channel:0x01 frame:97 vi_tstamp:3393480568 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.096514: rtcpu_vinotify_handle_msg: tstamp:63523047323 tag:ATOMP_FS channel:0x02 frame:98 vi_tstamp:3393504648 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.096515: rtcpu_vinotify_handle_msg: tstamp:63523047441 tag:ATOMP_FS channel:0x01 frame:98 vi_tstamp:3393504650 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.096517: rtcpu_vinotify_handle_msg: tstamp:63523055018 tag:CHANSEL_PXL_SOF channel:0x02 frame:98 vi_tstamp:3393512347 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.096518: rtcpu_vinotify_handle_msg: tstamp:63523055181 tag:CHANSEL_PXL_SOF channel:0x01 frame:98 vi_tstamp:3393512350 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.096520: rtcpu_vinotify_handle_msg: tstamp:63523285017 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:98 vi_tstamp:3393742457 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.096521: rtcpu_vinotify_handle_msg: tstamp:63523285801 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:98 vi_tstamp:3393743252 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.096523: rtcpu_vinotify_handle_msg: tstamp:63524063002 tag:CHANSEL_PXL_EOF channel:0x02 frame:98 vi_tstamp:3394520316 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.096524: rtcpu_vinotify_handle_msg: tstamp:63524063554 tag:CHANSEL_PXL_EOF channel:0x01 frame:98 vi_tstamp:3394520318 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.096525: rtcpu_vinotify_handle_msg: tstamp:63524063671 tag:ATOMP_FE channel:0x02 frame:98 vi_tstamp:3394520874 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.096527: rtcpu_vinotify_handle_msg: tstamp:63524063819 tag:ATOMP_FE channel:0x01 frame:98 vi_tstamp:3394520876 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.096528: rtcpu_vinotify_handle_msg: tstamp:63524087637 tag:ATOMP_FS channel:0x02 frame:99 vi_tstamp:3394544956 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.096530: rtcpu_vinotify_handle_msg: tstamp:63524087762 tag:ATOMP_FS channel:0x01 frame:99 vi_tstamp:3394544958 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.096531: rtcpu_vinotify_handle_msg: tstamp:63524095329 tag:CHANSEL_PXL_SOF channel:0x02 frame:99 vi_tstamp:3394552656 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.096533: rtcpu_vinotify_handle_msg: tstamp:63524095493 tag:CHANSEL_PXL_SOF channel:0x01 frame:99 vi_tstamp:3394552657 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.096534: rtcpu_vinotify_handle_msg: tstamp:63524378928 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:99 vi_tstamp:3394836347 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.096536: rtcpu_vinotify_handle_msg: tstamp:63524381179 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:99 vi_tstamp:3394838625 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.148502: rtcpu_vinotify_handle_msg: tstamp:63525103329 tag:CHANSEL_PXL_EOF channel:0x02 frame:99 vi_tstamp:3395560624 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.148505: rtcpu_vinotify_handle_msg: tstamp:63525103888 tag:CHANSEL_PXL_EOF channel:0x01 frame:99 vi_tstamp:3395560626 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.148506: rtcpu_vinotify_handle_msg: tstamp:63525104006 tag:ATOMP_FE channel:0x02 frame:99 vi_tstamp:3395561182 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.148508: rtcpu_vinotify_handle_msg: tstamp:63525104154 tag:ATOMP_FE channel:0x01 frame:99 vi_tstamp:3395561185 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.148509: rtcpu_vinotify_handle_msg: tstamp:63525127936 tag:ATOMP_FS channel:0x02 frame:100 vi_tstamp:3395585264 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.148510: rtcpu_vinotify_handle_msg: tstamp:63525128053 tag:ATOMP_FS channel:0x01 frame:100 vi_tstamp:3395585267 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.148511: rtcpu_vinotify_handle_msg: tstamp:63525135636 tag:CHANSEL_PXL_SOF channel:0x02 frame:100 vi_tstamp:3395592963 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.148513: rtcpu_vinotify_handle_msg: tstamp:63525135800 tag:CHANSEL_PXL_SOF channel:0x01 frame:100 vi_tstamp:3395592966 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.148516: rtos_queue_peek_from_isr_failed: tstamp:63525173270 queue:0x0b4a3c58
kworker/5:1-2669 [005] ...1 2023.148517: rtcpu_vinotify_handle_msg: tstamp:63525359480 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:100 vi_tstamp:3395816921 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.148519: rtcpu_vinotify_handle_msg: tstamp:63525360513 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:100 vi_tstamp:3395817960 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.148520: rtcpu_vinotify_handle_msg: tstamp:63526143617 tag:CHANSEL_PXL_EOF channel:0x02 frame:100 vi_tstamp:3396600932 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.148521: rtcpu_vinotify_handle_msg: tstamp:63526144172 tag:CHANSEL_PXL_EOF channel:0x01 frame:100 vi_tstamp:3396600935 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.148523: rtcpu_vinotify_handle_msg: tstamp:63526144289 tag:ATOMP_FE channel:0x02 frame:100 vi_tstamp:3396601490 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.148524: rtcpu_vinotify_handle_msg: tstamp:63526144438 tag:ATOMP_FE channel:0x01 frame:100 vi_tstamp:3396601493 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.148525: rtcpu_vinotify_handle_msg: tstamp:63526168246 tag:ATOMP_FS channel:0x02 frame:101 vi_tstamp:3396625572 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.148527: rtcpu_vinotify_handle_msg: tstamp:63526168362 tag:ATOMP_FS channel:0x01 frame:101 vi_tstamp:3396625575 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.148528: rtcpu_vinotify_handle_msg: tstamp:63526175947 tag:CHANSEL_PXL_SOF channel:0x02 frame:101 vi_tstamp:3396633272 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.148529: rtcpu_vinotify_handle_msg: tstamp:63526176111 tag:CHANSEL_PXL_SOF channel:0x01 frame:101 vi_tstamp:3396633274 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.200509: rtcpu_vinotify_handle_msg: tstamp:63526396574 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:101 vi_tstamp:3396854020 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.200511: rtcpu_vinotify_handle_msg: tstamp:63526409223 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:101 vi_tstamp:3396866662 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.200513: rtcpu_vinotify_handle_msg: tstamp:63527183938 tag:CHANSEL_PXL_EOF channel:0x02 frame:101 vi_tstamp:3397641240 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.200514: rtcpu_vinotify_handle_msg: tstamp:63527184485 tag:CHANSEL_PXL_EOF channel:0x01 frame:101 vi_tstamp:3397641243 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.200516: rtcpu_vinotify_handle_msg: tstamp:63527184600 tag:ATOMP_FE channel:0x02 frame:101 vi_tstamp:3397641798 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.200517: rtcpu_vinotify_handle_msg: tstamp:63527184747 tag:ATOMP_FE channel:0x01 frame:101 vi_tstamp:3397641800 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.200518: rtcpu_vinotify_handle_msg: tstamp:63527208555 tag:ATOMP_FS channel:0x02 frame:102 vi_tstamp:3397665881 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.200520: rtcpu_vinotify_handle_msg: tstamp:63527208673 tag:ATOMP_FS channel:0x01 frame:102 vi_tstamp:3397665883 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.200521: rtcpu_vinotify_handle_msg: tstamp:63527216251 tag:CHANSEL_PXL_SOF channel:0x02 frame:102 vi_tstamp:3397673580 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.200523: rtcpu_vinotify_handle_msg: tstamp:63527216414 tag:CHANSEL_PXL_SOF channel:0x01 frame:102 vi_tstamp:3397673582 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.200524: rtcpu_vinotify_handle_msg: tstamp:63527439676 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:102 vi_tstamp:3397897122 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.200525: rtcpu_vinotify_handle_msg: tstamp:63527440423 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:102 vi_tstamp:3397897874 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.252507: rtcpu_vinotify_handle_msg: tstamp:63528224230 tag:CHANSEL_PXL_EOF channel:0x02 frame:102 vi_tstamp:3398681549 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.252510: rtcpu_vinotify_handle_msg: tstamp:63528224781 tag:CHANSEL_PXL_EOF channel:0x01 frame:102 vi_tstamp:3398681551 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.252511: rtcpu_vinotify_handle_msg: tstamp:63528224899 tag:ATOMP_FE channel:0x02 frame:102 vi_tstamp:3398682106 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.252512: rtcpu_vinotify_handle_msg: tstamp:63528225047 tag:ATOMP_FE channel:0x01 frame:102 vi_tstamp:3398682108 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.252514: rtcpu_vinotify_handle_msg: tstamp:63528248863 tag:ATOMP_FS channel:0x02 frame:103 vi_tstamp:3398706189 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.252515: rtcpu_vinotify_handle_msg: tstamp:63528248981 tag:ATOMP_FS channel:0x01 frame:103 vi_tstamp:3398706191 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.252517: rtcpu_vinotify_handle_msg: tstamp:63528256564 tag:CHANSEL_PXL_SOF channel:0x02 frame:103 vi_tstamp:3398713887 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.252518: rtcpu_vinotify_handle_msg: tstamp:63528256728 tag:CHANSEL_PXL_SOF channel:0x01 frame:103 vi_tstamp:3398713889 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.252519: rtcpu_vinotify_handle_msg: tstamp:63528501186 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:103 vi_tstamp:3398958629 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.252521: rtcpu_vinotify_handle_msg: tstamp:63528501500 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:103 vi_tstamp:3398959080 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.252523: rtcpu_vinotify_handle_msg: tstamp:63529264560 tag:CHANSEL_PXL_EOF channel:0x02 frame:103 vi_tstamp:3399721857 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.252524: rtcpu_vinotify_handle_msg: tstamp:63529265116 tag:CHANSEL_PXL_EOF channel:0x01 frame:103 vi_tstamp:3399721858 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.252525: rtcpu_vinotify_handle_msg: tstamp:63529265231 tag:ATOMP_FE channel:0x02 frame:103 vi_tstamp:3399722414 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.252527: rtcpu_vinotify_handle_msg: tstamp:63529265378 tag:ATOMP_FE channel:0x01 frame:103 vi_tstamp:3399722417 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.252528: rtcpu_vinotify_handle_msg: tstamp:63529289174 tag:ATOMP_FS channel:0x02 frame:104 vi_tstamp:3399746497 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.252530: rtcpu_vinotify_handle_msg: tstamp:63529289291 tag:ATOMP_FS channel:0x01 frame:104 vi_tstamp:3399746500 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.252531: rtcpu_vinotify_handle_msg: tstamp:63529296878 tag:CHANSEL_PXL_SOF channel:0x02 frame:104 vi_tstamp:3399754197 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.252532: rtcpu_vinotify_handle_msg: tstamp:63529297041 tag:CHANSEL_PXL_SOF channel:0x01 frame:104 vi_tstamp:3399754197 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.304508: rtcpu_vinotify_handle_msg: tstamp:63529541342 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:104 vi_tstamp:3399998786 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.304510: rtcpu_vinotify_handle_msg: tstamp:63529542323 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:104 vi_tstamp:3399999772 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.304513: rtos_queue_peek_from_isr_failed: tstamp:63530173782 queue:0x0b4a3c58
kworker/5:1-2669 [005] ...1 2023.304514: rtcpu_vinotify_handle_msg: tstamp:63530304844 tag:CHANSEL_PXL_EOF channel:0x02 frame:104 vi_tstamp:3400762165 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.304515: rtcpu_vinotify_handle_msg: tstamp:63530305391 tag:CHANSEL_PXL_EOF channel:0x01 frame:104 vi_tstamp:3400762167 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.304517: rtcpu_vinotify_handle_msg: tstamp:63530305508 tag:ATOMP_FE channel:0x02 frame:104 vi_tstamp:3400762722 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.304518: rtcpu_vinotify_handle_msg: tstamp:63530305657 tag:ATOMP_FE channel:0x01 frame:104 vi_tstamp:3400762725 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.304520: rtcpu_vinotify_handle_msg: tstamp:63530329479 tag:ATOMP_FS channel:0x02 frame:105 vi_tstamp:3400786805 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.304521: rtcpu_vinotify_handle_msg: tstamp:63530329595 tag:ATOMP_FS channel:0x01 frame:105 vi_tstamp:3400786808 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.304523: rtcpu_vinotify_handle_msg: tstamp:63530337180 tag:CHANSEL_PXL_SOF channel:0x02 frame:105 vi_tstamp:3400794505 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.304524: rtcpu_vinotify_handle_msg: tstamp:63530337346 tag:CHANSEL_PXL_SOF channel:0x01 frame:105 vi_tstamp:3400794507 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.304526: rtcpu_vinotify_handle_msg: tstamp:63530579599 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:105 vi_tstamp:3401037013 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.304527: rtcpu_vinotify_handle_msg: tstamp:63530587976 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:105 vi_tstamp:3401045417 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.356505: rtcpu_vinotify_handle_msg: tstamp:63531345158 tag:CHANSEL_PXL_EOF channel:0x02 frame:105 vi_tstamp:3401802473 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.356508: rtcpu_vinotify_handle_msg: tstamp:63531345721 tag:CHANSEL_PXL_EOF channel:0x01 frame:105 vi_tstamp:3401802475 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.356509: rtcpu_vinotify_handle_msg: tstamp:63531345836 tag:ATOMP_FE channel:0x02 frame:105 vi_tstamp:3401803031 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.356510: rtcpu_vinotify_handle_msg: tstamp:63531345984 tag:ATOMP_FE channel:0x01 frame:105 vi_tstamp:3401803033 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.356512: rtcpu_vinotify_handle_msg: tstamp:63531369792 tag:ATOMP_FS channel:0x02 frame:106 vi_tstamp:3401827114 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.356513: rtcpu_vinotify_handle_msg: tstamp:63531369912 tag:ATOMP_FS channel:0x01 frame:106 vi_tstamp:3401827115 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.356515: rtcpu_vinotify_handle_msg: tstamp:63531377485 tag:CHANSEL_PXL_SOF channel:0x02 frame:106 vi_tstamp:3401834812 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.356516: rtcpu_vinotify_handle_msg: tstamp:63531377649 tag:CHANSEL_PXL_SOF channel:0x01 frame:106 vi_tstamp:3401834814 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.356517: rtcpu_vinotify_handle_msg: tstamp:63531621807 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:106 vi_tstamp:3402079249 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.356519: rtcpu_vinotify_handle_msg: tstamp:63531622460 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:106 vi_tstamp:3402079909 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.356520: rtcpu_vinotify_handle_msg: tstamp:63532385480 tag:CHANSEL_PXL_EOF channel:0x02 frame:106 vi_tstamp:3402842781 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.356522: rtcpu_vinotify_handle_msg: tstamp:63532386029 tag:CHANSEL_PXL_EOF channel:0x01 frame:106 vi_tstamp:3402842783 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.356523: rtcpu_vinotify_handle_msg: tstamp:63532386144 tag:ATOMP_FE channel:0x02 frame:106 vi_tstamp:3402843339 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.356525: rtcpu_vinotify_handle_msg: tstamp:63532386292 tag:ATOMP_FE channel:0x01 frame:106 vi_tstamp:3402843341 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.356526: rtcpu_vinotify_handle_msg: tstamp:63532410097 tag:ATOMP_FS channel:0x02 frame:107 vi_tstamp:3402867421 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.356527: rtcpu_vinotify_handle_msg: tstamp:63532410213 tag:ATOMP_FS channel:0x01 frame:107 vi_tstamp:3402867423 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.356529: rtcpu_vinotify_handle_msg: tstamp:63532417794 tag:CHANSEL_PXL_SOF channel:0x02 frame:107 vi_tstamp:3402875120 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.356530: rtcpu_vinotify_handle_msg: tstamp:63532417963 tag:CHANSEL_PXL_SOF channel:0x01 frame:107 vi_tstamp:3402875124 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.356532: rtcpu_vinotify_handle_msg: tstamp:63532662672 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:107 vi_tstamp:3403119914 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.356533: rtcpu_vinotify_handle_msg: tstamp:63532662789 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:107 vi_tstamp:3403120184 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.408540: rtcpu_vinotify_handle_msg: tstamp:63533425772 tag:CHANSEL_PXL_EOF channel:0x02 frame:107 vi_tstamp:3403883090 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.408545: rtcpu_vinotify_handle_msg: tstamp:63533426319 tag:CHANSEL_PXL_EOF channel:0x01 frame:107 vi_tstamp:3403883091 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.408548: rtcpu_vinotify_handle_msg: tstamp:63533426435 tag:ATOMP_FE channel:0x02 frame:107 vi_tstamp:3403883647 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.408551: rtcpu_vinotify_handle_msg: tstamp:63533426586 tag:ATOMP_FE channel:0x01 frame:107 vi_tstamp:3403883650 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.408553: rtcpu_vinotify_handle_msg: tstamp:63533450402 tag:ATOMP_FS channel:0x02 frame:108 vi_tstamp:3403907729 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.408556: rtcpu_vinotify_handle_msg: tstamp:63533450516 tag:ATOMP_FS channel:0x01 frame:108 vi_tstamp:3403907732 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.408558: rtcpu_vinotify_handle_msg: tstamp:63533458106 tag:CHANSEL_PXL_SOF channel:0x02 frame:108 vi_tstamp:3403915429 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.408561: rtcpu_vinotify_handle_msg: tstamp:63533458273 tag:CHANSEL_PXL_SOF channel:0x01 frame:108 vi_tstamp:3403915431 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.408563: rtcpu_vinotify_handle_msg: tstamp:63533681332 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:108 vi_tstamp:3404138774 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.408566: rtcpu_vinotify_handle_msg: tstamp:63533682093 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:108 vi_tstamp:3404139542 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.460498: rtcpu_vinotify_handle_msg: tstamp:63534466098 tag:CHANSEL_PXL_EOF channel:0x02 frame:108 vi_tstamp:3404923397 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.460503: rtcpu_vinotify_handle_msg: tstamp:63534466657 tag:CHANSEL_PXL_EOF channel:0x01 frame:108 vi_tstamp:3404923400 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.460504: rtcpu_vinotify_handle_msg: tstamp:63534466771 tag:ATOMP_FE channel:0x02 frame:108 vi_tstamp:3404923955 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.460506: rtcpu_vinotify_handle_msg: tstamp:63534466919 tag:ATOMP_FE channel:0x01 frame:108 vi_tstamp:3404923958 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.460507: rtcpu_vinotify_handle_msg: tstamp:63534490711 tag:ATOMP_FS channel:0x02 frame:109 vi_tstamp:3404948037 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.460509: rtcpu_vinotify_handle_msg: tstamp:63534490831 tag:ATOMP_FS channel:0x01 frame:109 vi_tstamp:3404948040 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.460510: rtcpu_vinotify_handle_msg: tstamp:63534498411 tag:CHANSEL_PXL_SOF channel:0x02 frame:109 vi_tstamp:3404955738 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.460511: rtcpu_vinotify_handle_msg: tstamp:63534498576 tag:CHANSEL_PXL_SOF channel:0x01 frame:109 vi_tstamp:3404955738 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.460513: rtcpu_vinotify_handle_msg: tstamp:63534957325 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:109 vi_tstamp:3405414760 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.460514: rtcpu_vinotify_handle_msg: tstamp:63534964355 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:109 vi_tstamp:3405421799 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.460516: rtos_queue_peek_from_isr_failed: tstamp:63535174309 queue:0x0b4a3c58
kworker/5:1-2669 [005] ...1 2023.460517: rtcpu_vinotify_handle_msg: tstamp:63535506391 tag:CHANSEL_PXL_EOF channel:0x02 frame:109 vi_tstamp:3405963705 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.460519: rtcpu_vinotify_handle_msg: tstamp:63535506949 tag:CHANSEL_PXL_EOF channel:0x01 frame:109 vi_tstamp:3405963708 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.460520: rtcpu_vinotify_handle_msg: tstamp:63535507068 tag:ATOMP_FE channel:0x02 frame:109 vi_tstamp:3405964264 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.460522: rtcpu_vinotify_handle_msg: tstamp:63535507216 tag:ATOMP_FE channel:0x01 frame:109 vi_tstamp:3405964266 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.460523: rtcpu_vinotify_handle_msg: tstamp:63535531023 tag:ATOMP_FS channel:0x02 frame:110 vi_tstamp:3405988346 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.460524: rtcpu_vinotify_handle_msg: tstamp:63535531142 tag:ATOMP_FS channel:0x01 frame:110 vi_tstamp:3405988348 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.460526: rtcpu_vinotify_handle_msg: tstamp:63535538717 tag:CHANSEL_PXL_SOF channel:0x02 frame:110 vi_tstamp:3405996046 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.460527: rtcpu_vinotify_handle_msg: tstamp:63535538878 tag:CHANSEL_PXL_SOF channel:0x01 frame:110 vi_tstamp:3405996047 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.460528: rtcpu_vinotify_handle_msg: tstamp:63535819169 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:110 vi_tstamp:3406276608 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.460530: rtcpu_vinotify_handle_msg: tstamp:63535820511 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:110 vi_tstamp:3406277960 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.512512: rtcpu_vinotify_handle_msg: tstamp:63536546701 tag:CHANSEL_PXL_EOF channel:0x02 frame:110 vi_tstamp:3407004014 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.512516: rtcpu_vinotify_handle_msg: tstamp:63536547251 tag:CHANSEL_PXL_EOF channel:0x01 frame:110 vi_tstamp:3407004016 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.512517: rtcpu_vinotify_handle_msg: tstamp:63536547367 tag:ATOMP_FE channel:0x02 frame:110 vi_tstamp:3407004572 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.512518: rtcpu_vinotify_handle_msg: tstamp:63536547514 tag:ATOMP_FE channel:0x01 frame:110 vi_tstamp:3407004574 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.512520: rtcpu_vinotify_handle_msg: tstamp:63536571328 tag:ATOMP_FS channel:0x02 frame:111 vi_tstamp:3407028654 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.512521: rtcpu_vinotify_handle_msg: tstamp:63536571447 tag:ATOMP_FS channel:0x01 frame:111 vi_tstamp:3407028656 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.512523: rtcpu_vinotify_handle_msg: tstamp:63536579029 tag:CHANSEL_PXL_SOF channel:0x02 frame:111 vi_tstamp:3407036353 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.512524: rtcpu_vinotify_handle_msg: tstamp:63536579195 tag:CHANSEL_PXL_SOF channel:0x01 frame:111 vi_tstamp:3407036355 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.512525: rtcpu_vinotify_handle_msg: tstamp:63536823331 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:111 vi_tstamp:3407280772 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.512527: rtcpu_vinotify_handle_msg: tstamp:63536823861 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:111 vi_tstamp:3407281292 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.512528: rtcpu_vinotify_handle_msg: tstamp:63537587023 tag:CHANSEL_PXL_EOF channel:0x02 frame:111 vi_tstamp:3408044322 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.512529: rtcpu_vinotify_handle_msg: tstamp:63537587577 tag:CHANSEL_PXL_EOF channel:0x01 frame:111 vi_tstamp:3408044324 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.512531: rtcpu_vinotify_handle_msg: tstamp:63537587692 tag:ATOMP_FE channel:0x02 frame:111 vi_tstamp:3408044880 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.512532: rtcpu_vinotify_handle_msg: tstamp:63537587840 tag:ATOMP_FE channel:0x01 frame:111 vi_tstamp:3408044882 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.564515: rtcpu_vinotify_handle_msg: tstamp:63537611634 tag:ATOMP_FS channel:0x02 frame:112 vi_tstamp:3408068962 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.564518: rtcpu_vinotify_handle_msg: tstamp:63537611750 tag:ATOMP_FS channel:0x01 frame:112 vi_tstamp:3408068965 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.564520: rtcpu_vinotify_handle_msg: tstamp:63537619338 tag:CHANSEL_PXL_SOF channel:0x02 frame:112 vi_tstamp:3408076661 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.564521: rtcpu_vinotify_handle_msg: tstamp:63537619501 tag:CHANSEL_PXL_SOF channel:0x01 frame:112 vi_tstamp:3408076664 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.564523: rtcpu_vinotify_handle_msg: tstamp:63538066972 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:112 vi_tstamp:3408524405 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.564524: rtcpu_vinotify_handle_msg: tstamp:63538073015 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:112 vi_tstamp:3408530454 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.564525: rtcpu_vinotify_handle_msg: tstamp:63538627332 tag:CHANSEL_PXL_EOF channel:0x02 frame:112 vi_tstamp:3409084630 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.564526: rtcpu_vinotify_handle_msg: tstamp:63538627888 tag:CHANSEL_PXL_EOF channel:0x01 frame:112 vi_tstamp:3409084632 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.564528: rtcpu_vinotify_handle_msg: tstamp:63538628004 tag:ATOMP_FE channel:0x02 frame:112 vi_tstamp:3409085187 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.564529: rtcpu_vinotify_handle_msg: tstamp:63538628152 tag:ATOMP_FE channel:0x01 frame:112 vi_tstamp:3409085190 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.564530: rtcpu_vinotify_handle_msg: tstamp:63538652066 tag:ATOMP_FS channel:0x02 frame:113 vi_tstamp:3409109270 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.564531: rtcpu_vinotify_handle_msg: tstamp:63538652183 tag:ATOMP_FS channel:0x00 frame:113 vi_tstamp:3409109273 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.564533: rtcpu_vinotify_handle_msg: tstamp:63538652298 tag:ATOMP_FS channel:0x01 frame:113 vi_tstamp:3409109273 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.564534: rtcpu_vinotify_handle_msg: tstamp:63538659770 tag:CHANSEL_PXL_SOF channel:0x02 frame:113 vi_tstamp:3409116968 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.564535: rtcpu_vinotify_handle_msg: tstamp:63538659936 tag:CHANSEL_PXL_SOF channel:0x01 frame:113 vi_tstamp:3409116972 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.564537: rtcpu_vinotify_handle_msg: tstamp:63538660098 tag:CHANSEL_PXL_SOF channel:0x00 frame:113 vi_tstamp:3409116973 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.564538: rtcpu_vinotify_handle_msg: tstamp:63538895703 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:113 vi_tstamp:3409353139 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.564539: rtcpu_vinotify_handle_msg: tstamp:63538901901 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:113 vi_tstamp:3409359345 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.616518: rtcpu_vinotify_handle_msg: tstamp:63539668336 tag:CHANSEL_PXL_EOF channel:0x02 frame:113 vi_tstamp:3410124938 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.616522: rtcpu_vinotify_handle_msg: tstamp:63539668465 tag:CHANSEL_PXL_EOF channel:0x01 frame:113 vi_tstamp:3410124941 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.616523: rtcpu_vinotify_handle_msg: tstamp:63539668580 tag:CHANSEL_PXL_EOF channel:0x00 frame:113 vi_tstamp:3410124941 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.616525: rtcpu_vinotify_handle_msg: tstamp:63539668695 tag:ATOMP_FE channel:0x02 frame:113 vi_tstamp:3410125496 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616526: rtcpu_vinotify_handle_msg: tstamp:63539668843 tag:ATOMP_FE channel:0x00 frame:113 vi_tstamp:3410125498 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616528: rtcpu_vinotify_handle_msg: tstamp:63539668982 tag:ATOMP_FE channel:0x01 frame:113 vi_tstamp:3410125498 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616530: rtcpu_vinotify_handle_msg: tstamp:63539692251 tag:ATOMP_FS channel:0x02 frame:114 vi_tstamp:3410149579 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616531: rtcpu_vinotify_handle_msg: tstamp:63539692365 tag:ATOMP_FS channel:0x01 frame:114 vi_tstamp:3410149581 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616533: rtcpu_vinotify_handle_msg: tstamp:63539699954 tag:CHANSEL_PXL_SOF channel:0x02 frame:114 vi_tstamp:3410157278 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.616535: rtcpu_vinotify_handle_msg: tstamp:63539700118 tag:CHANSEL_PXL_SOF channel:0x01 frame:114 vi_tstamp:3410157279 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.616536: rtcpu_vinotify_handle_msg: tstamp:63539980576 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:114 vi_tstamp:3410438020 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.616538: rtcpu_vinotify_handle_msg: tstamp:63539981722 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:114 vi_tstamp:3410439170 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.616541: rtos_queue_peek_from_isr_failed: tstamp:63540174787 queue:0x0b4a3c58
kworker/5:1-2669 [005] ...1 2023.616543: rtcpu_vinotify_handle_msg: tstamp:63540707932 tag:CHANSEL_PXL_EOF channel:0x02 frame:114 vi_tstamp:3411165246 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.616545: rtcpu_vinotify_handle_msg: tstamp:63540708484 tag:CHANSEL_PXL_EOF channel:0x01 frame:114 vi_tstamp:3411165249 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.616546: rtcpu_vinotify_handle_msg: tstamp:63540708600 tag:ATOMP_FE channel:0x02 frame:114 vi_tstamp:3411165804 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616548: rtcpu_vinotify_handle_msg: tstamp:63540708750 tag:ATOMP_FE channel:0x01 frame:114 vi_tstamp:3411165806 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616550: rtcpu_vinotify_handle_msg: tstamp:63540732564 tag:ATOMP_FS channel:0x02 frame:115 vi_tstamp:3411189887 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616551: rtcpu_vinotify_handle_msg: tstamp:63540732678 tag:ATOMP_FS channel:0x01 frame:115 vi_tstamp:3411189889 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.616553: rtcpu_vinotify_handle_msg: tstamp:63540740259 tag:CHANSEL_PXL_SOF channel:0x02 frame:115 vi_tstamp:3411197587 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.616554: rtcpu_vinotify_handle_msg: tstamp:63540740427 tag:CHANSEL_PXL_SOF channel:0x01 frame:115 vi_tstamp:3411197588 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.668515: rtcpu_vinotify_handle_msg: tstamp:63541043616 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:115 vi_tstamp:3411501057 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.668518: rtcpu_vinotify_handle_msg: tstamp:63541044145 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:115 vi_tstamp:3411501608 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.668520: rtcpu_vinotify_handle_msg: tstamp:63541748254 tag:CHANSEL_PXL_EOF channel:0x02 frame:115 vi_tstamp:3412205555 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.668522: rtcpu_vinotify_handle_msg: tstamp:63541748801 tag:CHANSEL_PXL_EOF channel:0x01 frame:115 vi_tstamp:3412205557 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.668523: rtcpu_vinotify_handle_msg: tstamp:63541748918 tag:ATOMP_FE channel:0x02 frame:115 vi_tstamp:3412206112 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.668525: rtcpu_vinotify_handle_msg: tstamp:63541749066 tag:ATOMP_FE channel:0x01 frame:115 vi_tstamp:3412206115 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.668526: rtcpu_vinotify_handle_msg: tstamp:63541772866 tag:ATOMP_FS channel:0x02 frame:116 vi_tstamp:3412230195 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.668528: rtcpu_vinotify_handle_msg: tstamp:63541772980 tag:ATOMP_FS channel:0x01 frame:116 vi_tstamp:3412230197 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.668530: rtcpu_vinotify_handle_msg: tstamp:63541780571 tag:CHANSEL_PXL_SOF channel:0x02 frame:116 vi_tstamp:3412237894 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.668531: rtcpu_vinotify_handle_msg: tstamp:63541780734 tag:CHANSEL_PXL_SOF channel:0x01 frame:116 vi_tstamp:3412237895 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.668533: rtcpu_vinotify_handle_msg: tstamp:63542120212 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:116 vi_tstamp:3412577648 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.668534: rtcpu_vinotify_handle_msg: tstamp:63542122217 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:116 vi_tstamp:3412579662 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.720526: rtcpu_vinotify_handle_msg: tstamp:63542788564 tag:CHANSEL_PXL_EOF channel:0x02 frame:116 vi_tstamp:3413245863 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.720530: rtcpu_vinotify_handle_msg: tstamp:63542789119 tag:CHANSEL_PXL_EOF channel:0x01 frame:116 vi_tstamp:3413245865 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.720532: rtcpu_vinotify_handle_msg: tstamp:63542789238 tag:ATOMP_FE channel:0x02 frame:116 vi_tstamp:3413246420 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.720534: rtcpu_vinotify_handle_msg: tstamp:63542789386 tag:ATOMP_FE channel:0x01 frame:116 vi_tstamp:3413246423 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.720536: rtcpu_vinotify_handle_msg: tstamp:63542813175 tag:ATOMP_FS channel:0x02 frame:117 vi_tstamp:3413270503 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.720538: rtcpu_vinotify_handle_msg: tstamp:63542813292 tag:ATOMP_FS channel:0x01 frame:117 vi_tstamp:3413270506 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.720540: rtcpu_vinotify_handle_msg: tstamp:63542820879 tag:CHANSEL_PXL_SOF channel:0x02 frame:117 vi_tstamp:3413278203 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.720542: rtcpu_vinotify_handle_msg: tstamp:63542821046 tag:CHANSEL_PXL_SOF channel:0x01 frame:117 vi_tstamp:3413278204 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.720544: rtcpu_vinotify_handle_msg: tstamp:63543099338 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:117 vi_tstamp:3413556780 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.720546: rtcpu_vinotify_handle_msg: tstamp:63543106415 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:117 vi_tstamp:3413563859 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.720548: rtcpu_vinotify_handle_msg: tstamp:63543828856 tag:CHANSEL_PXL_EOF channel:0x02 frame:117 vi_tstamp:3414286171 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.720549: rtcpu_vinotify_handle_msg: tstamp:63543829410 tag:CHANSEL_PXL_EOF channel:0x01 frame:117 vi_tstamp:3414286173 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.720551: rtcpu_vinotify_handle_msg: tstamp:63543829527 tag:ATOMP_FE channel:0x02 frame:117 vi_tstamp:3414286729 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.720553: rtcpu_vinotify_handle_msg: tstamp:63543829678 tag:ATOMP_FE channel:0x01 frame:117 vi_tstamp:3414286731 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.720555: rtcpu_vinotify_handle_msg: tstamp:63543853488 tag:ATOMP_FS channel:0x02 frame:118 vi_tstamp:3414310811 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.720557: rtcpu_vinotify_handle_msg: tstamp:63543853607 tag:ATOMP_FS channel:0x01 frame:118 vi_tstamp:3414310813 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.720559: rtcpu_vinotify_handle_msg: tstamp:63543861186 tag:CHANSEL_PXL_SOF channel:0x02 frame:118 vi_tstamp:3414318510 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.720561: rtcpu_vinotify_handle_msg: tstamp:63543861348 tag:CHANSEL_PXL_SOF channel:0x01 frame:118 vi_tstamp:3414318512 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.772527: rtcpu_vinotify_handle_msg: tstamp:63544202293 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:118 vi_tstamp:3414659737 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.772531: rtcpu_vinotify_handle_msg: tstamp:63544203942 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:118 vi_tstamp:3414661391 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.772533: rtcpu_vinotify_handle_msg: tstamp:63544869181 tag:CHANSEL_PXL_EOF channel:0x02 frame:118 vi_tstamp:3415326479 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.772535: rtcpu_vinotify_handle_msg: tstamp:63544869745 tag:CHANSEL_PXL_EOF channel:0x01 frame:118 vi_tstamp:3415326481 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.772536: rtcpu_vinotify_handle_msg: tstamp:63544869868 tag:ATOMP_FE channel:0x02 frame:118 vi_tstamp:3415327037 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.772538: rtcpu_vinotify_handle_msg: tstamp:63544870016 tag:ATOMP_FE channel:0x01 frame:118 vi_tstamp:3415327039 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.772540: rtcpu_vinotify_handle_msg: tstamp:63544893793 tag:ATOMP_FS channel:0x02 frame:119 vi_tstamp:3415351119 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.772542: rtcpu_vinotify_handle_msg: tstamp:63544893908 tag:ATOMP_FS channel:0x01 frame:119 vi_tstamp:3415351121 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.772544: rtcpu_vinotify_handle_msg: tstamp:63544901494 tag:CHANSEL_PXL_SOF channel:0x02 frame:119 vi_tstamp:3415358818 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.772546: rtcpu_vinotify_handle_msg: tstamp:63544901657 tag:CHANSEL_PXL_SOF channel:0x01 frame:119 vi_tstamp:3415358821 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.772548: rtos_queue_peek_from_isr_failed: tstamp:63545175295 queue:0x0b4a3c58
kworker/5:1-2669 [005] ...1 2023.772552: rtcpu_vinotify_handle_msg: tstamp:63545242663 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:119 vi_tstamp:3415699904 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.772554: rtcpu_vinotify_handle_msg: tstamp:63545242779 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:119 vi_tstamp:3415700049 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.824508: rtcpu_vinotify_handle_msg: tstamp:63545909929 tag:CHANSEL_PXL_EOF channel:0x02 frame:119 vi_tstamp:3416366788 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.824510: rtcpu_vinotify_handle_msg: tstamp:63545910063 tag:CHANSEL_PXL_EOF channel:0x01 frame:119 vi_tstamp:3416366789 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.824512: rtcpu_vinotify_handle_msg: tstamp:63545910179 tag:ATOMP_FE channel:0x02 frame:119 vi_tstamp:3416367345 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.824513: rtcpu_vinotify_handle_msg: tstamp:63545910327 tag:ATOMP_FE channel:0x01 frame:119 vi_tstamp:3416367347 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.824515: rtcpu_vinotify_handle_msg: tstamp:63545934102 tag:ATOMP_FS channel:0x02 frame:120 vi_tstamp:3416391427 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.824517: rtcpu_vinotify_handle_msg: tstamp:63545934217 tag:ATOMP_FS channel:0x01 frame:120 vi_tstamp:3416391430 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.824518: rtcpu_vinotify_handle_msg: tstamp:63545941804 tag:CHANSEL_PXL_SOF channel:0x02 frame:120 vi_tstamp:3416399126 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.824520: rtcpu_vinotify_handle_msg: tstamp:63545941968 tag:CHANSEL_PXL_SOF channel:0x01 frame:120 vi_tstamp:3416399129 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.824521: rtcpu_vinotify_handle_msg: tstamp:63546283375 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:120 vi_tstamp:3416740810 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.824523: rtcpu_vinotify_handle_msg: tstamp:63546285089 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:120 vi_tstamp:3416742537 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.824524: rtcpu_vinotify_handle_msg: tstamp:63546950235 tag:CHANSEL_PXL_EOF channel:0x02 frame:120 vi_tstamp:3417407095 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.824526: rtcpu_vinotify_handle_msg: tstamp:63546950367 tag:CHANSEL_PXL_EOF channel:0x01 frame:120 vi_tstamp:3417407097 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.824528: rtcpu_vinotify_handle_msg: tstamp:63546950482 tag:ATOMP_FE channel:0x02 frame:120 vi_tstamp:3417407653 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.824529: rtcpu_vinotify_handle_msg: tstamp:63546950631 tag:ATOMP_FE channel:0x01 frame:120 vi_tstamp:3417407656 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.824531: rtcpu_vinotify_handle_msg: tstamp:63546974408 tag:ATOMP_FS channel:0x02 frame:121 vi_tstamp:3417431735 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.824532: rtcpu_vinotify_handle_msg: tstamp:63546974524 tag:ATOMP_FS channel:0x01 frame:121 vi_tstamp:3417431738 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.824534: rtcpu_vinotify_handle_msg: tstamp:63546982112 tag:CHANSEL_PXL_SOF channel:0x02 frame:121 vi_tstamp:3417439435 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.824535: rtcpu_vinotify_handle_msg: tstamp:63546982278 tag:CHANSEL_PXL_SOF channel:0x01 frame:121 vi_tstamp:3417439437 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.824537: rtcpu_vinotify_handle_msg: tstamp:63547256446 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:121 vi_tstamp:3417713887 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.824539: rtcpu_vinotify_handle_msg: tstamp:63547266233 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:121 vi_tstamp:3417723679 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.876522: rtcpu_vinotify_handle_msg: tstamp:63547990089 tag:CHANSEL_PXL_EOF channel:0x02 frame:121 vi_tstamp:3418447403 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.876524: rtcpu_vinotify_handle_msg: tstamp:63547990647 tag:CHANSEL_PXL_EOF channel:0x01 frame:121 vi_tstamp:3418447406 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.876526: rtcpu_vinotify_handle_msg: tstamp:63547990766 tag:ATOMP_FE channel:0x02 frame:121 vi_tstamp:3418447961 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.876527: rtcpu_vinotify_handle_msg: tstamp:63547990915 tag:ATOMP_FE channel:0x01 frame:121 vi_tstamp:3418447964 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.876529: rtcpu_vinotify_handle_msg: tstamp:63548014722 tag:ATOMP_FS channel:0x02 frame:122 vi_tstamp:3418472044 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.876531: rtcpu_vinotify_handle_msg: tstamp:63548014836 tag:ATOMP_FS channel:0x01 frame:122 vi_tstamp:3418472046 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.876532: rtcpu_vinotify_handle_msg: tstamp:63548022422 tag:CHANSEL_PXL_SOF channel:0x02 frame:122 vi_tstamp:3418479742 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.876534: rtcpu_vinotify_handle_msg: tstamp:63548022584 tag:CHANSEL_PXL_SOF channel:0x01 frame:122 vi_tstamp:3418479745 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.876536: rtcpu_vinotify_handle_msg: tstamp:63548450833 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:122 vi_tstamp:3418908266 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.876537: rtcpu_vinotify_handle_msg: tstamp:63548452469 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:122 vi_tstamp:3418909917 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.928513: rtcpu_vinotify_handle_msg: tstamp:63549030859 tag:CHANSEL_PXL_EOF channel:0x02 frame:122 vi_tstamp:3419487711 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.928516: rtcpu_vinotify_handle_msg: tstamp:63549030987 tag:CHANSEL_PXL_EOF channel:0x01 frame:122 vi_tstamp:3419487714 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.928518: rtcpu_vinotify_handle_msg: tstamp:63549031105 tag:ATOMP_FE channel:0x02 frame:122 vi_tstamp:3419488270 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.928520: rtcpu_vinotify_handle_msg: tstamp:63549031251 tag:ATOMP_FE channel:0x01 frame:122 vi_tstamp:3419488272 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.928522: rtcpu_vinotify_handle_msg: tstamp:63549055025 tag:ATOMP_FS channel:0x02 frame:123 vi_tstamp:3419512352 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.928524: rtcpu_vinotify_handle_msg: tstamp:63549055140 tag:ATOMP_FS channel:0x01 frame:123 vi_tstamp:3419512354 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.928526: rtcpu_vinotify_handle_msg: tstamp:63549062724 tag:CHANSEL_PXL_SOF channel:0x02 frame:123 vi_tstamp:3419520052 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.928528: rtcpu_vinotify_handle_msg: tstamp:63549062892 tag:CHANSEL_PXL_SOF channel:0x01 frame:123 vi_tstamp:3419520053 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.928530: rtcpu_vinotify_handle_msg: tstamp:63549480684 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:123 vi_tstamp:3419938121 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.928532: rtcpu_vinotify_handle_msg: tstamp:63549484907 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:123 vi_tstamp:3419942358 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.928534: rtcpu_vinotify_handle_msg: tstamp:63550070723 tag:CHANSEL_PXL_EOF channel:0x02 frame:123 vi_tstamp:3420528020 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.928536: rtcpu_vinotify_handle_msg: tstamp:63550071283 tag:CHANSEL_PXL_EOF channel:0x01 frame:123 vi_tstamp:3420528022 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.928537: rtcpu_vinotify_handle_msg: tstamp:63550071398 tag:ATOMP_FE channel:0x02 frame:123 vi_tstamp:3420528578 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.928539: rtcpu_vinotify_handle_msg: tstamp:63550071554 tag:ATOMP_FE channel:0x01 frame:123 vi_tstamp:3420528580 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.928541: rtcpu_vinotify_handle_msg: tstamp:63550095332 tag:ATOMP_FS channel:0x02 frame:124 vi_tstamp:3420552660 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.928543: rtcpu_vinotify_handle_msg: tstamp:63550095448 tag:ATOMP_FS channel:0x01 frame:124 vi_tstamp:3420552662 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.928545: rtcpu_vinotify_handle_msg: tstamp:63550103033 tag:CHANSEL_PXL_SOF channel:0x02 frame:124 vi_tstamp:3420560359 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.928547: rtcpu_vinotify_handle_msg: tstamp:63550103197 tag:CHANSEL_PXL_SOF channel:0x01 frame:124 vi_tstamp:3420560362 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.928550: rtos_queue_peek_from_isr_failed: tstamp:63550175804 queue:0x0b4a3c58
kworker/5:1-2669 [005] ...1 2023.928552: rtcpu_vinotify_handle_msg: tstamp:63550430005 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:124 vi_tstamp:3420887438 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.928554: rtcpu_vinotify_handle_msg: tstamp:63550431988 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:124 vi_tstamp:3420889435 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.980516: rtcpu_vinotify_handle_msg: tstamp:63551111032 tag:CHANSEL_PXL_EOF channel:0x02 frame:124 vi_tstamp:3421568328 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.980520: rtcpu_vinotify_handle_msg: tstamp:63551111592 tag:CHANSEL_PXL_EOF channel:0x01 frame:124 vi_tstamp:3421568330 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.980522: rtcpu_vinotify_handle_msg: tstamp:63551111707 tag:ATOMP_FE channel:0x02 frame:124 vi_tstamp:3421568885 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.980524: rtcpu_vinotify_handle_msg: tstamp:63551111857 tag:ATOMP_FE channel:0x01 frame:124 vi_tstamp:3421568888 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.980526: rtcpu_vinotify_handle_msg: tstamp:63551135641 tag:ATOMP_FS channel:0x02 frame:125 vi_tstamp:3421592968 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.980528: rtcpu_vinotify_handle_msg: tstamp:63551135755 tag:ATOMP_FS channel:0x01 frame:125 vi_tstamp:3421592971 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.980530: rtcpu_vinotify_handle_msg: tstamp:63551143358 tag:CHANSEL_PXL_SOF channel:0x02 frame:125 vi_tstamp:3421600667 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.980532: rtcpu_vinotify_handle_msg: tstamp:63551143528 tag:CHANSEL_PXL_SOF channel:0x01 frame:125 vi_tstamp:3421600670 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.980534: rtcpu_vinotify_handle_msg: tstamp:63551421743 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:125 vi_tstamp:3421879184 data:0x18000000
kworker/5:1-2669 [005] ...1 2023.980536: rtcpu_vinotify_handle_msg: tstamp:63551428868 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:125 vi_tstamp:3421886314 data:0x28000000
kworker/5:1-2669 [005] ...1 2023.980538: rtcpu_vinotify_handle_msg: tstamp:63552151320 tag:CHANSEL_PXL_EOF channel:0x02 frame:125 vi_tstamp:3422608636 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.980540: rtcpu_vinotify_handle_msg: tstamp:63552151870 tag:CHANSEL_PXL_EOF channel:0x01 frame:125 vi_tstamp:3422608639 data:0x0da70002
kworker/5:1-2669 [005] ...1 2023.980542: rtcpu_vinotify_handle_msg: tstamp:63552151985 tag:ATOMP_FE channel:0x02 frame:125 vi_tstamp:3422609194 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.980543: rtcpu_vinotify_handle_msg: tstamp:63552152134 tag:ATOMP_FE channel:0x01 frame:125 vi_tstamp:3422609196 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.980545: rtcpu_vinotify_handle_msg: tstamp:63552175968 tag:ATOMP_FS channel:0x02 frame:126 vi_tstamp:3422633276 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.980547: rtcpu_vinotify_handle_msg: tstamp:63552176085 tag:ATOMP_FS channel:0x01 frame:126 vi_tstamp:3422633279 data:0x00000000
kworker/5:1-2669 [005] ...1 2023.980549: rtcpu_vinotify_handle_msg: tstamp:63552183647 tag:CHANSEL_PXL_SOF channel:0x02 frame:126 vi_tstamp:3422640974 data:0x00000001
kworker/5:1-2669 [005] ...1 2023.980551: rtcpu_vinotify_handle_msg: tstamp:63552183811 tag:CHANSEL_PXL_SOF channel:0x01 frame:126 vi_tstamp:3422640977 data:0x00000001
kworker/5:1-2669 [005] ...1 2024.032571: rtcpu_vinotify_handle_msg: tstamp:63552518796 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:126 vi_tstamp:3422976239 data:0x28000000
kworker/5:1-2669 [005] ...1 2024.032574: rtcpu_vinotify_handle_msg: tstamp:63552519326 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:126 vi_tstamp:3422976799 data:0x18000000
kworker/5:1-2669 [005] ...1 2024.032576: rtcpu_vinotify_handle_msg: tstamp:63553191653 tag:CHANSEL_PXL_EOF channel:0x02 frame:126 vi_tstamp:3423648944 data:0x0da70002
kworker/5:1-2669 [005] ...1 2024.032578: rtcpu_vinotify_handle_msg: tstamp:63553192208 tag:CHANSEL_PXL_EOF channel:0x01 frame:126 vi_tstamp:3423648947 data:0x0da70002
kworker/5:1-2669 [005] ...1 2024.032580: rtcpu_vinotify_handle_msg: tstamp:63553192324 tag:ATOMP_FE channel:0x02 frame:126 vi_tstamp:3423649502 data:0x00000000
kworker/5:1-2669 [005] ...1 2024.032582: rtcpu_vinotify_handle_msg: tstamp:63553192472 tag:ATOMP_FE channel:0x01 frame:126 vi_tstamp:3423649504 data:0x00000000
kworker/5:1-2669 [005] ...1 2024.032583: rtcpu_vinotify_handle_msg: tstamp:63553216257 tag:ATOMP_FS channel:0x02 frame:127 vi_tstamp:3423673585 data:0x00000000
kworker/5:1-2669 [005] ...1 2024.032585: rtcpu_vinotify_handle_msg: tstamp:63553216371 tag:ATOMP_FS channel:0x01 frame:127 vi_tstamp:3423673587 data:0x00000000
kworker/5:1-2669 [005] ...1 2024.032587: rtcpu_vinotify_handle_msg: tstamp:63553223959 tag:CHANSEL_PXL_SOF channel:0x02 frame:127 vi_tstamp:3423681284 data:0x00000001
kworker/5:1-2669 [005] ...1 2024.032589: rtcpu_vinotify_handle_msg: tstamp:63553224124 tag:CHANSEL_PXL_SOF channel:0x01 frame:127 vi_tstamp:3423681286 data:0x00000001
kworker/5:1-2669 [005] ...1 2024.032591: rtcpu_vinotify_handle_msg: tstamp:63553529197 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:127 vi_tstamp:3423986639 data:0x18000000
kworker/5:1-2669 [005] ...1 2024.032593: rtcpu_vinotify_handle_msg: tstamp:63553530919 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:127 vi_tstamp:3423988366 data:0x28000000
Please try to set below clock to try. And run the jetson_clocks.sh before to launch multiple sensors.
echo 1 /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
to set VI clock
# cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate
1011200000
echo 1011200000 > /sys/kernel/debug/bpmp/debug/clk/vi/rate
to set ISP clock
# cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate
1126400000
echo 1126400000 > /sys/kernel/debug/bpmp/debug/clk/isp/rate
to set EMC clock
# cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate
1600000000
echo 1600000000 > /sys/kernel/debug/bpmp/debug/clk/emc/rate
ezaro
September 27, 2018, 5:43pm
7
Hi Shane,
Thank you. I Just tried this and ran my application as before and I see the following output in dmesg now
[ 502.949568] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 502.961147] tegra-vi4 15700000.vi: master error
[ 502.965685] tegra-vi4 15700000.vi: master error
[ 502.994436] tegra-vi4 15700000.vi: master error
[ 502.998974] tegra-vi4 15700000.vi: master error
[ 503.027726] tegra-vi4 15700000.vi: master error
[ 503.032263] tegra-vi4 15700000.vi: master error
[ 503.061017] tegra-vi4 15700000.vi: master error
[ 503.065563] tegra-vi4 15700000.vi: master error
[ 503.094308] tegra-vi4 15700000.vi: master error
[ 503.098850] tegra-vi4 15700000.vi: master error
[ 503.127596] tegra-vi4 15700000.vi: master error
[ 503.132133] tegra-vi4 15700000.vi: master error
[ 503.160887] tegra-vi4 15700000.vi: master error
[ 503.165424] tegra-vi4 15700000.vi: master error
[ 503.194177] tegra-vi4 15700000.vi: master error
[ 503.198714] tegra-vi4 15700000.vi: master error
[ 503.227469] tegra-vi4 15700000.vi: master error
[ 503.232008] tegra-vi4 15700000.vi: master error
[ 503.260757] tegra-vi4 15700000.vi: master error
[ 503.265295] tegra-vi4 15700000.vi: master error
[ 503.294047] tegra-vi4 15700000.vi: master error
[ 503.298585] tegra-vi4 15700000.vi: master error
[ 503.327337] tegra-vi4 15700000.vi: master error
[ 503.331876] tegra-vi4 15700000.vi: master error
[ 503.360626] tegra-vi4 15700000.vi: master error
[ 503.365167] tegra-vi4 15700000.vi: master error
[ 503.389569] tegra-vi4 15700000.vi: ATOMP_FE syncpt timeout!
[ 503.393917] tegra-vi4 15700000.vi: master error
[ 503.394701] tegra-vi4 15700000.vi: master error
I have seen this issue before and have been using the following commands to bump up the vi clock rate to MAX.
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/state
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate > /sys/kernel/debug/bpmp/debug/clk/vi/rate
This prevents the master error issue, but I still see the same behavior when I combine your suggestion with the VI clock max I have already been trying. Two of the camera’s frames arrive fine, but I only receive approximately every 20th frame from the last device, and intermittent errors in DMESG relating to PXL_SOF timeout
[ 512.277584] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 513.281581] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 514.289586] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 515.297590] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 516.305588] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 517.317592] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 518.325591] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 519.333595] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 520.341599] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 521.349597] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 522.357599] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
I didn’t see any error from the trace log. It’s possible the one of the sensors didn’t output correct data to the MIPI bus. You may need to probe the sigal to make sure of it.
ezaro
September 28, 2018, 11:52pm
9
I’ve validated on a scope that all three cameras are outputting frames at the same synced rate. It’s interesting to me that there are no errors, since I DO receive valid frames on that third device. The rate of frame reception on that third device is significantly slower and embedded metadata in those frames shows that the frame counter increments about 20 frames between the ones I receive. That seems to me to point to some sort of at hardware reception failure, as if the input device does not even realize it is missing the majority of the frames that are being output by the device.
kworker/5:1-2669 [005] ...1 2023.148516: rtos_queue_peek_from_isr_failed: tstamp:63525173270 queue:0x0b4a3c58
Does this error point to anything?
ezaro
October 3, 2018, 8:12pm
10
To try and provide some additional information here. This only occurs when trying to use the video4linux interface to capture camera streams.
When running gstreamer with nvcamerasrc the synchronized cameras I am able to display 3 streams synchronized without issue. When using v4l2 I am unable to retrieve 3 synchronized streams, I am only able to use 2 at a time.
Is there a difference between the performance or operational limits of the data paths of using nvcamerasrc, which I believe uses the ISP data path, vs v4l2 which is the “bypass mode”?
Thank you
v4l2 pipeline: VI->MEMORY
ezaro
October 16, 2018, 10:26pm
12
Hi Shane,
Yes I understand that aspect of it. But I am curious as to why there appears to be a synced camera performance difference between nvcamerasrc pipeline and the v4l2 pipeline. I am continually unable to stream 3 synchronized cameras using the v4l2 pipeline, receiving the PXL_SOF errors, while I am able to stream 3 synchronized cameras using the nvcamerasrc pipeline.
Thanks
@ezaro
ezaro
October 18, 2018, 10:42pm
14
I tried that, doubling the values in the dtsi file and trying with the new DTB. I am still seeing the same results.
are the CHANSEL or CSI MUX configured differently when using the nvcamerasrc pipeline? Or is there some configuration that I can replicate on my v4l2 application? It’s interesting to me that the only error I see comes from the CHANSEL, which is a shared block between both of the pipelines. I wonder if it could be some type of configuration mismatch, but am not sure how to cross reference.
Thanks
Does any frame sync code in you sensor driver? Have a check it could be the root cause.
ezaro
November 12, 2018, 11:06pm
16
Shane,
After some investigation I found that the source of the issue came from a long delay time at the start of mipi_calibration. There is a tegra_mipi_wait call at the start of calibration that times out 500ms before continuing. This seems very excessive. Is there a reason the wait occurring in this function is so long? I have tested with removing it completely and setting it to a much lower value without seeing adverse effects.
#endif
static int _tegra_mipi_calibration(struct tegra_mipi *mipi, int lanes)
{
int err;
mutex_lock(&mipi->lock);
/* clean up lanes */
clear_all(mipi);
/* Apply MIPI_CAL PROD_Set */
if (lanes & (CSIA|CSIB|CSIC|CSID|CSIE|CSIF))
tegra_mipi_apply_csi_prod(mipi->regmap, mipi->prod_csi,
lanes);
else
tegra_mipi_apply_dsi_prod(mipi->regmap, mipi->prod_dsi,
lanes);
/*Select lanes */
select_lanes(mipi, lanes);
/* Start calibration */
[b] err = tegra_mipi_wait(mipi, lanes);
[/b]
#ifdef CONFIG_DEBUG_FS
regmap_read(mipi->regmap, ADDR(CIL_MIPI_CAL_STATUS), &mipical_status);
counts++;
if (err)
timeout_ct++;
#endif
mutex_unlock(&mipi->lock);
return err;
}
If your sensor can working well without calibration you can just remove it.