Crashing camera stream brings down entire video pipeline

JDSchroeder · August 28, 2020, 10:54pm

First, in my system I am frequently testing with multiple cameras (up to six simultaneously). Sometimes while I am testing over a long duration one of the camera streams will have an issue (unsure of root cause often times). The kernel prints a bunch of information on the debug console, but I cannot seem to even find out from the information generated which video input port had the issue.

For example, here is the output from a recent video pipeline crash:

[21945.341919] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 41 (15700000.vi_2) clientid 251440, HW thresh 922, done 923
[21945.353295] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 40 (15700000.vi_1) clientid 251440, HW thresh 303996, done 303996
[21945.365192] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 39 (15700000.vi_0) clientid 251440, HW thresh 303974, done 303974
[21945.377030] ---- mlocks ----
[21945.377043] 8: locked by channel 7

[21945.377061] ---- syncpts ----
[21945.377076] id 5 (disp_d) min 55940 max 55940 refs 1 (previous client : )
[21945.377079] id 6 (disp_e) min 2 max 2 refs 1 (previous client : )
[21945.377084] id 7 (disp_f) min 2 max 2 refs 1 (previous client : )
[21945.377088] id 8 (vblank1) min 1315645 max -4 refs 1 (previous client : )
[21945.377102] id 20 (gp10b_507) min 321216 max 321216 refs 1 (previous client : )
[21945.377106] id 21 (gp10b_506) min 114 max 114 refs 1 (previous client : )
[21945.377110] id 22 (gp10b_505) min 286344 max 286344 refs 1 (previous client : gp10b_505)
[21945.377113] id 23 (15340000.vic_argus_camera_0) min 753527 max 753527 refs 1 (previous client : 15340000.vic_argus_camera_0)
[21945.377117] id 24 (15600000.isp_nvargus-daemon_0) min 607864 max 607868 refs 4 (previous client : 15600000.isp_nvargus-daemon_0)
[21945.377122] id 25 (15600000.isp_nvargus-daemon_1) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_1)
[21945.377127] id 28 (15600000.isp_nvargus-daemon_2) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_2)
[21945.377131] id 29 (15600000.isp_nvargus-daemon_3) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_3)
[21945.377134] id 30 (15600000.isp_nvargus-daemon_4) min 2127725 max 2127734 refs 8 (previous client : 15600000.isp_nvargus-daemon_4)
[21945.377139] id 31 (15600000.isp_nvargus-daemon_5) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_5)
[21945.377144] id 33 (gp10b_504) min 68 max 68 refs 1 (previous client : gp10b_503)
[21945.377150] id 37 (15700000.vi_0) min 485 max 485 refs 2 (previous client : 15700000.vi_0)
[21945.377154] id 38 (gp10b_496) min 307536 max 307536 refs 1 (previous client : 15700000.vi_0)
[21945.377157] id 39 (15700000.vi_0) min 303974 max 303975 refs 4 (previous client : 15700000.vi_1)
[21945.377161] id 40 (15700000.vi_1) min 303996 max 303997 refs 4 (previous client : 15700000.vi_2)
[21945.377165] id 41 (15700000.vi_2) min 922 max 924 refs 4 (previous client : 150c0000.nvcsi_0)
[21945.377168] id 42 (15340000.vic_nvargus-daemon_0) min 815 max 815 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21945.377172] id 43 (gp10b_492) min 156 max 156 refs 1 (previous client : gp10b_491)
[21945.377175] id 44 (gp10b_500) min 100 max 100 refs 1 (previous client : gp10b_495)
[21945.377178] id 45 (15340000.vic_nvargus-daemon_0) min 152 max 152 refs 1 (previous client : gp10b_489)
[21945.377183] id 46 (gp10b_498) min 80 max 80 refs 1 (previous client : gp10b_494)
[21945.377186] id 47 (gp10b_499) min 64 max 64 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21945.377190] id 48 (gp10b_494) min 54 max 54 refs 1 (previous client : gp10b_499)
[21945.377193] id 49 (gp10b_489) min 80 max 80 refs 1 (previous client : gp10b_498)
[21945.377196] id 50 (gp10b_491) min 170 max 170 refs 1 (previous client : gp10b_496)
[21945.377200] id 51 (gp10b_493) min 240 max 240 refs 1 (previous client : gp10b_493)
[21945.377203] id 52 (gp10b_495) min 240 max 240 refs 1 (previous client : gp10b_490)
[21945.377207] id 53 (gp10b_490) min 228 max 228 refs 1 (previous client : gp10b_500)
[21945.377210] id 54 (150c0000.nvcsi_0) min 232 max 232 refs 2 (previous client : gp10b_492)
[21945.377213] id 55 (gp10b_497) min 964361 max 964361 refs 1 (previous client : gp10b_497)

[21945.377725] ---- channels ----
[21945.377735]
channel 2 - 15820000.se

[21945.377737] NvHost basic channel registers:
[21945.377740] CMDFIFO_STAT_0: 00002040
[21945.377743] CMDFIFO_RDATA_0: 60b66080
[21945.377747] CMDP_OFFSET_0: 00000000
[21945.377749] CMDP_CLASS_0: 00000000
[21945.377752] CHANNELSTAT_0: 00000000
[21945.377754] The CDMA sync queue is empty.

[21945.377758]
channel 3 - 15830000.se

[21945.377759] NvHost basic channel registers:
[21945.377762] CMDFIFO_STAT_0: 00002040
[21945.377764] CMDFIFO_RDATA_0: c190401c
[21945.377767] CMDP_OFFSET_0: 00000000
[21945.377770] CMDP_CLASS_0: 00000000
[21945.377772] CHANNELSTAT_0: 00000000
[21945.377773] The CDMA sync queue is empty.

[21945.377777]
channel 4 - 15840000.se

[21945.377779] NvHost basic channel registers:
[21945.377781] CMDFIFO_STAT_0: 00002040
[21945.377783] CMDFIFO_RDATA_0: 68090d15
[21945.377787] CMDP_OFFSET_0: 00000000
[21945.377789] CMDP_CLASS_0: 00000000
[21945.377791] CHANNELSTAT_0: 00000000
[21945.377793] The CDMA sync queue is empty.

[21945.377796]
channel 7 - 15600000.isp

[21945.377798] NvHost basic channel registers:
[21945.377800] CMDFIFO_STAT_0: 00004000
[21945.377803] CMDFIFO_RDATA_0: 004e0041
[21945.377806] CMDP_OFFSET_0: 00000050
[21945.377808] CMDP_CLASS_0: 00000001
[21945.377810] CHANNELSTAT_0: 00000000
[21945.377814]
ffffffc1e1593400: JOB, syncpt_id=24, syncpt_val=607866, first_get=00000000, timeout=10000, num_slots=19
[21945.377821] GATHER at ff5f8000+3c20, 12 words
[21945.377822] 20000001
[21945.377824] 00006418
[21945.377826] 20000001
[21945.377827] 00006819
[21945.377828] 20000001
[21945.377830] 00006c1c
[21945.377831] 20000001
[21945.377833] 00007c1d
[21945.377834] 20000001
[21945.377836] 0000741f
[21945.377837] 200c0001
[21945.377839] 00000005

[21945.377843] GATHER at ff5f8000+3c50, 2 words
[21945.377844] 20000001
[21945.377846] 0000001e

[21945.377851]
channel 6 - 150c0000.nvcsi

[21945.377853] NvHost basic channel registers:
[21945.377855] CMDFIFO_STAT_0: 00002040
[21945.377857] CMDFIFO_RDATA_0: 00000036
[21945.377861] CMDP_OFFSET_0: 00000000
[21945.377863] CMDP_CLASS_0: 00000001
[21945.377865] CHANNELSTAT_0: 00000000
[21945.377867] The CDMA sync queue is empty.

[21945.377870]
channel 1 - 15700000.vi

[21945.377872] NvHost basic channel registers:
[21945.377874] CMDFIFO_STAT_0: 00002040
[21945.377876] CMDFIFO_RDATA_0: 00000025
[21945.377880] CMDP_OFFSET_0: 00000000
[21945.377882] CMDP_CLASS_0: 00000001
[21945.377884] CHANNELSTAT_0: 00000000
[21945.377886] The CDMA sync queue is empty.

[21945.377888]
channel 5 - 15700000.vi

[21945.377890] NvHost basic channel registers:
[21945.377892] CMDFIFO_STAT_0: 0000001b
[21945.377894] CMDFIFO_RDATA_0: 304e0005
[21945.377898] CMDP_OFFSET_0: 00000050
[21945.377900] CMDP_CLASS_0: 00000001
[21945.377902] CHANNELSTAT_0: 00000000
[21945.377906]
ffffffc17d15b000: JOB, syncpt_id=41, syncpt_val=923, first_get=00000f10, timeout=10000, num_slots=7
[21945.377909] GATHER at ffd68000+0000, 6 words
[21945.377911] 304e0005
[21945.377912] 0020776d
[21945.377914] 0000001e
[21945.377915] 304e0005
[21945.377917] 0004a37b
[21945.377918] 00000028

[21945.377922] GATHER at ffd68000+0018, 29 words
[21945.377923] 90000002
[21945.377925] a0004008
[21945.377926] 00007f2f
[21945.377927] 00000b3f
[21945.377929] 9000000c
[21945.377930] a000400c
[21945.377932] 00000780
[21945.377933] 00000438
[21945.377934] 00000001
[21945.377936] 00000001
[21945.377937] 00000000
[21945.377939] 00000000
[21945.377940] 00000000
[21945.377942] 00000000
[21945.377943] 00000000
[21945.377944] 00000000
[21945.377946] 00000780
[21945.377947] 00000438
[21945.377949] 90000002
[21945.377950] a0004018
[21945.377951] 007f0000
[21945.377953] 00000104
[21945.377954] 90000002
[21945.377956] a0004021
[21945.377957] 00000000
[21945.377959] 00000000
[21945.377960] 90000001
[21945.377961] a0004045
[21945.377963] 0003d5ea

[21945.377966] GATHER at ffd68000+008c, 26 words
[21945.377968] 90000001
[21945.377969] a0000401
[21945.377970] 00000000
[21945.377972] 90000001
[21945.377973] a000404d
[21945.377975] 00000001
[21945.377976] 90000001
[21945.377978] a0004020
[21945.377979] 00000000
[21945.377980] 90000003
[21945.377982] a0004042
[21945.377983] 5597e000
[21945.377985] 00000001
[21945.377986] 00000001
[21945.377987] 90000001
[21945.377989] a0004048
[21945.377990] 00000000
[21945.377991] 90000001
[21945.377993] a000400a
[21945.377994] 00000000
[21945.377996] 90000001
[21945.377997] a0004001
[21945.377998] 00000001
[21945.378000] 90000001
[21945.378001] a0004007
[21945.378003] 00000003

[21945.378006] GATHER at ffd68000+00f4, 2 words
[21945.378007] 20000001
[21945.378009] 00000027

[21945.378013]
---- host general irq ----

[21945.378016] sync_intc0mask = 0x00000001
[21945.378018] sync_intmask = 0x5000000b
[21945.378020]
---- host syncpt irq mask ----

[21945.378021]
---- host syncpt irq status ----

[21945.378024] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[21945.378027] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[21945.378030] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[21945.378032] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[21945.378035] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[21945.378037] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[21945.378039] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[21945.378042] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[21945.378044] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[21945.378047] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[21945.378049] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[21945.378052] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[21945.378054] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[21945.378057] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[21945.378059] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[21945.378062] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[21945.378064] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[21945.378067] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[21945.378076] tegra-vi4 15700000.vi: SYNCPT_ID 41
[21945.378078] tegra-vi4 15700000.vi: SYNCPT_VAL 923
[21945.378080] tegra-vi4 15700000.vi: FIRST_GET 0xf10
[21945.378082] tegra-vi4 15700000.vi: TIMEOUT 10000
[21945.378084] tegra-vi4 15700000.vi: NUM_SLOTS 7
[21945.378087] tegra-vi4 15700000.vi: NUM_HANDLES 2
[21945.378096] tegra-vi4 15700000.vi: SYNCPT_ID 41
[21945.378098] tegra-vi4 15700000.vi: SYNCPT_VAL 924
[21945.378100] tegra-vi4 15700000.vi: FIRST_GET 0xf48
[21945.378102] tegra-vi4 15700000.vi: TIMEOUT 10000
[21945.378104] tegra-vi4 15700000.vi: NUM_SLOTS 7
[21945.378106] tegra-vi4 15700000.vi: NUM_HANDLES 2
[21946.740384] fence timeout on [ffffffc1e1c4ccc0] after 1500ms
[21946.740391] fence timeout on [ffffffc1ae6b6cc0] after 1500ms
[21946.740399] name=[nvhost_sync:29], current value=303932 waiting value=303933
[21946.740402] name=[nvhost_sync:24], current value=607864 waiting value=607865
[21946.740407] ---- mlocks ----
[21946.740408] ---- mlocks ----
[21946.740419] 8: locked by channel 7
[21946.740420] 8: locked by channel 7

[21946.740437] ---- syncpts ----
[21946.740438] ---- syncpts ----
[21946.740448] id 5 (disp_d) min 55943 max 55943 refs 1 (previous client : )
[21946.740451] id 6 (disp_e) min 2 max 2 refs 1 (previous client : )
[21946.740455] id 7 (disp_f) min 2 max 2 refs 1 (previous client : )
[21946.740459] id 8 (vblank1) min 1315727 max -4 refs 1 (previous client : )
[21946.740474] id 20 (gp10b_507) min 321360 max 321360 refs 1 (previous client : )
[21946.740477] id 21 (gp10b_506) min 114 max 114 refs 1 (previous client : )
[21946.740480] id 22 (gp10b_505) min 286360 max 286360 refs 1 (previous client : gp10b_505)
[21946.740484] id 23 (15340000.vic_argus_camera_0) min 753527 max 753527 refs 1 (previous client : 15340000.vic_argus_camera_0)
[21946.740488] id 24 (15600000.isp_nvargus-daemon_0) min 607864 max 607868 refs 4 (previous client : 15600000.isp_nvargus-daemon_0)
[21946.740491] id 25 (15600000.isp_nvargus-daemon_1) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_1)
[21946.740496] id 28 (15600000.isp_nvargus-daemon_2) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_2)
[21946.740500] id 29 (15600000.isp_nvargus-daemon_3) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_3)
[21946.740503] id 30 (15600000.isp_nvargus-daemon_4) min 2127725 max 2127734 refs 8 (previous client : 15600000.isp_nvargus-daemon_4)
[21946.740507] id 31 (15600000.isp_nvargus-daemon_5) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_5)
[21946.740512] id 33 (gp10b_504) min 68 max 68 refs 1 (previous client : gp10b_503)
[21946.740518] id 37 (15700000.vi_0) min 485 max 485 refs 2 (previous client : 15700000.vi_0)
[21946.740522] id 38 (gp10b_496) min 307536 max 307536 refs 1 (previous client : 15700000.vi_0)
[21946.740528] id 42 (15340000.vic_nvargus-daemon_0) min 815 max 815 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21946.740532] id 43 (gp10b_492) min 156 max 156 refs 1 (previous client : gp10b_491)
[21946.740535] id 44 (gp10b_500) min 100 max 100 refs 1 (previous client : gp10b_495)
[21946.740539] id 45 (15340000.vic_nvargus-daemon_0) min 152 max 152 refs 1 (previous client : gp10b_489)
[21946.740542] id 46 (gp10b_498) min 80 max 80 refs 1 (previous client : gp10b_494)
[21946.740545] id 47 (gp10b_499) min 64 max 64 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21946.740549] id 48 (gp10b_494) min 54 max 54 refs 1 (previous client : gp10b_499)
[21946.740552] id 49 (gp10b_489) min 80 max 80 refs 1 (previous client : gp10b_498)
[21946.740555] id 50 (gp10b_491) min 170 max 170 refs 1 (previous client : gp10b_496)
[21946.740559] id 51 (gp10b_493) min 240 max 240 refs 1 (previous client : gp10b_493)
[21946.740562] id 52 (gp10b_495) min 240 max 240 refs 1 (previous client : gp10b_490)
[21946.740565] id 53 (gp10b_490) min 228 max 228 refs 1 (previous client : gp10b_500)
[21946.740568] id 54 (150c0000.nvcsi_0) min 232 max 232 refs 2 (previous client : gp10b_492)
[21946.740572] id 55 (gp10b_497) min 964361 max 964361 refs 1 (previous client : gp10b_497)

[21946.741086] ---- channels ----
[21946.741092] id 5 (disp_d) min 55943 max 55943 refs 1 (previous client : )
[21946.741095] id 6 (disp_e) min 2 max 2 refs 1 (previous client : )
[21946.741097]
channel 2 - 15820000.se

[21946.741098] id 7 (disp_f) min 2 max 2 refs 1 (previous client : )
[21946.741099] NvHost basic channel registers:
[21946.741102] CMDFIFO_STAT_0: 00002040
[21946.741102] id 8 (vblank1) min 1315727 max -4 refs 1 (previous client : )
[21946.741105] CMDFIFO_RDATA_0: 60b66080
[21946.741109] CMDP_OFFSET_0: 00000000
[21946.741111] CMDP_CLASS_0: 00000000
[21946.741114] CHANNELSTAT_0: 00000000
[21946.741116] The CDMA sync queue is empty.
[21946.741116] id 20 (gp10b_507) min 321360 max 321360 refs 1 (previous client : )

[21946.741119] id 21 (gp10b_506) min 114 max 114 refs 1 (previous client : )
[21946.741120]
channel 3 - 15830000.se

[21946.741122] NvHost basic channel registers:
[21946.741122] id 22 (gp10b_505) min 286360 max 286360 refs 1 (previous client : gp10b_505)
[21946.741125] CMDFIFO_STAT_0: 00002040
[21946.741125] id 23 (15340000.vic_argus_camera_0) min 753527 max 753527 refs 1 (previous client : 15340000.vic_argus_camera_0)
[21946.741128] CMDFIFO_RDATA_0: c190401c
[21946.741129] id 24 (15600000.isp_nvargus-daemon_0) min 607864 max 607868 refs 4 (previous client : 15600000.isp_nvargus-daemon_0)
[21946.741132] id 25 (15600000.isp_nvargus-daemon_1) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_1)
[21946.741132] CMDP_OFFSET_0: 00000000
[21946.741134] CMDP_CLASS_0: 00000000
[21946.741136] CHANNELSTAT_0: 00000000
[21946.741137] id 28 (15600000.isp_nvargus-daemon_2) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_2)
[21946.741138] The CDMA sync queue is empty.

[21946.741140] id 29 (15600000.isp_nvargus-daemon_3) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_3)
[21946.741143]
channel 4 - 15840000.se

[21946.741144] id 30 (15600000.isp_nvargus-daemon_4) min 2127725 max 2127734 refs 8 (previous client : 15600000.isp_nvargus-daemon_4)
[21946.741145] NvHost basic channel registers:
[21946.741146] id 31 (15600000.isp_nvargus-daemon_5) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_5)
[21946.741147] CMDFIFO_STAT_0: 00002040
[21946.741149] CMDFIFO_RDATA_0: 68090d15
[21946.741151] id 33 (gp10b_504) min 68 max 68 refs 1 (previous client : gp10b_503)
[21946.741153] CMDP_OFFSET_0: 00000000
[21946.741155] CMDP_CLASS_0: 00000000
[21946.741157] id 37 (15700000.vi_0) min 485 max 485 refs 2 (previous client : 15700000.vi_0)
[21946.741158] CHANNELSTAT_0: 00000000
[21946.741159] The CDMA sync queue is empty.

[21946.741161] id 38 (gp10b_496) min 307536 max 307536 refs 1 (previous client : 15700000.vi_0)
[21946.741163]
channel 7 - 15600000.isp

[21946.741165] NvHost basic channel registers:
[21946.741166] id 42 (15340000.vic_nvargus-daemon_0) min 815 max 815 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21946.741167] CMDFIFO_STAT_0: 00004000
[21946.741169] CMDFIFO_RDATA_0: 004e0041
[21946.741170] id 43 (gp10b_492) min 156 max 156 refs 1 (previous client : gp10b_491)
[21946.741173] id 44 (gp10b_500) min 100 max 100 refs 1 (previous client : gp10b_495)
[21946.741174] CMDP_OFFSET_0: 00000050
[21946.741175] CMDP_CLASS_0: 00000001
[21946.741177] id 45 (15340000.vic_nvargus-daemon_0) min 152 max 152 refs 1 (previous client : gp10b_489)
[21946.741179] CHANNELSTAT_0: 00000000
[21946.741180] id 46 (gp10b_498) min 80 max 80 refs 1 (previous client : gp10b_494)
[21946.741182] id 47 (gp10b_499) min 64 max 64 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21946.741183]
ffffffc1e1593400: JOB, syncpt_id=24, syncpt_val=607866, first_get=00000000, timeout=10000, num_slots=19
[21946.741186] id 48 (gp10b_494) min 54 max 54 refs 1 (previous client : gp10b_499)
[21946.741187] GATHER at ff5f8000+3c20, 12 words
[21946.741189] id 49 (gp10b_489) min 80 max 80 refs 1 (previous client : gp10b_498)
[21946.741190] 20000001
[21946.741191] 00006418
[21946.741193] id 50 (gp10b_491) min 170 max 170 refs 1 (previous client : gp10b_496)
[21946.741193] 20000001
[21946.741194] 00006819
[21946.741196] 20000001
[21946.741196] id 51 (gp10b_493) min 240 max 240 refs 1 (previous client : gp10b_493)
[21946.741198] 00006c1c
[21946.741199] id 52 (gp10b_495) min 240 max 240 refs 1 (previous client : gp10b_490)
[21946.741200] 20000001
[21946.741200] 00007c1d
[21946.741202] 20000001
[21946.741202] id 53 (gp10b_490) min 228 max 228 refs 1 (previous client : gp10b_500)
[21946.741204] 0000741f
[21946.741205] id 54 (150c0000.nvcsi_0) min 232 max 232 refs 2 (previous client : gp10b_492)
[21946.741205] 200c0001
[21946.741206] 00000005

[21946.741208] id 55 (gp10b_497) min 964361 max 964361 refs 1 (previous client : gp10b_497)
[21946.741211] GATHER at ff5f8000+3c50, 2 words
[21946.741213] 20000001
[21946.741215] 0000001e

[21946.741220]
channel 6 - 150c0000.nvcsi

[21946.741222] NvHost basic channel registers:
[21946.741225] CMDFIFO_STAT_0: 00002040
[21946.741227] CMDFIFO_RDATA_0: 00000036
[21946.741230] CMDP_OFFSET_0: 00000000
[21946.741233] CMDP_CLASS_0: 00000001
[21946.741235] CHANNELSTAT_0: 00000000
[21946.741237] The CDMA sync queue is empty.

[21946.741241]
channel 1 - 15700000.vi

[21946.741242] NvHost basic channel registers:
[21946.741245] CMDFIFO_STAT_0: 00002040
[21946.741247] CMDFIFO_RDATA_0: 00000025
[21946.741250] CMDP_OFFSET_0: 00000000
[21946.741253] CMDP_CLASS_0: 00000001
[21946.741255] CHANNELSTAT_0: 00000000
[21946.741257] The CDMA sync queue is empty.

[21946.741260]
---- host general irq ----

[21946.741263] sync_intc0mask = 0x00000001
[21946.741265] sync_intmask = 0x50000003
[21946.741266]
---- host syncpt irq mask ----

[21946.741268]
---- host syncpt irq status ----

[21946.741271] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[21946.741273] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[21946.741276] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[21946.741278] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[21946.741281] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[21946.741283] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[21946.741286] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[21946.741288] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[21946.741290] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[21946.741293] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[21946.741295] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[21946.741298] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[21946.741300] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[21946.741303] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[21946.741305] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[21946.741308] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[21946.741310] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[21946.741312] syncpt_thresh_cpu0_int_status(17) = 0x00000000

[21946.741722] ---- channels ----
[21946.741729]
channel 2 - 15820000.se

[21946.741730] NvHost basic channel registers:
[21946.741734] CMDFIFO_STAT_0: 00002040
[21946.741736] CMDFIFO_RDATA_0: 60b66080
[21946.741740] CMDP_OFFSET_0: 00000000
[21946.741743] CMDP_CLASS_0: 00000000
[21946.741745] CHANNELSTAT_0: 00000000
[21946.741747] The CDMA sync queue is empty.

[21946.741750]
channel 3 - 15830000.se

[21946.741752] NvHost basic channel registers:
[21946.741754] CMDFIFO_STAT_0: 00002040
[21946.741756] CMDFIFO_RDATA_0: c190401c
[21946.741760] CMDP_OFFSET_0: 00000000
[21946.741762] CMDP_CLASS_0: 00000000
[21946.741764] CHANNELSTAT_0: 00000000
[21946.741766] The CDMA sync queue is empty.

[21946.741769]
channel 4 - 15840000.se

[21946.741771] NvHost basic channel registers:
[21946.741773] CMDFIFO_STAT_0: 00002040
[21946.741775] CMDFIFO_RDATA_0: 68090d15
[21946.741779] CMDP_OFFSET_0: 00000000
[21946.741781] CMDP_CLASS_0: 00000000
[21946.741783] CHANNELSTAT_0: 00000000
[21946.741785] The CDMA sync queue is empty.

[21946.741788]
channel 7 - 15600000.isp

[21946.741789] NvHost basic channel registers:
[21946.741792] CMDFIFO_STAT_0: 00004000
[21946.741794] CMDFIFO_RDATA_0: 004e0041
[21946.741798] CMDP_OFFSET_0: 00000050
[21946.741800] CMDP_CLASS_0: 00000001
[21946.741802] CHANNELSTAT_0: 00000000
[21946.741806]
ffffffc1e1593400: JOB, syncpt_id=24, syncpt_val=607866, first_get=00000000, timeout=10000, num_slots=19
[21946.741809] GATHER at ff5f8000+3c20, 12 words
[21946.741811] 20000001
[21946.741813] 00006418
[21946.741814] 20000001
[21946.741816] 00006819
[21946.741818] 20000001
[21946.741819] 00006c1c
[21946.741820] 20000001
[21946.741822] 00007c1d
[21946.741823] 20000001
[21946.741825] 0000741f
[21946.741826] 200c0001
[21946.741828] 00000005

[21946.741832] GATHER at ff5f8000+3c50, 2 words
[21946.741833] 20000001
[21946.741835] 0000001e

[21946.741840]
channel 6 - 150c0000.nvcsi

[21946.741841] NvHost basic channel registers:
[21946.741843] CMDFIFO_STAT_0: 00002040
[21946.741846] CMDFIFO_RDATA_0: 00000036
[21946.741849] CMDP_OFFSET_0: 00000000
[21946.741852] CMDP_CLASS_0: 00000001
[21946.741854] CHANNELSTAT_0: 00000000
[21946.741855] The CDMA sync queue is empty.

[21946.741859]
channel 1 - 15700000.vi

[21946.741860] NvHost basic channel registers:
[21946.741862] CMDFIFO_STAT_0: 00002040
[21946.741865] CMDFIFO_RDATA_0: 00000025
[21946.741868] CMDP_OFFSET_0: 00000000
[21946.741870] CMDP_CLASS_0: 00000001
[21946.741873] CHANNELSTAT_0: 00000000
[21946.741874] The CDMA sync queue is empty.

[21946.741878]
---- host general irq ----

[21946.741880] sync_intc0mask = 0x00000001
[21946.741883] sync_intmask = 0x50000003
[21946.741884]
---- host syncpt irq mask ----

[21946.741885]
---- host syncpt irq status ----

[21946.741888] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[21946.741891] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[21946.741893] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[21946.741896] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[21946.741898] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[21946.741901] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[21946.741903] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[21946.741906] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[21946.741908] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[21946.741911] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[21946.741913] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[21946.741916] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[21946.741918] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[21946.741921] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[21946.741923] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[21946.741926] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[21946.741928] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[21946.741930] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[21951.316398] host1x 13e10000.host1x: CaptureSchedule: syncpoint id 30 (15600000.isp_nvargus-daemon_4) stuck waiting 2127726, timeout=-1
[21951.328510] ---- syncpts ----
[21951.328523] id 5 (disp_d) min 55975 max 55975 refs 1 (previous client : )
[21951.328528] id 6 (disp_e) min 2 max 2 refs 1 (previous client : )
[21951.328532] id 7 (disp_f) min 2 max 2 refs 1 (previous client : )
[21951.328536] id 8 (vblank1) min 1316003 max -4 refs 1 (previous client : )
[21951.328552] id 20 (gp10b_507) min 323172 max 323172 refs 1 (previous client : )
[21951.328555] id 21 (gp10b_506) min 114 max 114 refs 1 (previous client : )
[21951.328559] id 22 (gp10b_505) min 286524 max 286524 refs 1 (previous client : gp10b_505)
[21951.328563] id 23 (15340000.vic_argus_camera_0) min 753527 max 753527 refs 1 (previous client : 15340000.vic_argus_camera_0)
[21951.328567] id 24 (15600000.isp_nvargus-daemon_0) min 607864 max 607868 refs 4 (previous client : 15600000.isp_nvargus-daemon_0)
[21951.328570] id 25 (15600000.isp_nvargus-daemon_1) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_1)
[21951.328576] id 28 (15600000.isp_nvargus-daemon_2) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_2)
[21951.328579] id 29 (15600000.isp_nvargus-daemon_3) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_3)
[21951.328583] id 30 (15600000.isp_nvargus-daemon_4) min 2127725 max 2127734 refs 8 (previous client : 15600000.isp_nvargus-daemon_4)
[21951.328586] id 31 (15600000.isp_nvargus-daemon_5) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_5)
[21951.328592] id 33 (gp10b_504) min 68 max 68 refs 1 (previous client : gp10b_503)
[21951.328598] id 37 (15700000.vi_0) min 485 max 485 refs 2 (previous client : 15700000.vi_0)
[21951.328601] id 38 (gp10b_496) min 307536 max 307536 refs 1 (previous client : 15700000.vi_0)
[21951.328608] id 42 (15340000.vic_nvargus-daemon_0) min 815 max 815 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21951.328611] id 43 (gp10b_492) min 156 max 156 refs 1 (previous client : gp10b_491)
[21951.328615] id 44 (gp10b_500) min 100 max 100 refs 1 (previous client : gp10b_495)
[21951.328618] id 45 (15340000.vic_nvargus-daemon_0) min 152 max 152 refs 1 (previous client : gp10b_489)
[21951.328621] id 46 (gp10b_498) min 80 max 80 refs 1 (previous client : gp10b_494)
[21951.328625] id 47 (gp10b_499) min 64 max 64 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21951.328629] id 48 (gp10b_494) min 54 max 54 refs 1 (previous client : gp10b_499)
[21951.328632] id 49 (gp10b_489) min 80 max 80 refs 1 (previous client : gp10b_498)
[21951.328635] id 50 (gp10b_491) min 170 max 170 refs 1 (previous client : gp10b_496)
[21951.328639] id 51 (gp10b_493) min 240 max 240 refs 1 (previous client : gp10b_493)
[21951.328642] id 52 (gp10b_495) min 240 max 240 refs 1 (previous client : gp10b_490)
[21951.328645] id 53 (gp10b_490) min 228 max 228 refs 1 (previous client : gp10b_500)
[21951.328649] id 54 (150c0000.nvcsi_0) min 232 max 232 refs 2 (previous client : gp10b_492)
[21951.328652] id 55 (gp10b_497) min 964361 max 964361 refs 1 (previous client : gp10b_497)

[21955.412377] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 24 (15600000.isp_nvargus-daemon_0) clientid 251424, HW thresh 607864, done 607866
[21955.425605] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 25 (15600000.isp_nvargus-daemon_1) clientid 251424, HW thresh 303932, done 303933
[21955.438827] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 28 (15600000.isp_nvargus-daemon_2) clientid 251424, HW thresh 303932, done 303933
[21955.452101] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 29 (15600000.isp_nvargus-daemon_3) clientid 251424, HW thresh 303932, done 303933
[21955.465380] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 31 (15600000.isp_nvargus-daemon_5) clientid 251424, HW thresh 303932, done 303933
[21955.478636] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 30 (15600000.isp_nvargus-daemon_4) clientid 251424, HW thresh 2127725, done 2127725
[21955.492027] ---- mlocks ----
[21955.492039] 8: locked by channel 7

[21955.492056] ---- syncpts ----
[21955.492066] id 5 (disp_d) min 55983 max 55983 refs 1 (previous client : )
[21955.492070] id 6 (disp_e) min 2 max 2 refs 1 (previous client : )
[21955.492074] id 7 (disp_f) min 2 max 2 refs 1 (previous client : )
[21955.492078] id 8 (vblank1) min 1316252 max -4 refs 1 (previous client : )
[21955.492093] id 20 (gp10b_507) min 323560 max 323560 refs 1 (previous client : )
[21955.492099] id 21 (gp10b_506) min 114 max 114 refs 1 (previous client : )
[21955.492102] id 22 (gp10b_505) min 286564 max 286564 refs 1 (previous client : gp10b_505)
[21955.492106] id 23 (15340000.vic_argus_camera_0) min 753527 max 753527 refs 1 (previous client : 15340000.vic_argus_camera_0)
[21955.492109] id 24 (15600000.isp_nvargus-daemon_0) min 607864 max 607868 refs 4 (previous client : 15600000.isp_nvargus-daemon_0)
[21955.492113] id 25 (15600000.isp_nvargus-daemon_1) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_1)
[21955.492118] id 28 (15600000.isp_nvargus-daemon_2) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_2)
[21955.492122] id 29 (15600000.isp_nvargus-daemon_3) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_3)
[21955.492125] id 30 (15600000.isp_nvargus-daemon_4) min 2127725 max 2127734 refs 8 (previous client : 15600000.isp_nvargus-daemon_4)
[21955.492128] id 31 (15600000.isp_nvargus-daemon_5) min 303932 max 303934 refs 4 (previous client : 15600000.isp_nvargus-daemon_5)
[21955.492133] id 33 (gp10b_504) min 68 max 68 refs 1 (previous client : gp10b_503)
[21955.492140] id 37 (15700000.vi_0) min 485 max 485 refs 2 (previous client : 15700000.vi_0)
[21955.492143] id 38 (gp10b_496) min 307536 max 307536 refs 1 (previous client : 15700000.vi_0)
[21955.492149] id 42 (15340000.vic_nvargus-daemon_0) min 815 max 815 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21955.492153] id 43 (gp10b_492) min 156 max 156 refs 1 (previous client : gp10b_491)
[21955.492156] id 44 (gp10b_500) min 100 max 100 refs 1 (previous client : gp10b_495)
[21955.492161] id 45 (15340000.vic_nvargus-daemon_0) min 152 max 152 refs 1 (previous client : gp10b_489)
[21955.492164] id 46 (gp10b_498) min 80 max 80 refs 1 (previous client : gp10b_494)
[21955.492167] id 47 (gp10b_499) min 64 max 64 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[21955.492172] id 48 (gp10b_494) min 54 max 54 refs 1 (previous client : gp10b_499)
[21955.492176] id 49 (gp10b_489) min 80 max 80 refs 1 (previous client : gp10b_498)
[21955.492179] id 50 (gp10b_491) min 170 max 170 refs 1 (previous client : gp10b_496)
[21955.492183] id 51 (gp10b_493) min 240 max 240 refs 1 (previous client : gp10b_493)
[21955.492186] id 52 (gp10b_495) min 240 max 240 refs 1 (previous client : gp10b_490)
[21955.492189] id 53 (gp10b_490) min 228 max 228 refs 1 (previous client : gp10b_500)
[21955.492193] id 54 (150c0000.nvcsi_0) min 232 max 232 refs 2 (previous client : gp10b_492)
[21955.492196] id 55 (gp10b_497) min 964361 max 964361 refs 1 (previous client : gp10b_497)

[21955.492743] ---- channels ----
[21955.492754]
channel 2 - 15820000.se

[21955.492757] NvHost basic channel registers:
[21955.492760] CMDFIFO_STAT_0: 00002040
[21955.492762] CMDFIFO_RDATA_0: 60b66080
[21955.492767] CMDP_OFFSET_0: 00000000
[21955.492770] CMDP_CLASS_0: 00000000
[21955.492772] CHANNELSTAT_0: 00000000
[21955.492774] The CDMA sync queue is empty.

[21955.492779]
channel 3 - 15830000.se

[21955.492781] NvHost basic channel registers:
[21955.492783] CMDFIFO_STAT_0: 00002040
[21955.492786] CMDFIFO_RDATA_0: c190401c
[21955.492789] CMDP_OFFSET_0: 00000000
[21955.492791] CMDP_CLASS_0: 00000000
[21955.492795] CHANNELSTAT_0: 00000000
[21955.492796] The CDMA sync queue is empty.

[21955.492800]
channel 4 - 15840000.se

[21955.492802] NvHost basic channel registers:
[21955.492804] CMDFIFO_STAT_0: 00002040
[21955.492807] CMDFIFO_RDATA_0: 68090d15
[21955.492810] CMDP_OFFSET_0: 00000000
[21955.492813] CMDP_CLASS_0: 00000000
[21955.492815] CHANNELSTAT_0: 00000000
[21955.492816] The CDMA sync queue is empty.

[21955.492820]
channel 7 - 15600000.isp

[21955.492821] NvHost basic channel registers:
[21955.492824] CMDFIFO_STAT_0: 00004000
[21955.492827] CMDFIFO_RDATA_0: 004e0041
[21955.492830] CMDP_OFFSET_0: 00000050
[21955.492833] CMDP_CLASS_0: 00000001
[21955.492836] CHANNELSTAT_0: 00000000
[21955.492839]
ffffffc1e1593400: JOB, syncpt_id=24, syncpt_val=607866, first_get=00000000, timeout=10000, num_slots=19
[21955.492843] GATHER at ff5f8000+3c20, 12 words
[21955.492846] 20000001
[21955.492848] 00006418
[21955.492850] 20000001
[21955.492851] 00006819
[21955.492853] 20000001
[21955.492854] 00006c1c
[21955.492856] 20000001
[21955.492858] 00007c1d
[21955.492859] 20000001
[21955.492861] 0000741f
[21955.492862] 200c0001
[21955.492864] 00000005

[21955.492868] GATHER at ff5f8000+3c50, 2 words
[21955.492869] 20000001
[21955.492871] 0000001e

[21955.492876]
channel 6 - 150c0000.nvcsi

[21955.492878] NvHost basic channel registers:
[21955.492881] CMDFIFO_STAT_0: 00002040
[21955.492883] CMDFIFO_RDATA_0: 00000036
[21955.492887] CMDP_OFFSET_0: 00000000
[21955.492889] CMDP_CLASS_0: 00000001
[21955.492892] CHANNELSTAT_0: 00000000
[21955.492894] The CDMA sync queue is empty.

[21955.492898]
channel 1 - 15700000.vi

[21955.492899] NvHost basic channel registers:
[21955.492903] CMDFIFO_STAT_0: 00002040
[21955.492905] CMDFIFO_RDATA_0: 00000025
[21955.492909] CMDP_OFFSET_0: 00000000
[21955.492911] CMDP_CLASS_0: 00000001
[21955.492914] CHANNELSTAT_0: 00000000
[21955.492915] The CDMA sync queue is empty.

[21955.492920]
---- host general irq ----

[21955.492923] sync_intc0mask = 0x00000001
[21955.492925] sync_intmask = 0x50000003
[21955.492926]
---- host syncpt irq mask ----

[21955.492928]
---- host syncpt irq status ----

[21955.492932] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[21955.492935] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[21955.492937] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[21955.492940] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[21955.492943] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[21955.492945] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[21955.492948] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[21955.492951] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[21955.492953] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[21955.492956] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[21955.492959] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[21955.492961] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[21955.492964] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[21955.492966] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[21955.492969] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[21955.492971] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[21955.492974] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[21955.492978] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[21955.513305] isp 15600000.isp: SYNCPT_ID 24
[21955.513309] isp 15600000.isp: SYNCPT_VAL 607866
[21955.513312] isp 15600000.isp: FIRST_GET 0x0
[21955.513314] isp 15600000.isp: TIMEOUT 10000
[21955.513317] isp 15600000.isp: NUM_SLOTS 19
[21955.513319] isp 15600000.isp: NUM_HANDLES 1
[21955.513467] isp 15600000.isp: SYNCPT_ID 30
[21955.513469] isp 15600000.isp: SYNCPT_VAL 2127727
[21955.513471] isp 15600000.isp: FIRST_GET 0x98
[21955.513474] isp 15600000.isp: TIMEOUT 10000
[21955.513476] isp 15600000.isp: NUM_SLOTS 12
[21955.513478] isp 15600000.isp: NUM_HANDLES 1
[21955.513521] isp 15600000.isp: SYNCPT_ID 30
[21955.513523] isp 15600000.isp: SYNCPT_VAL 2127729
[21955.513525] isp 15600000.isp: FIRST_GET 0xf8
[21955.513527] isp 15600000.isp: TIMEOUT 10000
[21955.513529] isp 15600000.isp: NUM_SLOTS 9
[21955.513532] isp 15600000.isp: NUM_HANDLES 1
[21955.513537] isp 15600000.isp: SYNCPT_ID 30
[21955.513570] isp 15600000.isp: SYNCPT_VAL 2127731
[21955.513572] isp 15600000.isp: FIRST_GET 0x140
[21955.513574] isp 15600000.isp: TIMEOUT 10000
[21955.513576] isp 15600000.isp: NUM_SLOTS 9
[21955.513578] isp 15600000.isp: NUM_HANDLES 5
[21955.513583] isp 15600000.isp: SYNCPT_ID 24
[21955.513585] isp 15600000.isp: SYNCPT_VAL 607868
[21955.513587] isp 15600000.isp: FIRST_GET 0x188
[21955.513589] isp 15600000.isp: TIMEOUT 10000
[21955.513621] isp 15600000.isp: NUM_SLOTS 19
[21955.513623] isp 15600000.isp: NUM_HANDLES 1
[21955.513750] isp 15600000.isp: SYNCPT_ID 30
[21955.513752] isp 15600000.isp: SYNCPT_VAL 2127734
[21955.513754] isp 15600000.isp: FIRST_GET 0x220
[21955.513756] isp 15600000.isp: TIMEOUT 10000
[21955.513759] isp 15600000.isp: NUM_SLOTS 12
[21955.513761] isp 15600000.isp: NUM_HANDLES 1
[21965.652376] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 24 (15600000.isp_nvargus-daemon_0) clientid 251424, HW thresh 607868, done 607870
[21965.665602] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 25 (15600000.isp_nvargus-daemon_1) clientid 251424, HW thresh 303934, done 303935
[21965.678824] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 28 (15600000.isp_nvargus-daemon_2) clientid 251424, HW thresh 303934, done 303935
[21965.692169] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 29 (15600000.isp_nvargus-daemon_3) clientid 251424, HW thresh 303934, done 303935
[21965.705436] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 31 (15600000.isp_nvargus-daemon_5) clientid 251424, HW thresh 303934, done 303935
[21965.718711] host1x 13e10000.host1x: cdma_handle_timeout: timeout: 30 (15600000.isp_nvargus-daemon_4) clientid 251424, HW thresh 2127739, done 2127739
[21965.732100] ---- mlocks ----
[21965.732117] 8: locked by channel 7

[21965.732137] ---- syncpts ----
[21965.732148] id 5 (disp_d) min 55984 max 55984 refs 1 (previous client : )
[21965.732151] id 6 (disp_e) min 2 max 2 refs 1 (previous client : )
[21965.732155] id 7 (disp_f) min 2 max 2 refs 1 (previous client : )
[21965.732158] id 8 (vblank1) min 1316867 max -4 refs 1 (previous client : )
[21965.732173] id 20 (gp10b_507) min 324132 max 324132 refs 1 (previous client : )
[21965.732176] id 21 (gp10b_506) min 114 max 114 refs 1 (previous client : )
[21965.732180] id 22 (gp10b_505) min 286570 max 286570 refs 1 (previous client : gp10b_505)
[21965.732184] id 23 (15340000.vic_argus_camera_0) min 753527 max 753527 refs 1 (previous client : 15340000.vic_argus_camera_0)
[21965.732187] id 24 (15600000.isp_nvargus-daemon_0) min 607868 max 607870 refs 2 (previous client : 15600000.isp_nvargus-daemon_0)
[21965.732192] id 25 (15600000.isp_nvargus-daemon_1) min 303934 max 303935 refs 2 (previous client : 15600000.isp_nvargus-daemon_1)
[21965.732197] id 28 (15600000.isp_nvargus-daemon_2) min 303934 max 303935 refs 2 (previous client : 15600000.isp_nvargus-daemon_2)
[21965.732200] id 29 (15600000.isp_nvargus-daemon_3) min 303934 max 303935 refs 2 (previous client : 15600000.isp_nvargus-daemon_3)
[21965.732205] id 30 (15600000.isp_nvargus-daemon_4) min 2127739 max 2127741 refs 3 (previous client : 15600000.isp_nvargus-daemon_4)
[21965.732208] id 31 (15600000.isp_nvargus-daemon_5) min 303934 max 303935 refs 2 (previous client : 15600000.isp_nvargus-daemon_5)
[21965.732236] id 55 (gp10b_497) min 964361 max 964361 refs 1 (previous client : gp10b_497)

[21965.732761] ---- channels ----
[21965.732773]
channel 2 - 15820000.se

[21965.732775] NvHost basic channel registers:
[21965.732778] CMDFIFO_STAT_0: 00002040
[21965.732781] CMDFIFO_RDATA_0: 60b66080
[21965.732785] CMDP_OFFSET_0: 00000000
[21965.732787] CMDP_CLASS_0: 00000000
[21965.732790] CHANNELSTAT_0: 00000000
[21965.732793] The CDMA sync queue is empty.

[21965.732797]
channel 3 - 15830000.se

[21965.732798] NvHost basic channel registers:
[21965.732801] CMDFIFO_STAT_0: 00002040
[21965.732803] CMDFIFO_RDATA_0: c190401c
[21965.732807] CMDP_OFFSET_0: 00000000
[21965.732809] CMDP_CLASS_0: 00000000
[21965.732811] CHANNELSTAT_0: 00000000
[21965.732813] The CDMA sync queue is empty.

[21965.732817]
channel 4 - 15840000.se

[21965.732818] NvHost basic channel registers:
[21965.732820] CMDFIFO_STAT_0: 00002040
[21965.732823] CMDFIFO_RDATA_0: 68090d15
[21965.732826] CMDP_OFFSET_0: 00000000
[21965.732828] CMDP_CLASS_0: 00000000
[21965.732831] CHANNELSTAT_0: 00000000
[21965.732832] The CDMA sync queue is empty.

[21965.732836]
channel 7 - 15600000.isp

[21965.732837] NvHost basic channel registers:
[21965.732840] CMDFIFO_STAT_0: 0000000e
[21965.732842] CMDFIFO_RDATA_0: 004e0041
[21965.732845] CMDP_OFFSET_0: 00000050
[21965.732848] CMDP_CLASS_0: 00000001
[21965.732850] CHANNELSTAT_0: 00000000
[21965.732854]
ffffffc1a042ec00: JOB, syncpt_id=24, syncpt_val=607870, first_get=00000310, timeout=10000, num_slots=19
[21965.732859] GATHER at ff5f8000+3c50, 12 words
[21965.732862] 20000001
[21965.732863] 00006418
[21965.732865] 20000001
[21965.732867] 00006819
[21965.732868] 20000001
[21965.732869] 00006c1c
[21965.732871] 20000001
[21965.732872] 00007c1d
[21965.732874] 20000001
[21965.732875] 0000741f
[21965.732877] 200c0001
[21965.732878] 00000005

[21965.732882] GATHER at ff5f8000+3c80, 2 words
[21965.732884] 20000001
[21965.732885] 0000001e

[21965.732892]
---- host general irq ----

[21965.732895] sync_intc0mask = 0x00000001
[21965.732897] sync_intmask = 0x50000003
[21965.732899]
---- host syncpt irq mask ----

[21965.732900]
---- host syncpt irq status ----

[21965.732903] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[21965.732906] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[21965.732908] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[21965.732911] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[21965.732913] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[21965.732916] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[21965.732919] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[21965.732921] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[21965.732925] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[21965.732927] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[21965.732930] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[21965.732932] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[21965.732935] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[21965.732937] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[21965.732940] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[21965.732942] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[21965.732944] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[21965.732947] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[21965.753282] isp 15600000.isp: SYNCPT_ID 24
[21965.753286] isp 15600000.isp: SYNCPT_VAL 607870
[21965.753288] isp 15600000.isp: FIRST_GET 0x310
[21965.753291] isp 15600000.isp: TIMEOUT 10000
[21965.753293] isp 15600000.isp: NUM_SLOTS 19
[21965.753295] isp 15600000.isp: NUM_HANDLES 1
[21965.753314] isp 15600000.isp: SYNCPT_ID 30
[21965.753316] isp 15600000.isp: SYNCPT_VAL 2127741
[21965.753318] isp 15600000.isp: FIRST_GET 0x3a8
[21965.753320] isp 15600000.isp: TIMEOUT 10000
[21965.753322] isp 15600000.isp: NUM_SLOTS 12
[21965.753324] isp 15600000.isp: NUM_HANDLES 1

So where in all that information is /dev/videoX is the stream that had an issue?

I am testing and using L4T 32.4.3 with argus_camera in ‘Multi Session’ mode. Frequently, when one camera stream has an issue it brings down the entire video input system. Sometimes restarting nvargus-daemon can allow restarting the cameras and other times an entire reboot is required. Are there any kernel options that can be enabled or out of tree kernel patches that can be applied that make the video pipelines more robust and independent of each other?

For example, if one camera crashes or has an issue and stops working, the expectation in a production system is that you would be able to independently restart just that camera and all of the other camera streams would continue to operate without interruption or an issue.

Hardware Configuration

Two 1080p cameras connected to CSI 0/1 using 4 data lanes w/ SerDes chip and Virtual Channel IDs
Two 1080p cameras connected to CSI 2/3 using 4 data lanes w/ SerDes chip and Virtual Channel IDs
Two 1080p cameras connected to CSI 4/5 using 4 data lanes w/ SerDes chip and Virtual Channel IDs

ShaneCCC · August 30, 2020, 4:02pm

Have you try to boost the system as performance to check?
sudo nvpmodel -m 0
sudo jetson_clocks&

D3_growe · August 31, 2020, 6:12pm

D3 has been working with NVIDIA on some kernel patches and an nvarguscamerasrc update to improve camera streaming robustness. These changes appear to be applicable to your situation. We would need to know more about your specific configuration and software environment to determine if we can provide early access. Please message me for more specific information.

Regards,
Greg

JDSchroeder · September 1, 2020, 1:41pm

I will try that, but that is not specifically what this post is about. Could you please review the log and tell me where in the debug log information the relevant data/line is that points me to which camera/port had an issue?

I will work with D3 to improve the robustness of the system, but I would also like NVIDIA to know that the error handling situation with the video pipelines and the nvargus-daemon seems woefully inadequate for a production system. If there is something that can be done in the kernel or nvargus-daemon to increase the reliability/robustness, even at the cost of performance/power, that would be much welcome for companies and individuals trying to productize a Jetson SoM system.

ShaneCCC · September 1, 2020, 4:10pm

The log tell the CSI/VI capture frame failed from sensors that’s could be the signal or bandwidth problem. You can boost the nvcsi/vi/isp clock to clarify the bandwidth.

Did you run with gstreamer command?

JDSchroeder · September 1, 2020, 4:18pm

Which CSI/VI port failed?

No, this is an output from running argus_camera.

ShaneCCC · September 1, 2020, 4:23pm

Have you apply those patches for camera from below link?
https://elinux.org/Jetson/L4T/r32.4.x_patches

JDSchroeder · September 1, 2020, 4:33pm

I have applied L4T R32.4.3 Breaks argus_camera and causes segmentation fault (SIGSEGV).

Which patch allows me to see which CSI/VI port failed and why?

Does my debug log and all of the information that NVIDIA prints tell you which CSI/VI port failed or not?

JerryChang · September 2, 2020, 2:14am

hello JDSchroeder,

since Argus and also camera driver expect sensor stream continuous sending without failures. we may need your confirmation of this, could you please check your hardware signaling if there’re intermittent failures.
also, it’s software approaches to wait for hardware signal with a sync-point; it’s expect that one of streaming failure might cause the pipeline crash.
please search the errors from the beginning; you should found timeout failures with waitCsiFrameEnd(), which should also indicate the stream-id.

FYI,
we’re having implementation of error resilience, but this feature is not included in r32.4.3 yet.

JDSchroeder · September 2, 2020, 2:11pm

Again. This thread is not really about lets see what we can try and find what might be going wrong. I’ll restate it once more:
Does my debug log and all of the information that NVIDIA prints tell you which CSI/VI port failed or not?

If the answer to the above, is “no” then what exactly does it tell you other than there was some kind of problem?

I’m not sure where waitCsiFrameEnd() is located. I searched the kernel sources and there is no match to be found.

Good to know. I’m sure this will be a welcome change for many who have tried to do camera work with the Jetson platform.

I am starting to understand that the debug console logging messages currently being printed when an error occurs are quite useless, even for NVIDIA to answer a simple question of, “Which camera port had an issue?”

JerryChang · September 2, 2020, 2:39pm

hello JDSchroeder,

due to different pipeline flows, you cannot found waitCsiFrameEnd() in the kernel sources. it’s located in the [Camera_Core].
you would need to check argus_daemon logs since you’re testing with argus_camera application.
please check /var/log/syslog , searching nvargus-daemon for the info.
or, you may disable original camera service, running it manually in the terminal to gather camera logs.
for example,

$ sudo pkill nvargus-daemon
$ sudo nvargus-daemon

please also share the logs for reference,
thanks

JerryChang · September 2, 2020, 3:14pm

hello JDSchroeder,

BTW,
you may also configure log levels before running argus_camera application for gather more details.
for example,

export enableCamScfLogs=5

JerryChang · September 3, 2020, 2:20am

hello JDSchroeder,

please moving to JetPack-4.4 GA / l4t-r32.4.3, and please also refer to Argus sample. i.e. Argus/public/samples/userAutoExposure,
there’s first stage of error resilience update included in JP-4.4 GA, Argus will report failures via EVENT_TYPE_ERROR, and the application has to shutdown.
thanks

JDSchroeder · September 3, 2020, 3:06pm

I am already testing and using L4T 32.4.3.

I understand there are other places to get error/log information. I’m curious what can be determined/understood from all the messages that are being output to the kernel log (see log output from post #1) when the video pipeline has an issue???

JerryChang · September 10, 2020, 5:29am

hello JDSchroeder,

usually we only trace to the beginning or failures, you may check the numbers of 15700000.vi_X for which camera device it is.
since your pipeline already crash, you might see all your camera devices were reporting errors, and your kernel logs were being overwritten due to lots of failures.
suggest you should setup another terminal, or using $ dmesg --follow to gather complete logs.

while your working with Argus camera, you may checking logs from daemon side.
here’s sample logs for camera crash,

Error: waitCsiFrameEnd timeout guid 3
VI Stream Id = 3 Virtual Channel = 0
CSI Stream Id = 3 Brick Id = 1