MIPI CSI capture: crash in hier_ictlr driver

We are developing soc camera based v4l2 driver for Omni Vision sensor OV4689 on jetson tk1 board using 21.4 L4T release. OV camera is connected on CSI0 interface of tegra. We are able to get images from camera but we have observed that after long run (more than 10 hours continuous), we get CSI timeout error and after that kernel crash occurs. There is no memory leak or CPU usage is below 10% during test. Crash occurs due to bug statement in file linux-tegra-3.10.40/drivers/platform/tegra/hier_ictlr/hier_ictlr.c line 54.

What could be reason for crash after long run ?
Why bug statement returned in hier_ictlr.c line 54 ?
how to recover hier_ictlr if any such error is encounter instead of crash of complete system ?

Following is the crash log.

[ 291.153281] vi vi.0: CSI_A syncpt timeout, syncpt = 1196, err = -11
[ 291.160041] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 291.164854] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 291.169618] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 291.174335] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 291.179068] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 291.183825] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 291.188542] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 291.194045] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 291.199529] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 291.204317] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 291.409258] vi vi.0: CSI_A syncpt timeout, syncpt = 1197, err = -11
[ 291.415717] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 291.420446] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 291.425066] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 291.429787] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 291.434513] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 291.439275] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 291.443977] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 291.449457] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 291.454968] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 291.459794] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 291.664118] vi vi.0: CSI_A syncpt timeout, syncpt = 1198, err = -11
[ 291.670511] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 291.675341] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 291.680110] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 291.684832] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 291.689534] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 291.694325] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 291.699029] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 291.704590] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 291.710085] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 291.714889] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 291.919182] vi vi.0: CSI_A syncpt timeout, syncpt = 1199, err = -11
[ 291.925840] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 291.930662] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 291.935371] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 291.940238] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 291.944987] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 291.949696] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 291.954398] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 291.959968] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 291.965508] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 291.970298] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 292.175285] vi vi.0: CSI_A syncpt timeout, syncpt = 1200, err = -11
[ 292.181668] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 292.186533] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 292.191170] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 292.195873] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 292.200622] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 292.205335] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 292.210098] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 292.215610] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 292.221090] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 292.225877] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 292.430097] vi vi.0: CSI_A syncpt timeout, syncpt = 1201, err = -11
[ 292.436508] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 292.441423] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 292.446150] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 292.451212] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 292.455928] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 292.460629] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 292.465329] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 292.470810] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 292.476301] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 292.481099] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 292.686102] vi vi.0: CSI_A syncpt timeout, syncpt = 1202, err = -11
[ 292.692631] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 292.697485] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 292.702195] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 292.707054] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 292.711776] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 292.716479] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 292.721211] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 292.726700] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 292.732236] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 292.737050] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 292.941246] vi vi.0: CSI_A syncpt timeout, syncpt = 1203, err = -11
[ 292.947696] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 292.952475] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 292.957127] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 292.961835] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 292.966591] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 292.971367] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 292.976127] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 292.981622] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 292.987115] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 292.991919] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 293.196227] vi vi.0: CSI_A syncpt timeout, syncpt = 1204, err = -11
[ 293.202657] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 293.207384] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 293.212009] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 293.216719] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 293.221431] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 293.226225] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 293.230926] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 293.236406] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 293.241886] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 293.246700] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 293.451097] vi vi.0: CSI_A syncpt timeout, syncpt = 1205, err = -11
[ 293.457720] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 293.462529] TEGRA_CSI_CSI_CILA_STATUS 0x00000000
[ 293.467283] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 293.472179] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 293.476902] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 293.481611] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 293.486310] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 293.491791] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 293.497280] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 293.502088] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 293.523962] vi vi.0: Failed to create debugfs directory
[ 293.728951] vi vi.0: CSI_A syncpt timeout, syncpt = 1206, err = -11
[ 293.735264] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 293.739992] TEGRA_CSI_CSI_CILA_STATUS 0x00000001
[ 293.744652] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 293.749367] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 293.754082] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 293.758895] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 293.763637] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 293.769157] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 293.774640] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 293.779428] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 293.984153] vi vi.0: CSI_A syncpt timeout, syncpt = 1207, err = -11
[ 293.990537] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 293.995378] TEGRA_CSI_CSI_CILA_STATUS 0x00000001
[ 294.000151] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 294.004875] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 294.009579] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 294.014457] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 294.019164] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 294.024721] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 294.030219] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 294.035034] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 294.239268] vi vi.0: CSI_A syncpt timeout, syncpt = 1208, err = -11
[ 294.245784] TEGRA_CSI_CSI_CIL_A_STATUS 0x00000000
[ 294.250591] TEGRA_CSI_CSI_CILA_STATUS 0x00000001
[ 294.255297] TEGRA_CSI_CSI_CIL_B_STATUS 0x00000000
[ 294.260239] TEGRA_CSI_CSI_CIL_C_STATUS 0x00000000
[ 294.264959] TEGRA_CSI_CSI_CIL_D_STATUS 0x00000000
[ 294.269709] TEGRA_CSI_CSI_CIL_E_STATUS 0x00000000
[ 294.274412] TEGRA_CSI_CSI_PIXEL_PARSER_A_STATUS 0x00000000
[ 294.280038] TEGRA_CSI_CSI_PIXEL_PARSER_B_STATUS 0x00000000
[ 294.285532] TEGRA_VI_CSI_0_ERROR_STATUS 0x00000000
[ 294.290331] TEGRA_VI_CSI_1_ERROR_STATUS 0x00000000
[ 294.495121] vi vi.0: CSI_A syncpt timeout, syncpt = 1209, err = -11
[ 295.077501] Host read timeout at address 5408093c
[ 295.082458] MSELECT error detected! status=0x1
[ 295.087084] ------------[ cut here ]------------
[ 295.091757] kernel BUG at /home/sandip/linux-tegra-3.10.40/drivers/platform/tegra/hier_ictlr/hier_ictlr.c:54!
[ 295.105003] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[ 295.110884] Modules linked in: tegra_camera videobuf2_dma_contig ov4689_v4l2 [last unloaded: ov4689_v4l2]
[ 295.127819] CPU: 0 PID: 29 Comm: kworker/0:1 Not tainted 3.10.40 #2
[ 295.134168] Workqueue: events tegra_camera_work [tegra_camera]
[ 295.140090] task: edb08580 ti: edb66000 task.ti: edb66000
[ 295.145569] PC is at tegra_hier_ictlr_irq_handler+0x3c/0x40
[ 295.151192] LR is at tegra_hier_ictlr_irq_handler+0x3c/0x40
[ 295.156816] pc : [] lr : [] psr: 20000193
[ 295.156816] sp : edb67ce0 ip : 00000000 fp : c10aee30
[ 295.168331] r10: 00000000 r9 : 00000000 r8 : edb66000
[ 295.173600] r7 : edb67d80 r6 : 0000009f r5 : c0f73910 r4 : ed7a1580
[ 295.180171] r3 : 00000002 r2 : 00010100 r1 : c21f470c r0 : 00000022
[ 295.186750] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 295.194190] Control: 10c5387d Table: acb8c06a DAC: 00000015
[ 295.199980]
[ 295.199980] PC: 0xc06098c8:
[ 295.204294] 98c8 e8bd4010 eaf85d08 e92d4010 e52de004 e8bd4000 e1a04000 e59000a8 ebffffba
[ 295.212928] 98e8 e3500000 08bd8010 e1a01004 e8bd4010 eaf85d63 e52de004 e8bd4000 e3a00004
[ 295.221553] 9908 e12fff1e e92d4008 e52de004 e8bd4000 e1a00001 ebf81ea0 e590300c e5931060
[ 295.230178] 9928 f57ff04f e3510000 1a000001 e3a00001 e8bd8008 e30606ac e34c00a7 eb07aa4b
[ 295.238797] 9948 e7f001f2 e92d40f0 e24dd014 e52de004 e8bd4000 e2805010 e1a06000 e3a01010
[ 295.247412] 9968 e3a020d0 e1a00005 ebf82c8a e2504000 0a000058 e1a01004 e1a00005 ebf81f22
[ 295.256030] 9988 e3a01c02 e1a00006 e3a02000 ebf82699 e2501000 0a000051 e1a00005 ebf2d27c
[ 295.264647] 99a8 e3500000 e5840008 0a000051 e3a01c02 e1a00006 e3a02001 ebf8268e e2501000
[ 295.273268]
[ 295.273268] LR: 0xc06098c8:
[ 295.277580] 98c8 e8bd4010 eaf85d08 e92d4010 e52de004 e8bd4000 e1a04000 e59000a8 ebffffba
[ 295.286204] 98e8 e3500000 08bd8010 e1a01004 e8bd4010 eaf85d63 e52de004 e8bd4000 e3a00004
[ 295.294820] 9908 e12fff1e e92d4008 e52de004 e8bd4000 e1a00001 ebf81ea0 e590300c e5931060
[ 295.303437] 9928 f57ff04f e3510000 1a000001 e3a00001 e8bd8008 e30606ac e34c00a7 eb07aa4b
[ 295.312050] 9948 e7f001f2 e92d40f0 e24dd014 e52de004 e8bd4000 e2805010 e1a06000 e3a01010
[ 295.320667] 9968 e3a020d0 e1a00005 ebf82c8a e2504000 0a000058 e1a01004 e1a00005 ebf81f22
[ 295.329285] 9988 e3a01c02 e1a00006 e3a02000 ebf82699 e2501000 0a000051 e1a00005 ebf2d27c
[ 295.337902] 99a8 e3500000 e5840008 0a000051 e3a01c02 e1a00006 e3a02001 ebf8268e e2501000
[ 295.346523]
[ 295.346523] SP: 0xedb67c60:
[ 295.350837] 7c60 00000000 00000000 00000000 00000000 c10cddfe 00000022 00000000 00000000
[ 295.359459] 7c80 c060994c 00000000 c000ef98 00000000 e7100000 c000ecec 00000022 c21f470c
[ 295.368076] 7ca0 00010100 00000002 ed7a1580 c0f73910 0000009f edb67d80 edb66000 00000000
[ 295.376690] 7cc0 00000000 c10aee30 00000000 edb67ce0 c0609948 c0609948 20000193 ffffffff
[ 295.385308] 7ce0 c060990c c00d287c b2d9af4a 00000044 c10aee1c c0f738c0 ee003880 c0f738c0
[ 295.393928] 7d00 c0f73910 ed7a1580 edb67d80 edb66000 fdc42000 c10aee08 c104b144 c00d2ad4
[ 295.402544] 7d20 c0f738c0 c0f73910 edb67e60 c00d5aa4 c00d5a18 c0f67364 0000009f c00d2090
[ 295.411162] 7d40 000003c0 c000f984 0000009f fdc4200c c0fa0638 c00084e8 c10ee2c0 c006c868
[ 295.419786]
[ 295.419786] FP: 0xc10aedb0:
[ 295.424098] edb0 00000000 00000000 c0aa6b80 00000000 00000000 00000000 00000000 c0aa6b70
[ 295.432725] edd0 00000000 00000000 00000000 00000000 c0aa6b60 00000000 00000000 00000000
[ 295.441339] edf0 00000000 c0aa6b48 00000000 00000000 00000000 00000000 c0aa6b54 00000000
[ 295.449958] ee10 00000000 00000000 00000000 c0aa6b90 00000000 00000000 00000000 00000000
[ 295.458575] ee30 c0aa6ba4 00000000 00000000 00000000 00000000 c0aa6bc8 00000000 00000000
[ 295.467190] ee50 00000000 00000000 c0aa6bb8 00000000 00000000 00000000 00000000 c0aa6c60
[ 295.475811] ee70 00000000 00000000 00000000 00000000 c0aa6c84 00000000 00000000 00000000
[ 295.484428] ee90 00000000 c0aa6c70 00000000 00000000 00000000 00000000 c0aa6ca4 00000000
[ 295.493052]
[ 295.493052] R1: 0xc21f468c:
[ 295.497364] 468c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.505989] 46ac 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.514606] 46cc 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.523220] 46ec 00000000 00000003 00000000 c21f46fc 00000001 000000fe 00000001 00000007
[ 295.531840] 470c 00000000 c0066d78 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.540456] 472c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.549072] 474c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.557688] 476c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.566307]
[ 295.566307] R4: 0xed7a1500:
[ 295.570619] 1500 ed7a1180 c1090514 c061b9ac 00000000 00000000 00000000 00000000 00000000
[ 295.579240] 1520 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.587865] 1540 65647768 616c0070 726f6674 72642f6d 72657669 65742f73 2d617267 72656968
[ 295.596486] 1560 7463692d 0000726c 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.605105] 1580 c060990c c0fabcd0 00000000 00000000 00000000 00000000 0000009f 00000004
[ 295.613721] 15a0 00000000 00000000 c0a76774 ed6d7240 00000000 00000000 00000000 00000000
[ 295.622345] 15c0 c0fabe20 ed7a1600 c00d61fc 00000000 0000009f c0fabcd0 00000000 00000000
[ 295.630969] 15e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.639591]
[ 295.639591] R5: 0xc0f73890:
[ 295.643906] 3890 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.652528] 38b0 00000000 00000000 00000000 00000000 0000009f 0000009f 00000000 00042004
[ 295.661152] 38d0 c104b9a0 ee0038c0 00000000 c0fa0638 00000000 0000000f c0f6af60 c00d5a18
[ 295.669779] 38f0 ed7a1580 00000104 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.678394] 3910 00080008 00000000 00000000 00000000 00000000 00000000 00000000 c0f7392c
[ 295.687018] 3930 c0f7392c edb4b0c0 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.695636] 3950 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.704251] 3970 00000000 00000000 00000000 00000000 000000a0 000000a0 00000000 00002004
[ 295.712871]
[ 295.712871] R7: 0xedb67d00:
[ 295.717186] 7d00 c0f73910 ed7a1580 edb67d80 edb66000 fdc42000 c10aee08 c104b144 c00d2ad4
[ 295.725814] 7d20 c0f738c0 c0f73910 edb67e60 c00d5aa4 c00d5a18 c0f67364 0000009f c00d2090
[ 295.734432] 7d40 000003c0 c000f984 0000009f fdc4200c c0fa0638 c00084e8 c10ee2c0 c006c868
[ 295.743043] 7d60 20000113 ffffffff edb67db4 edb66008 fdc42000 c10aedb8 c104b144 c000ec40
[ 295.751659] 7d80 00000002 ffffecae c10ee2c0 00000000 60000193 00000082 00000000 00000000
[ 295.760276] 7da0 edb66008 fdc42000 c10aedb8 c104b144 00000000 edb67dc8 c10ee2c0 c006c868
[ 295.768891] 7dc0 20000113 ffffffff a8bf840b 00000044 6c703d4d 00000000 c0f65af8 c10ee2c0
[ 295.777508] 7de0 0000000a ffffecb0 c0f6c0c0 04208060 c104b144 60000193 0000001d 00000000
[ 295.786134]
[ 295.786134] R8: 0xedb65f80:
[ 295.790447] 5f80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.799065] 5fa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.807684] 5fc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.816302] 5fe0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.824920] 6000 00000002 00010101 00000000 edb08580 c1037ef8 00000000 00000015 c21f74c0
[ 295.833541] 6020 edb08580 c0fa87c0 c0fa04e8 ed331200 00000000 c0f684c0 edb67dac edb67d00
[ 295.842165] 6040 c07f8224 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.850784] 6060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 295.859412] Process kworker/0:1 (pid: 29, stack limit = 0xedb66238)
[ 295.865721] Stack: (0xedb67ce0 to 0xedb68000)
[ 295.870136] 7ce0: c060990c c00d287c b2d9af4a 00000044 c10aee1c c0f738c0 ee003880 c0f738c0
[ 295.878363] 7d00: c0f73910 ed7a1580 edb67d80 edb66000 fdc42000 c10aee08 c104b144 c00d2ad4
[ 295.886590] 7d20: c0f738c0 c0f73910 edb67e60 c00d5aa4 c00d5a18 c0f67364 0000009f c00d2090
[ 295.894817] 7d40: 000003c0 c000f984 0000009f fdc4200c c0fa0638 c00084e8 c10ee2c0 c006c868
[ 295.903041] 7d60: 20000113 ffffffff edb67db4 edb66008 fdc42000 c10aedb8 c104b144 c000ec40
[ 295.911264] 7d80: 00000002 ffffecae c10ee2c0 00000000 60000193 00000082 00000000 00000000
[ 295.919491] 7da0: edb66008 fdc42000 c10aedb8 c104b144 00000000 edb67dc8 c10ee2c0 c006c868
[ 295.927715] 7dc0: 20000113 ffffffff a8bf840b 00000044 6c703d4d 00000000 c0f65af8 c10ee2c0
[ 295.935943] 7de0: 0000000a ffffecb0 c0f6c0c0 04208060 c104b144 60000193 0000001d 00000000
[ 295.944170] 7e00: edb67e60 edb66000 fdc42000 c10aee08 c104b144 c006cb30 edb66020 c006cdf8
[ 295.952393] 7e20: c0f67364 c000f988 0000001d fdc4200c c0fa0638 c00084e8 bf06b718 c07f4278
[ 295.960618] 7e40: 60000113 ffffffff edb67e94 ec8426d0 ed081010 00000001 c104b144 c000ec40
[ 295.968841] 7e60: bf06c490 00000000 00000000 fe480000 ec842610 fffffff5 00000001 ec80c400
[ 295.977064] 7e80: ec8426d0 ed081010 00000001 c104b144 00000000 edb67ea8 bf06b718 c07f4278
[ 295.985290] 7ea0: 60000113 ffffffff 00000000 00000000 ec80c400 000004b4 ec80c000 ec8426f0
[ 295.993516] 7ec0: ec80c400 ec842610 00000007 bf069224 c032574c ec842700 fffffff5 ec8426d4
[ 296.001741] 7ee0: c21f7040 ec8426f0 edb521c0 c10aef98 c21f7040 c21fae00 00000000 c21fae05
[ 296.009968] 7f00: edb66008 c0083910 c21fce00 c0081310 00000000 00000000 c21fce00 edb521c0
[ 296.018194] 7f20: c21f7040 edb521d8 c21f7054 edb66000 c10ae984 00000009 c21f7040 c0084654
[ 296.026417] 7f40: edb08580 edb66000 ee0ffea8 edb521c0 c0084520 00000000 00000000 00000000
[ 296.034642] 7f60: 00000000 c008ab9c c103faf0 00000000 ee0ae040 edb521c0 00000000 00000000
[ 296.042868] 7f80: edb67f80 edb67f80 00000000 00000000 edb67f90 edb67f90 ee0ffea8 c008aac8
[ 296.051091] 7fa0: 00000000 00000000 00000000 c000f118 00000000 00000000 00000000 00000000
[ 296.059316] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 296.067539] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 296.075821] [] (tegra_hier_ictlr_irq_handler+0x3c/0x40) from [] (handle_irq_event_percpu+0x7c/0x290)
[ 296.086766] [] (handle_irq_event_percpu+0x7c/0x290) from [] (handle_irq_event+0x44/0x64)
[ 296.096664] [] (handle_irq_event+0x44/0x64) from [] (handle_fasteoi_irq+0x8c/0x154)
[ 296.106126] [] (handle_fasteoi_irq+0x8c/0x154) from [] (generic_handle_irq+0x30/0x40)
[ 296.115776] [] (generic_handle_irq+0x30/0x40) from [] (handle_IRQ+0x48/0x98)
[ 296.124626] [] (handle_IRQ+0x48/0x98) from [] (gic_handle_irq+0x4c/0x140)
[ 296.133211] [] (gic_handle_irq+0x4c/0x140) from [] (__irq_svc+0x40/0x70)
[ 296.141685] Exception stack(0xedb67d80 to 0xedb67dc8)
[ 296.146790] 7d80: 00000002 ffffecae c10ee2c0 00000000 60000193 00000082 00000000 00000000
[ 296.155018] 7da0: edb66008 fdc42000 c10aedb8 c104b144 00000000 edb67dc8 c10ee2c0 c006c868
[ 296.163232] 7dc0: 20000113 ffffffff
[ 296.166812] [] (__irq_svc+0x40/0x70) from [] (__do_softirq+0xb8/0x2d0)
[ 296.175148] [] (__do_softirq+0xb8/0x2d0) from [] (do_softirq+0x58/0x60)
[ 296.183567] [] (do_softirq+0x58/0x60) from [] (irq_exit+0xa4/0xd8)
[ 296.191551] [] (irq_exit+0xa4/0xd8) from [] (handle_IRQ+0x4c/0x98)
[ 296.199531] [] (handle_IRQ+0x4c/0x98) from [] (gic_handle_irq+0x4c/0x140)
[ 296.208114] [] (gic_handle_irq+0x4c/0x140) from [] (__irq_svc+0x40/0x70)
[ 296.216588] Exception stack(0xedb67e60 to 0xedb67ea8)
[ 296.221692] 7e60: bf06c490 00000000 00000000 fe480000 ec842610 fffffff5 00000001 ec80c400
[ 296.229916] 7e80: ec8426d0 ed081010 00000001 c104b144 00000000 edb67ea8 bf06b718 c07f4278
[ 296.238131] 7ea0: 60000113 ffffffff
[ 296.241704] [] (__irq_svc+0x40/0x70) from [] (printk+0x0/0x4c)
[ 296.249344] [] (printk+0x0/0x4c) from [] (0xec8426f0)
[ 296.256188] Code: e8bd8008 e30606ac e34c00a7 eb07aa4b (e7f001f2)
[ 296.262365] —[ end trace d3120d85a5857108 ]—
[ 296.267033] Kernel panic - not syncing: Fatal exception in interrupt
[ 296.273450] CPU3: stopping
[ 296.276239] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D 3.10.40 #2
[ 296.283320] [] (unwind_backtrace+0x0/0x134) from [] (show_stack+0x18/0x1c)
[ 296.292009] [] (show_stack+0x18/0x1c) from [] (handle_IPI+0x194/0x1a8)
[ 296.300333] [] (handle_IPI+0x194/0x1a8) from [] (gic_handle_irq+0x84/0x140)
[ 296.309094] [] (gic_handle_irq+0x84/0x140) from [] (__irq_svc+0x40/0x70)
[ 296.317571] Exception stack(0xee287f08 to 0xee287f50)
[ 296.322680] 7f00: ee287f58 00000000 00000000 0000019f 0000019f 00000044
[ 296.330907] 7f20: c22117d8 00000000 f9d2acae 00000044 00000000 ee286000 3b9ac9ff ee287f50
[ 296.339126] 7f40: c02b8514 c003dde0 20040013 ffffffff
[ 296.344254] [] (__irq_svc+0x40/0x70) from [] (tegra_idle_enter_clock_gating+0x58/0x78)
[ 296.353983] [] (tegra_idle_enter_clock_gating+0x58/0x78) from [] (cpuidle_enter_state+0x48/0xec)
[ 296.364574] [] (cpuidle_enter_state+0x48/0xec) from [] (cpuidle_idle_call+0xf8/0x29c)
[ 296.374207] [] (cpuidle_idle_call+0xf8/0x29c) from [] (arch_cpu_idle+0x10/0x40)
[ 296.383326] [] (arch_cpu_idle+0x10/0x40) from [] (cpu_startup_entry+0x14c/0x274)
[ 296.392522] [] (cpu_startup_entry+0x14c/0x274) from [<807ee1e4>] (0x807ee1e4)
[ 296.400577] CPU2: stopping
[ 296.403374] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D 3.10.40 #2
[ 296.410449] [] (unwind_backtrace+0x0/0x134) from [] (show_stack+0x18/0x1c)
[ 296.419140] [] (show_stack+0x18/0x1c) from [] (handle_IPI+0x194/0x1a8)
[ 296.427471] [] (handle_IPI+0x194/0x1a8) from [] (gic_handle_irq+0x84/0x140)
[ 296.436239] [] (gic_handle_irq+0x84/0x140) from [] (__irq_svc+0x40/0x70)
[ 296.444725] Exception stack(0xee285f08 to 0xee285f50)
[ 296.449841] 5f00: ee285f58 00000000 00000000 000001a2 000001a2 00000044
[ 296.458080] 5f20: c22087d8 00000000 f9d2a0f6 00000044 00000000 ee284000 3b9ac9ff ee285f50
[ 296.466306] 5f40: c02b8514 c003dde0 20000113 ffffffff
[ 296.471441] [] (__irq_svc+0x40/0x70) from [] (tegra_idle_enter_clock_gating+0x58/0x78)
[ 296.481174] [] (tegra_idle_enter_clock_gating+0x58/0x78) from [] (cpuidle_enter_state+0x48/0xec)
[ 296.491767] [] (cpuidle_enter_state+0x48/0xec) from [] (cpuidle_idle_call+0xf8/0x29c)
[ 296.501413] [] (cpuidle_idle_call+0xf8/0x29c) from [] (arch_cpu_idle+0x10/0x40)
[ 296.510539] [] (arch_cpu_idle+0x10/0x40) from [] (cpu_startup_entry+0x14c/0x274)
[ 296.519752] [] (cpu_startup_entry+0x14c/0x274) from [<807ee1e4>] (0x807ee1e4)

Are you doing continuous image capture for 10 hours and then observed crash? I know this is a long experiment but how many runs this was observed? Once? The BUG() will trigger if there is timeout from host bus of a particular client such as CSI sensor. One thing you could experiment is to change BUG()to WARN() so it won’t hang when hit it. However, still need to dig out where the error coming from.

Also, OV4689 is bayer sensor, right? Can you select YUV sensor instead?

yes, OV4689 sensor is bayer and it is not possible to change sensor for us now.

we replaced BUG() statement by commenting BUG() line but what we observed is that tegra continuously generates interrupt ‘tegra_hier_ictlr_irq_handle’. so tegra is no more usable even after commenting this line.

as per our further analysis, Another observation is crash is observed at the end of our test, after v4l2 device(/dev/video0) close in our application. in our test application(standard v4l2 based application) we provide number of frames to capture. Once all frames are captured, our application performs STREAMOFF and then closes /dev/video0 device node.

When we run multiple time our application (with just 200 frames to capture), then crash is observed once we close /dev/video0 node.

In our 10 hours test also, crash is observed once we close /dev/video0 device.

Crash can be observed with standard gstreamer command when gstreamer command is run multiple time. we use following command to capture images from sensor and display over hdmi.
gst-launch-1.0 v4l2src device="/dev/video0" ! ‘video/x-bayer,format=bggr,width=2688,height=1520’ ! bayer2rgb ! videoconvert ! xvimagesink sync=false

When you said,
crash is observed once we close /dev/video0 node. at both short 200 frames of capture or long 10 hours running. What does it actually mean? The hang occurs right after you close the node? But the program is supposed to exit once close, no? Am I missing something here?

Like to see if there is a way to shorten repro step to make debugging easier.

There is a document about V4L2 driver: V4L2 User Guide for Jetson TK1 under Developer’s Guide. It uses IMX 135 as a sample sensor to describe its driver development. not sure if you have chance to check it out.

“Crash can be observed with standard gstreamer command when gstreamer command is run multiple time. we use following command to capture images from sensor and display over hdmi.
gst-launch-1.0 v4l2src device=”/dev/video0" ! ‘video/x-bayer,format=bggr,width=2688,height=1520’ ! bayer2rgb ! videoconvert ! xvimagesink sync=false

=> we can try to see if we could duplicate this issue using our own camera sensor.

yes. crash occurs after close of /dev/video0 node. after close of node program just exists.

after further debug we found that crash occurs from workqueue tegra_camera_work. this workqueue was not cancelled in streamoff ioctl. so when application performed streamoff and then close, still workqueue was running and it was trying to capture frame but sensor capture was turned off in streamoff ioctl. After applying following fix we have not observed the crash. (fix is to cancel workqueue in streamoff ioctl call)

static int tegra_camera_stop_streaming(struct vb2_queue *q)
{
struct soc_camera_device *icd = container_of(q,
struct soc_camera_device,
vb2_vidq);
struct soc_camera_host *ici = to_soc_camera_host(icd->parent);
struct tegra_camera_dev *cam = ici->priv;
struct soc_camera_subdev_desc *ssdesc = &icd->sdesc->subdev_desc;
struct tegra_camera_platform_data *pdata = ssdesc->drv_priv;
int port = pdata->port;

  •    // cancel the work queue
    
  •    cancel_work_sync(&cam->work);
    
      mutex_lock(&cam->work_mutex);
      cam->ops->capture_stop(cam, port);
      mutex_unlock(&cam->work_mutex);
    
      return 0;
    

}

we have observed another crash when we use multiple times gstreamer command to capture and display over HDMI. In this scenario, crash occurs when we run gstreamer command again. Earlier fix we applied was for crash happening after driver close, but here crash happens next time when tegra tries to capture frame. crash logs are similar first post of this ticket. Following gstreamer command is used. To run this command multiple time, we use ctrl+c or killall gst-launch-1.0 command to close current command.

gst-launch-1.0 v4l2src device="/dev/video0" ! ‘video/x-bayer,format=bggr,width=2688,height=1520’ ! bayer2rgb ! videoconvert ! xvimagesink sync=false

To over come this crash, workaround is to unload and load tegra_camera driver again before running next gstreamer command. After further debugging we found that sometimes there is error observed in vi2_capture_stop call in file vi2.c. there is no action taken in case there is error in vi2_capture_stop function. and when this error occurs, next time frame capture shows error and then kernel crash occurs. if we unload and load tegra_camera driver in case of error in vi2_capture_stop function then on next capture there is no error. after analyzing what happens in tegra_camera driver unload and load, we implemented following workaround in vi2_capture_stop function to fix this issue.


+static void vi2_sw_reset(struct tegra_camera_dev *cam);
static int vi2_capture_stop(struct tegra_camera_dev *cam, int port)
{
u32 val;
@@ -1079,6 +1080,23 @@ static int vi2_capture_stop(struct tegra_camera_dev *cam, int port)
NULL);
TC_VI_REG_WT(cam, TEGRA_CSI_PIXEL_STREAM_PPB_COMMAND, 0xf002);
}

  •   if (err)
    
  •   {
    
  •           printk("vi2_capture_stop ret=%d\n",err);
    
  •           if (port == TEGRA_CAMERA_PORT_CSI_A)
    
  •                   dev_err(&cam->ndev->dev,
    
  •                           "CSI_A syncpt timeout, syncpt = %d, err = %d\n",
    
  •                           cam->syncpt_csi_a, err);
    
  •           else if (port == TEGRA_CAMERA_PORT_CSI_B ||
    
  •                           port == TEGRA_CAMERA_PORT_CSI_C)
    
  •                   dev_err(&cam->ndev->dev,
    
  •                           "CSI_B/CSI_C syncpt timeout, syncpt = %d, err = %d\n",
    
  •                           cam->syncpt_csi_b, err);
    
  •           vi2_capture_error_status(cam);
    
  •           vi2_sw_reset(cam);
    
  •           vi2_free_syncpts(cam);
    
  •           vi2_init_syncpts(cam);
    
  •   }
    
      return err;
    

}


Hope this will be helpful to other developers who encounters similar issue with other sensors.

one more issue with vi2 driver is it is not possible to do VIDIOC_STREAMON again once VIDIOC_STREAMOFF is called without closing v4l2 driver (/dev/video0) and again configuring driver to capture again.

Our application opens driver just once and then starts/stops capture based on user request and driver close is never called. we observed that once we stop capture using VIDIOC_STREAMOFF ioctl then again it is not possible to start capture using QBUF and VIDIOC_STREAMON ioctl.

after further debug we found that function vi2_capture_setup returns without configuring CSI0 again if sof flag is zero. to fix this issue we not set sof flag to 1 in vi2_capture_stop function. after this fix we are able to start/stop capture operation without closing driver. following is the patch


-1079,6 +1085,25 @@ static int vi2_capture_stop(struct tegra_camera_dev *cam, int port)
NULL);
TC_VI_REG_WT(cam, TEGRA_CSI_PIXEL_STREAM_PPB_COMMAND, 0xf002);
}

  •   if (err)
    
  •   {
    
  •           printk("vi2_capture_stop ret=%d\n",err);
    
  •           if (port == TEGRA_CAMERA_PORT_CSI_A)
    
  •                   dev_err(&cam->ndev->dev,
    
  •                           "CSI_A syncpt timeout, syncpt = %d, err = %d\n",
    
  •                           cam->syncpt_csi_a, err);
    
  •           else if (port == TEGRA_CAMERA_PORT_CSI_B ||
    
  •                           port == TEGRA_CAMERA_PORT_CSI_C)
    
  •                   dev_err(&cam->ndev->dev,
    
  •                           "CSI_B/CSI_C syncpt timeout, syncpt = %d, err = %d\n",
    
  •                           cam->syncpt_csi_b, err);
    
  •           vi2_capture_error_status(cam);
    
  •           vi2_sw_reset(cam);
    
  •           vi2_free_syncpts(cam);
    
  •           vi2_init_syncpts(cam);
    
  •   }
    
  •   // set sof to 1 so that when streamon is again called, CSI is configured again
    
  •   cam->sof = 1;
    
      return err;
    

}

Hope this will be helpful to other developers.

I am seeing this same kernel crash in our system if the CSI data stops. We are using an HDMI to CSI bridge chip which will stop sending CSI data when the HDMI input is unplugged while capturing. I expect the continuous syncpt timeouts but not the crash. The crash happens after about 6-8 syncpt timeouts. I have not yet found a solution to this problem.

Hi Merwin,

For syncpt timeouts you mert, we did the update in another thread you posted:
https://devtalk.nvidia.com/default/topic/976606/jetson-tk1/kernel-hang-with-tc358840-hdmi-to-csi-2-bridge-on-tk1-when-data-stops/post/5023233/#5023233

Thanks

Hi,

I am planning to interface multiple Small size spy camera(Mini size cameras) to the TK1 Board (Any interface USB,CSI etc.)for capturing 360 degree view.

Is that possible to interface spy cameras on TK1 Board? which spy camera would be suitable for the TK1 Board.

Hoping for the earnest reply.

Regards,
Saurabh Pahune

Hi Saurabh,

You could check the TK1 wiki for more information and discussions of USB cameras and CSI cameras, stereo cameras and Time-Of-Flight 3D depth cameras:
http://elinux.org/Jetson/Cameras

Thanks

1 Like