[V4L2] Vi capture release failed leading to a Kernel oops

Mobilaz · April 18, 2019, 2:15pm

Hello,

I am using the release 32.1 of the L4T and I am trying to connect four OV24A1B sensors over four 4 CSI lanes to the Jetson Xavier platform.

I am using the V4L2 driver IOCTL to capture images from the sensors and I am successful in doing so.

One of my test is to do the following sequence multiple times in a row:

Start sensor and configure it
Start Streaming
Capture images
Stop Streaming
Close the sensor

The sequence works and I can capture images without any issues until after multiple runs (it can go up to 400-500 sequences without any issues) I get a kernel oops when releasing the resources:

Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.465956] tegra194-vi5 15c10000.vi: unexpected response from camera processor
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.466225] video4linux video3: vi capture release failed
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.466351] tegra194-vi5 15c10000.vi: unexpected response from camera processor
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.466949] tegra194-vi5 15c10000.vi: vi_capture_release: control failed, errno 1
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.467208] Unable to handle kernel NULL pointer dereference at virtual address 00000000
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.467444] Mem abort info:
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.467527]   ESR = 0x96000005
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.467607]   Exception class = DABT (current EL), IL = 32 bits
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.467717]   SET = 0, FnV = 0
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.467781]   EA = 0, S1PTW = 0
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.467875] Data abort info:
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.467951]   ISV = 0, ISS = 0x00000005
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.468042]   CM = 0, WnR = 0
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.468146] user pgtable: 4k pages, 39-bit VAs, pgd = ffffffc3b2061000
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.468285] [0000000000000000] *pgd=0000000000000000
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.469883] ov24a1b 30-0036: ov24a1b_power_off: power off
Jan 28 18:23:06 nvidia-desktop kernel: [ 5099.471682] , *pud=0000000000000000<0>[ 5099.471699] Internal error: Oops: 96000005 [#1] PREEMPT SMP
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.477477] Modules linked in:
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.480647] CPU: 5 PID: 17823 Comm: v4l2_single_cam Not tainted 4.9.140+ #4
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.487459] Hardware name: jetson-xavier (DT)
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.492088] task: ffffffc3a2714600 task.stack: ffffffc39ece8000
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.498138] PC is at vi_capture_request_unpin+0x48/0x108
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.503296] LR is at vi_capture_request_unpin+0x44/0x108
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.508975] pc : [<ffffff8008b154e0>] lr : [<ffffff8008b154dc>] pstate: 20400045
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.516410] sp : ffffffc39eceba30
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.519823] x29: ffffffc39eceba30 x28: ffffffc3aa5caf00
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.525267] x27: 0000000000000001 x26: ffffffc39ecebd10
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.531108] x25: 0000000000000000 x24: ffffffc373bcfe68
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.536884] x23: ffffffc373bcfc00 x22: ffffffc3d8ddab58
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.542239] x21: 0000000000000000 x20: ffffffc373bcfc00
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.547995] x19: ffffffc38949e300 x18: 0000000000000010
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.553527] x17: 000000000002a74f x16: 000088b30cc50c73
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.559375] x15: 0000000000000006 x14: 6e6f63203a657361
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.564813] x13: 656c65725f657275 x12: 747061635f697620
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.570655] x11: 3a69762e30303030 x10: 3163353120356976
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.576013] x9 : 0000000000000000 x8 : ffffffc39eceb840
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.582034] x7 : ffffffc37440b440 x6 : ffffffc35d819c7f
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.587546] x5 : ffffff80085218ec x4 : ffffffbf0d760650
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.592885] x3 : 000000018040003e x2 : ffffffc35d819c80
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.597973] x1 : ffffffc373bcfe68 x0 : 0000000000000000
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.603326]
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.604727] Process v4l2_single_cam (pid: 17823, stack limit = 0xffffffc39ece8000)
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.611967] Call trace:
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.614537] [<ffffff8008b154e0>] vi_capture_request_unpin+0x48/0x108
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.620373] [<ffffff8008b146ac>] vi_capture_shutdown+0x8c/0x120
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.625965] [<ffffff8008b15644>] vi_channel_close_ex+0x2c/0x88
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.631654] [<ffffff8008b165bc>] vi5_channel_stop_streaming+0x64/0xb0
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.637778] [<ffffff8008b097f4>] tegra_channel_stop_streaming+0x34/0x48
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.643993] [<ffffff8008b01c4c>] __vb2_queue_cancel+0x3c/0x170
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.649337] [<ffffff8008b02000>] vb2_core_streamoff+0x58/0xc0
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.654840] [<ffffff8008b062fc>] vb2_streamoff+0x54/0x78
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.660003] [<ffffff8008b06378>] vb2_ioctl_streamoff+0x58/0x68
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.665348] [<ffffff8008ae0e94>] v4l_streamoff+0x3c/0x50
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.670940] [<ffffff8008ae5c7c>] __video_do_ioctl+0x1bc/0x288
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.676458] [<ffffff8008ae5694>] video_usercopy+0x2cc/0x6a8
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.682330] [<ffffff8008ae5aac>] video_ioctl2+0x3c/0x50
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.687136] [<ffffff8008adf940>] v4l2_ioctl+0xf8/0x128
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.692300] [<ffffff800826c760>] do_vfs_ioctl+0xb0/0x8f8
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.697889] [<ffffff800826d03c>] SyS_ioctl+0x94/0xa8
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.702808] [<ffffff80080838c0>] el0_svc_naked+0x34/0x38
Jan 28 18:23:07 nvidia-desktop kernel: [ 5099.708331] ---[ end trace 53200a8e277556bb ]--

I am so far unable to locate which one is the null pointer.
Do you have any ideas what I am doing wrong or how can I solve this issue ?

Thanks in advance,

VickNV · April 22, 2019, 6:04am

Hi Mobilaz,

It seems that we haven’t hit “unexpected response from camera processor” before.
Could you try if running the sequence multiple times with other camera boards will also hit the issue?

JerryChang · April 22, 2019, 6:04am

hello Mobilaz,

please check below

could you please share your commands to enable IOCTL to capture images, could it be simplify to v4l2-ctl commands and we could also test from our side.
it might be stability issue since the procedure could pass 400-500 sequences. could you let us know how many minutes it takes till the failure reproduced.
could you also analysis the memory usage, thanks

Mobilaz · April 23, 2019, 2:25pm

Hello,

Thanks for your answers.

Please find the answers to your questions below:

IOCTL sequence:

VIDIOC_REQBUFS
VIDIOC_QUERYBUF
VIDIOC_QBUF
VIDIOC_STREAMON
For loop to DQ/Q the buffers: VIDIOC_DQBUF/VIDIOC_QBUF
VIDIOC_STREAMOFF
VIDIOC_REQBUFS with the number of buffers requested equal to 0 to release the buffers

I am still working on a solution you could test on your side.

It seems to be after 1h - 1h30
When the crash happened, the free memory was around 12Gb out of 16Gb

While working on this issue, with the same set up, I got the following error after 190 iterations (~30min):

Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.454613] CPU:0, Error:RCE-NOC@0xbe00000,irq=491
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.454806] **************************************
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.454948] * For more Internal Decode Help
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.455074] *     http://nv/cbberr
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.455181] * NVIDIA userID is required to access
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.455309] **************************************
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.455589] CPU:0, Error:RCE-NOC
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.455706]   Error Logger            : 1
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.455836]   ErrLog0                 : 0x80030600
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.455951]     Transaction Type      : RD  - Read, Incrementing
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.456092]     Error Code            : TMO
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.456195]     Error Source          : Target NIU
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.456316]     Error Description     : Target time-out error
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.456457]     Packet header Lock    : 0
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.456567]     Packet header Len1    : 3
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.456706]     NOC protocol version  : version >= 2.7
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.456845]   ErrLog1                 : 0x157600
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.456952]   ErrLog2                 : 0x0
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.457044]     RouteId               : 0x157600
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.457151]     InitFlow              : cpu_p_i/I/0
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.457263]     Targflow              : cbb_t/T/0
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.457468]     TargSubRange          : 27
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.457574]     SeqId                 : 0
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.457668]   ErrLog3                 : 0x5c01400
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.457818]   ErrLog4                 : 0x0
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.460334]     Address               : 0x15c01400 -- /host1x/slvs-ec@15ac0000 + 0x1400
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.466832]   ErrLog5                 : 0x387e31
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.469998]     Master ID             : RCE
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.473143]     Security Group(GRPSEC): 0x3f
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.477193]     Cache                 : 0x1 -- Device
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.481179]     Protection            : 0x3 -- Privileged, Non-Secure, Data Access
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.487825]     FALCONSEC             : 0x0
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.490978]     Virtual Queuing Channel(VQC): 0x0
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.495698]   **************************************
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326] ###################### RTCPU EXCEPTION ######################
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326] [RCE] Data abort
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   R0:  00000001 R1:  ffffffcf R2:  15c00000 R3:  15c01400
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   R4:  0bca2c28 R5:  00000000 R6:  78015300 R7:  00000010
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   R8:  000000a0 R9:  00000000 R10: 0bca36e8 R11: 0bca3788
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   R12: 0bcb51d0 SP:  0bcdf800 LR:  0bc8a638 PC:  0bc80308
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   SP: 0bcb5df0 LR: 0bc847bc
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   CPSR: 20000197 SPSR: 2000001f
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   DFSR: 00001008 DFAR: 15c01400 ADFSR: 00000000
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   IFSR: 00000000 IFAR: 00000000 AIFSR: 00000000
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326] Callstack
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   [0bcb5dec]: 0bc8a5f8
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   [0bcb5e90]: 0bc8a5dc
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   [0bcb5ff4]: 0bca01f9
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   [0bcb6b78]: 0bc83610
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   [0bcb6b7c]: 0bc848c8
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   [0bcb6ba4]: 0bc92844
Jan 28 17:59:54 nvidia-desktop kernel: [ 3708.505326]   [0bcb6be8]: 0bc92820
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.528652] CPU:0, Error:CBB-NOC@0x2300000,irq=485
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.528997] **************************************
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.529276] * For more Internal Decode Help
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.529394] *     http://nv/cbberr
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.529492] * NVIDIA userID is required to access
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.529608] **************************************
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.529731] CPU:0, Error:CBB-NOC
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.530777]   Error Logger            : 1
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.533948]   ErrLog0                 : 0x80030600
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.537512]     Transaction Type      : RD  - Read, Incrementing
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.542765]     Error Code            : TMO
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.546442]     Error Source          : Target NIU
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.550462]     Error Description     : Target time-out error
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.555800]     Packet header Lock    : 0
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.559473]     Packet header Len1    : 3
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.563062]     NOC protocol version  : version >= 2.7
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.567703]   ErrLog1                 : 0x9528aa
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.571373]   ErrLog2                 : 0x0
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.573827]     RouteId               : 0x9528aa
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.577329]     InitFlow              : rce_p2ps/I/rce_p2ps
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.581554]     Targflow              : host1x_p2pm/T/host1x_p2pm
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.586776]     TargSubRange          : 20
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.590185]     SeqId                 : 0
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.592815]   ErrLog3                 : 0x1400
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.595789]   ErrLog4                 : 0x0
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.598880]     Address               : 0x15c01400 -- /host1x/slvs-ec@15ac0000 + 0x1400
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.605067]   ErrLog5                 : 0x2af0fc71
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.608741]     Non-Modify            : 0x1
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.612236]     AXI ID                : 0x55
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.615386]     Master ID             : RCE
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.618361]     Security Group(GRPSEC): 0x3f
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.622393]     Cache                 : 0x1 -- Device
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.626071]     Protection            : 0x3 -- Privileged, Non-Secure, Data Access
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.632896]     FALCONSEC             : 0x0
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.636048]     Virtual Queuing Channel(VQC): 0x0
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.640681]   **************************************
Jan 28 17:59:55 nvidia-desktop kernel: [ 3708.774474] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!

I am still trying to isolate the issue.

Kind regards,

Mobilaz · April 26, 2019, 3:06pm

Hi,

I did another test where I let the sensors streaming while never stopping them:

VIDIOC_REQBUFS
VIDIOC_QUERYBUF
VIDIOC_QBUF
VIDIOC_STREAMON
For loop to DQ/Q the buffers: VIDIOC_DQBUF/VIDIOC_QBUF

The test ran successfully over a full night (more than 12hours) without any issue.

The issue seems to be linked with the power on/off sequence of the sensors.

Kind regards,

JerryChang · April 29, 2019, 7:38am

hello Mobilaz,

may I have more details about your testing setups.
thanks

(1) how many cameras did you launching for testing.
(2) following (1), had you tried to reduce number of camera for testing.
(3) could you please review power-off sequence in the sensor specification.

Mobilaz · April 30, 2019, 1:14pm

Hello,

Thanks for your answer.

(1) We used 4 sensors → failed
3 sensors → failed
2 sensors → Crashed once but the two others tests last more than 12 hours (need more tests)
1 sensors → I couldn’t make it crash but I also didn’t do a lot of these tests (I have only one Jetson available at the moment)

(2) Yes see above

(3) The sensor power-off sequence is similar to the power sequence of the ov5693 sensor but I am still investigating this to double check everything.

I increased the sleep time between a STREAMOFF and a POWER OFF which seems to increase the stability of the system but it eventually crashed too (after 10 hours instead of 1-2hours usually)

Thanks in advance,

Mobilaz · May 7, 2019, 7:15am

Hello,

I tried to probe some of the signals while running the same test and I got the following error on a sensor:

May  1 02:29:16 nvidia-desktop kernel: [34406.314687] ov24a1b 30-0036: Success to start stream
May  1 02:29:16 nvidia-desktop kernel: [34406.323644] ov24a1b 30-0036: ov24a1b_s_stream--
May  1 02:29:16 nvidia-desktop kernel: [34406.323788] ov24a1b 30-0036: ov24a1b_s_stream--
May  1 02:29:16 nvidia-desktop kernel: [34406.324452] video4linux video0: failed to run kthread for capture enqueue
May  1 02:29:16 nvidia-desktop kernel: [34406.324622] Unable to handle kernel NULL pointer dereference at virtual address 0000002c
May  1 02:29:16 nvidia-desktop kernel: [34406.324784] Mem abort info:
May  1 02:29:16 nvidia-desktop kernel: [34406.324870]   ESR = 0x96000005
May  1 02:29:16 nvidia-desktop kernel: [34406.324923]   Exception class = DABT (current EL), IL = 32 bits
May  1 02:29:16 nvidia-desktop kernel: [34406.325037]   SET = 0, FnV = 0
May  1 02:29:16 nvidia-desktop kernel: [34406.325087]   EA = 0, S1PTW = 0
May  1 02:29:16 nvidia-desktop kernel: [34406.325135] Data abort info:
May  1 02:29:16 nvidia-desktop kernel: [34406.325181]   ISV = 0, ISS = 0x00000005
May  1 02:29:16 nvidia-desktop kernel: [34406.325241]   CM = 0, WnR = 0
May  1 02:29:16 nvidia-desktop kernel: [34406.325295] user pgtable: 4k pages, 39-bit VAs, pgd = ffffffc343f37000
May  1 02:29:16 nvidia-desktop kernel: [34406.325426] [000000000000002c] *pgd=0000000000000000, *pud=0000000000000000
May  1 02:29:16 nvidia-desktop kernel: [34406.325630] Internal error: Oops: 96000005 [#1] PREEMPT SMP
May  1 02:29:16 nvidia-desktop kernel: [34406.325761] Modules linked in:
May  1 02:29:16 nvidia-desktop kernel: [34406.325873] CPU: 1 PID: 17714 Comm: v4l2_single_cam Tainted: G        W       4.9.140+ #22
May  1 02:29:16 nvidia-desktop kernel: [34406.326053] Hardware name: jetson-xavier (DT)
May  1 02:29:16 nvidia-desktop kernel: [34406.326345] task: ffffffc3a15e7000 task.stack: ffffffc33b1d4000
May  1 02:29:16 nvidia-desktop kernel: [34406.326821] PC is at kthread_stop+0x2c/0x1e8
May  1 02:29:16 nvidia-desktop kernel: [34406.329153] LR is at vi5_channel_stop_kthreads+0x30/0x58
May  1 02:29:16 nvidia-desktop kernel: [34406.334574] pc : [<ffffff80080dc4fc>] lr : [<ffffff8008b164d8>] pstate: 00400045
May  1 02:29:16 nvidia-desktop kernel: [34406.341832] sp : ffffffc33b1d7a50
May  1 02:29:16 nvidia-desktop kernel: [34406.345505] x29: ffffffc33b1d7a50 x28: ffffffc17c024b80 
May  1 02:29:16 nvidia-desktop kernel: [34406.351026] x27: 0000000000000001 x26: ffffffc33b1d7d10 
May  1 02:29:16 nvidia-desktop kernel: [34406.356799] x25: ffffff8009f903a8 x24: 0000000000000000 
May  1 02:29:16 nvidia-desktop kernel: [34406.362395] x23: ffffffc3da5bc048 x22: ffffffc3d90d8518 
May  1 02:29:16 nvidia-desktop kernel: [34406.367759] x21: 000000000000002c x20: ffffffc3d90d8a28 
May  1 02:29:16 nvidia-desktop kernel: [34406.373260] x19: fffffffffffffffc x18: 0000000000000010 
May  1 02:29:16 nvidia-desktop kernel: [34406.379040] x17: 0000000000000020 x16: 0000000000000000 
May  1 02:29:16 nvidia-desktop kernel: [34406.384878] x15: 0000000000000006 x14: 70616320726f6620 
May  1 02:29:16 nvidia-desktop kernel: [34406.390566] x13: 6461657268746b20 x12: 6e7572206f742064 
May  1 02:29:16 nvidia-desktop kernel: [34406.395923] x11: 656c696166203a30 x10: 6f65646976207875 
May  1 02:29:16 nvidia-desktop kernel: [34406.401550] x9 : 000000000008d773 x8 : 0000000000000050 
May  1 02:29:16 nvidia-desktop kernel: [34406.407540] x7 : ffffff8009e740d8 x6 : ffffffc3ffdbabf0 
May  1 02:29:16 nvidia-desktop kernel: [34406.412808] x5 : ffffffc3ffdbabf0 x4 : 0000000000000000 
May  1 02:29:16 nvidia-desktop kernel: [34406.418147] x3 : ffffffc3ffdbe7f8 x2 : ffffffc3a15e7000 
May  1 02:29:16 nvidia-desktop kernel: [34406.423727] x1 : 000000000000002c x0 : 0000000000000001 
May  1 02:29:16 nvidia-desktop kernel: [34406.429065] 
May  1 02:29:16 nvidia-desktop kernel: [34406.430489] Process v4l2_single_cam (pid: 17714, stack limit = 0xffffffc33b1d4000)
May  1 02:29:16 nvidia-desktop kernel: [34406.437382] Call trace:
May  1 02:29:16 nvidia-desktop kernel: [34406.439764] [<ffffff80080dc4fc>] kthread_stop+0x2c/0x1e8
May  1 02:29:16 nvidia-desktop kernel: [34406.444852] [<ffffff8008b164d8>] vi5_channel_stop_kthreads+0x30/0x58
May  1 02:29:16 nvidia-desktop kernel: [34406.450964] [<ffffff8008b16644>] vi5_channel_start_streaming+0x94/0x3d8
May  1 02:29:16 nvidia-desktop kernel: [34406.457430] [<ffffff8008b0979c>] tegra_channel_start_streaming+0x3c/0x58
May  1 02:29:16 nvidia-desktop kernel: [34406.463566] [<ffffff8008b0196c>] vb2_start_streaming+0x6c/0x158
May  1 02:29:16 nvidia-desktop kernel: [34406.469436] [<ffffff8008b03650>] vb2_core_streamon+0x178/0x1a8
May  1 02:29:16 nvidia-desktop kernel: [34406.474756] [<ffffff8008b06214>] vb2_streamon+0x54/0x78
May  1 02:29:16 nvidia-desktop kernel: [34406.479565] [<ffffff8008b06290>] vb2_ioctl_streamon+0x58/0x68
May  1 02:29:16 nvidia-desktop kernel: [34406.484912] [<ffffff8008ae0e3c>] v4l_streamon+0x3c/0x50
May  1 02:29:16 nvidia-desktop kernel: [34406.489980] [<ffffff8008ae5c74>] __video_do_ioctl+0x1bc/0x288
May  1 02:29:16 nvidia-desktop kernel: [34406.495407] [<ffffff8008ae568c>] video_usercopy+0x2cc/0x6a8
May  1 02:29:16 nvidia-desktop kernel: [34406.501176] [<ffffff8008ae5aa4>] video_ioctl2+0x3c/0x50
May  1 02:29:16 nvidia-desktop kernel: [34406.505990] [<ffffff8008adf938>] v4l2_ioctl+0xf8/0x128
May  1 02:29:16 nvidia-desktop kernel: [34406.511071] [<ffffff800826c760>] do_vfs_ioctl+0xb0/0x8f8
May  1 02:29:16 nvidia-desktop kernel: [34406.516659] [<ffffff800826d03c>] SyS_ioctl+0x94/0xa8
May  1 02:29:16 nvidia-desktop kernel: [34406.521260] [<ffffff80080838c0>] el0_svc_naked+0x34/0x38
May  1 02:29:16 nvidia-desktop kernel: [34406.526743] ---[ end trace 7ee4054493ba2ad8 ]---

And I got another error on a different sensor at a different time:

Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.395844] channel context at 1 is busy
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.395894] ------------[ cut here ]------------
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396008] WARNING: CPU: 3 PID: 17717 at /home/bacchus/bacchus/public_sources/Linux_source_32.1/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:224 tegra_capture_ivc_notify_chan_id+0x190/0x1a8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396301] Modules linked in:
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396304] 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396312] CPU: 3 PID: 17717 Comm: v4l2_single_cam Not tainted 4.9.140+ #22
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396313] Hardware name: jetson-xavier (DT)
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396316] task: ffffffc3a15e5400 task.stack: ffffffc33b210000
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396320] PC is at tegra_capture_ivc_notify_chan_id+0x190/0x1a8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396323] LR is at tegra_capture_ivc_notify_chan_id+0x190/0x1a8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396325] pc : [<ffffff8008c618a0>] lr : [<ffffff8008c618a0>] pstate: 60400045
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396326] sp : ffffffc33b213890
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396347] x29: ffffffc33b213890 x28: ffffffc3b64c0680 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396351] x27: 0000000000000001 x26: ffffffc33b213d10 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396354] x25: ffffffc33b213928 x24: ffffffc3dabea020 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396356] x23: 0000000000000001 x22: ffffffc3dabeae78 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396359] x21: 0000000000000073 x20: 0000000000000e60 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396362] x19: ffffffc3dabea018 x18: 0000000000000000 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396364] x17: 0000000000000001 x16: ffffff8008f39000 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396366] x15: ffffffffffffffff x14: ffffff800a105240 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396369] x13: ffffff800a104e74 x12: ffffffffffffffff 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396371] x11: 0000000000000040 x10: 0000000000000a20 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396388] x9 : 0000000000006ead x8 : 20747865746e6f63 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396391] x7 : 206c656e6e616863 x6 : ffffff800a10413c 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396394] x5 : 00ffffffffffffff x4 : 0000000000000006 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396397] x3 : 0000000000010000 x2 : 0000000000010000 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396399] x1 : ffffffc3a15e5400 x0 : 000000000000001c 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396402] 
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396403] ---[ end trace 7ee4054493ba2ad5 ]---
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396494] Call trace:
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396501] [<ffffff8008c618a0>] tegra_capture_ivc_notify_chan_id+0x190/0x1a8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396508] [<ffffff8008b14b1c>] vi_capture_setup+0x3e4/0x4f8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396511] [<ffffff8008b16330>] tegra_channel_capture_setup+0xe8/0x208
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396521] [<ffffff8008b1681c>] vi5_channel_start_streaming+0x26c/0x3d8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396525] [<ffffff8008b0979c>] tegra_channel_start_streaming+0x3c/0x58
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396529] [<ffffff8008b0196c>] vb2_start_streaming+0x6c/0x158
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396551] [<ffffff8008b03650>] vb2_core_streamon+0x178/0x1a8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396554] [<ffffff8008b06214>] vb2_streamon+0x54/0x78
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396557] [<ffffff8008b06290>] vb2_ioctl_streamon+0x58/0x68
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396561] [<ffffff8008ae0e3c>] v4l_streamon+0x3c/0x50
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396565] [<ffffff8008ae5c74>] __video_do_ioctl+0x1bc/0x288
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396568] [<ffffff8008ae568c>] video_usercopy+0x2cc/0x6a8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396571] [<ffffff8008ae5aa4>] video_ioctl2+0x3c/0x50
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396573] [<ffffff8008adf938>] v4l2_ioctl+0xf8/0x128
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396578] [<ffffff800826c760>] do_vfs_ioctl+0xb0/0x8f8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396581] [<ffffff800826d03c>] SyS_ioctl+0x94/0xa8
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396585] [<ffffff80080838c0>] el0_svc_naked+0x34/0x38
Apr 30 17:19:05 nvidia-desktop kernel: [ 1395.396596] tegra194-vi5 15c10000.vi: failed to update control callback

Do you have an idea how I could avoid this issues and what is the cause of it ?

Thanks in advance,

JerryChang · May 9, 2019, 8:20am

hello Mobilaz,

there’s NULL pointer access and cause the kernel crash.
are you able to share the test binary and we could reproduce the failure from our side.
thanks

Mobilaz · May 9, 2019, 2:40pm

Hello,

Our test binary needs four ov24A1B sensors connected to the camera connector. Without these sensors, the test binary won’t start and you will not be able to reproduce the issue.

I tried to replicate the issue with the ov5693 sensor (the sensor officially supported) but the problem seems to only appear when multiple sensors are running but we do not have a multi sensor setup with the ov5693 sensor to try to replicate the issue.

Do you have an idea why a null pointer could appear after more than 1 hour ?

Kind regards,

JerryChang · May 10, 2019, 3:11am

hello Mobilaz,

suspect the callback function is not being called.
could you please have a quick try to add 10ms delay after wait_for_completion_timeout(&capture->control_resp…) function call.
for example,

+++ b/drivers/media/platform/tegra/camera/vi/capture.c

@@ -236,6 +239,7 @@ static int vi_capture_ivc_send_control(struct tegra_vi_channel *chan,
                goto fail;
        }

+       msleep_range(10);
        if (memcmp(&resp_header, &capture->control_resp_msg.header,
                        sizeof(resp_header)) != 0) {
                dev_err(chan->dev,

Mobilaz · May 13, 2019, 6:34am

Hello,

I added the delay in the capture.c file but the platform crashed after 3 hours.

I can see many discarded frames in the log and in my software I am seeing some timeouts on the select call used to get notifications from the driver to know when an image is ready.

Jan 28 19:50:29 nvidia-desktop kernel: [10343.464723] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 112, err_data 6292069
Jan 28 19:50:30 nvidia-desktop kernel: [10344.664327] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 160, err_data 1189
Jan 28 19:50:30 nvidia-desktop kernel: [10344.665024] tegra194-vi5 15c10000.vi: corr_err: discarding frame 10, flags: 0, err_data 64
Jan 28 19:50:30 nvidia-desktop kernel: [10344.864857] tegra194-vi5 15c10000.vi: corr_err: discarding frame 14, flags: 0, err_data 256
Jan 28 19:50:31 nvidia-desktop kernel: [10345.463803] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 32, err_data 165
Jan 28 19:50:32 nvidia-desktop kernel: [10346.263316] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 160, err_data 165
Jan 28 19:50:32 nvidia-desktop kernel: [10346.264121] tegra194-vi5 15c10000.vi: corr_err: discarding frame 18, flags: 0, err_data 64
Jan 28 19:50:32 nvidia-desktop kernel: [10346.463945] tegra194-vi5 15c10000.vi: corr_err: discarding frame 22, flags: 32, err_data 512
Jan 28 19:50:32 nvidia-desktop kernel: [10346.663264] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 112, err_data 6293093
Jan 28 19:50:33 nvidia-desktop kernel: [10347.862705] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 32, err_data 165
Jan 28 19:50:33 nvidia-desktop kernel: [10347.863343] tegra194-vi5 15c10000.vi: corr_err: discarding frame 26, flags: 128, err_data 512
Jan 28 19:50:34 nvidia-desktop kernel: [10348.262979] tegra194-vi5 15c10000.vi: corr_err: discarding frame 30, flags: 32, err_data 165
Jan 28 19:50:34 nvidia-desktop kernel: [10348.463512] tegra194-vi5 15c10000.vi: corr_err: discarding frame 32, flags: 0, err_data 512
Jan 28 19:50:34 nvidia-desktop kernel: [10348.662425] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 32, err_data 165
Jan 28 19:50:35 nvidia-desktop kernel: [10349.462004] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 32, err_data 165
Jan 28 19:50:35 nvidia-desktop kernel: [10349.463064] tegra194-vi5 15c10000.vi: corr_err: discarding frame 38, flags: 0, err_data 512
Jan 28 19:50:35 nvidia-desktop kernel: [10349.752505] CPU:0, Error:RCE-NOC@0xbe00000,irq=491
Jan 28 19:50:35 nvidia-desktop kernel: [10349.752894] **************************************
Jan 28 19:50:35 nvidia-desktop kernel: [10349.753164] * For more Internal Decode Help
Jan 28 19:50:35 nvidia-desktop kernel: [10349.753421] *     http://nv/cbberr
Jan 28 19:50:35 nvidia-desktop kernel: [10349.753638] * NVIDIA userID is required to access
Jan 28 19:50:35 nvidia-desktop kernel: [10349.753881] **************************************
Jan 28 19:50:35 nvidia-desktop kernel: [10349.754127] CPU:0, Error:RCE-NOC
Jan 28 19:50:35 nvidia-desktop kernel: [10349.754336]   Error Logger            : 1
Jan 28 19:50:35 nvidia-desktop kernel: [10349.754662]   ErrLog0                 : 0x80030600
Jan 28 19:50:35 nvidia-desktop kernel: [10349.754903]     Transaction Type      : RD  - Read, Incrementing
Jan 28 19:50:35 nvidia-desktop kernel: [10349.755189]     Error Code            : TMO
Jan 28 19:50:35 nvidia-desktop kernel: [10349.755415]     Error Source          : Target NIU
Jan 28 19:50:35 nvidia-desktop kernel: [10349.755703]     Error Description     : Target time-out error
Jan 28 19:50:35 nvidia-desktop kernel: [10349.755998]     Packet header Lock    : 0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.756255]     Packet header Len1    : 3
Jan 28 19:50:35 nvidia-desktop kernel: [10349.756466]     NOC protocol version  : version >= 2.7
Jan 28 19:50:35 nvidia-desktop kernel: [10349.756760]   ErrLog1                 : 0x157600
Jan 28 19:50:35 nvidia-desktop kernel: [10349.756958]   ErrLog2                 : 0x0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.757140]     RouteId               : 0x157600
Jan 28 19:50:35 nvidia-desktop kernel: [10349.757338]     InitFlow              : cpu_p_i/I/0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.757856]     Targflow              : cbb_t/T/0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.758100]     TargSubRange          : 27
Jan 28 19:50:35 nvidia-desktop kernel: [10349.758294]     SeqId                 : 0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.758475]   ErrLog3                 : 0x5c00414
Jan 28 19:50:35 nvidia-desktop kernel: [10349.758659]   ErrLog4                 : 0x0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.758981]     Address               : 0x15c00414 -- /host1x/slvs-ec@15ac0000 + 0x414
Jan 28 19:50:35 nvidia-desktop kernel: [10349.764574]   ErrLog5                 : 0x387e31
Jan 28 19:50:35 nvidia-desktop kernel: [10349.768239]     Master ID             : RCE
Jan 28 19:50:35 nvidia-desktop kernel: [10349.771389]     Security Group(GRPSEC): 0x3f
Jan 28 19:50:35 nvidia-desktop kernel: [10349.775415]     Cache                 : 0x1 -- Device
Jan 28 19:50:35 nvidia-desktop kernel: [10349.778925]     Protection            : 0x3 -- Privileged, Non-Secure, Data Access
Jan 28 19:50:35 nvidia-desktop kernel: [10349.785569]     FALCONSEC             : 0x0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.788717]     Virtual Queuing Channel(VQC): 0x0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.793446]   **************************************
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715] ###################### RTCPU EXCEPTION ######################
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715] [RCE] Data abort
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   R0:  0bca2c28 R1:  15c00000 R2:  00000000 R3:  0bc89c2c
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   R4:  0bca19b8 R5:  00000228 R6:  0bca1790 R7:  6845b39a
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   R8:  0bca0c88 R9:  0bca2c28 R10: 10101010 R11: 11111111
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   R12: 00000000 SP:  0bcdf800 LR:  0bc89c50 PC:  0bc80308
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   SP: 0bcdff98 LR: 0bc8149c
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   CPSR: 20000197 SPSR: 20000192
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   DFSR: 00001008 DFAR: 15c00414 ADFSR: 00000000
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   IFSR: 00000000 IFAR: 00000000 AIFSR: 00000000
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715] Callstack
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   [0bcdff7c]: 0bc84680
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   [0bcdffa4]: 0bc89c2c
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   [0bcdffc4]: 0bc8149c
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   [0bcdffdc]: 0bc886f8
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   [0bcdffe4]: 0bc88468
Jan 28 19:50:35 nvidia-desktop kernel: [10349.799715]   [0bcdfffc]: 0bc818cc
Jan 28 19:50:35 nvidia-desktop kernel: [10349.826446] CPU:0, Error:CBB-NOC@0x2300000,irq=485
Jan 28 19:50:35 nvidia-desktop kernel: [10349.826647] **************************************
Jan 28 19:50:35 nvidia-desktop kernel: [10349.826802] * For more Internal Decode Help
Jan 28 19:50:35 nvidia-desktop kernel: [10349.826937] *     http://nv/cbberr
Jan 28 19:50:35 nvidia-desktop kernel: [10349.827166] * NVIDIA userID is required to access
Jan 28 19:50:35 nvidia-desktop kernel: [10349.827314] **************************************
Jan 28 19:50:35 nvidia-desktop kernel: [10349.827514] CPU:0, Error:CBB-NOC
Jan 28 19:50:35 nvidia-desktop kernel: [10349.828508]   Error Logger            : 1
Jan 28 19:50:35 nvidia-desktop kernel: [10349.831582]   ErrLog0                 : 0x80030600
Jan 28 19:50:35 nvidia-desktop kernel: [10349.835146]     Transaction Type      : RD  - Read, Incrementing
Jan 28 19:50:35 nvidia-desktop kernel: [10349.840914]     Error Code            : TMO
Jan 28 19:50:35 nvidia-desktop kernel: [10349.844081]     Error Source          : Target NIU
Jan 28 19:50:35 nvidia-desktop kernel: [10349.848093]     Error Description     : Target time-out error
Jan 28 19:50:35 nvidia-desktop kernel: [10349.853688]     Packet header Lock    : 0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.857362]     Packet header Len1    : 3
Jan 28 19:50:35 nvidia-desktop kernel: [10349.860951]     NOC protocol version  : version >= 2.7
Jan 28 19:50:35 nvidia-desktop kernel: [10349.865593]   ErrLog1                 : 0x9528aa
Jan 28 19:50:35 nvidia-desktop kernel: [10349.869261]   ErrLog2                 : 0x0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.871718]     RouteId               : 0x9528aa
Jan 28 19:50:35 nvidia-desktop kernel: [10349.875214]     InitFlow              : rce_p2ps/I/rce_p2ps
Jan 28 19:50:35 nvidia-desktop kernel: [10349.879418]     Targflow              : host1x_p2pm/T/host1x_p2pm
Jan 28 19:50:35 nvidia-desktop kernel: [10349.884663]     TargSubRange          : 20
Jan 28 19:50:35 nvidia-desktop kernel: [10349.887824]     SeqId                 : 0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.890969]   ErrLog3                 : 0x414
Jan 28 19:50:35 nvidia-desktop kernel: [10349.893938]   ErrLog4                 : 0x0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.896552]     Address               : 0x15c00414 -- /host1x/slvs-ec@15ac0000 + 0x414
Jan 28 19:50:35 nvidia-desktop kernel: [10349.902961]   ErrLog5                 : 0x2af0fc71
Jan 28 19:50:35 nvidia-desktop kernel: [10349.906631]     Non-Modify            : 0x1
Jan 28 19:50:35 nvidia-desktop kernel: [10349.910127]     AXI ID                : 0x55
Jan 28 19:50:35 nvidia-desktop kernel: [10349.913279]     Master ID             : RCE
Jan 28 19:50:35 nvidia-desktop kernel: [10349.916251]     Security Group(GRPSEC): 0x3f
Jan 28 19:50:35 nvidia-desktop kernel: [10349.920276]     Cache                 : 0x1 -- Device
Jan 28 19:50:35 nvidia-desktop kernel: [10349.923960]     Protection            : 0x3 -- Privileged, Non-Secure, Data Access
Jan 28 19:50:35 nvidia-desktop kernel: [10349.930779]     FALCONSEC             : 0x0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.933930]     Virtual Queuing Channel(VQC): 0x0
Jan 28 19:50:35 nvidia-desktop kernel: [10349.938570]   **************************************
Jan 28 19:50:36 nvidia-desktop kernel: [10350.087375] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!

JerryChang · May 13, 2019, 7:06am

hello Mobilaz,

may I have your confirmation that all your 4-lane OV24A1B sensors were all slvs-ec sensors
your failure seems related to I/O access timing, could you please review the sensor driver of the power-on/off sequence.
are you able to arrange hardware resource to investigate the hardware signaling?

Mobilaz · May 14, 2019, 6:54am

Hello,

The sensors are not slvs-ec (Scalable Low Voltage Signalling Embedded Clock) compliant. What can go wrong if it is not the case?
We are checking it again but it seems that we don’t have any issue with the power-on/off sequence
We probed most of the signals and we couldn’t highlight any big issues in the signaling. We managed to let a single sensor running for more than 24 hours without any issue giving us confidence in the signaling.

We also tried a new test:

start sensor 0
capture with sensor 0
stop and close sensor 0
repeat from 1 with the next sensor over the 4 sensors

This test ran without any issue over 24 hours. The issues seem to appear when multiple sensors are ran in parallel while being shut on/off repeatedly.

Thanks in advance,

JerryChang · May 14, 2019, 8:03am

hello Mobilaz,

FYI,
we had also see the stability issue when access four camera sensors simultaneously.
here’s command we used to access 4 camera sensor together.

$ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video0 &
v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video1 &
v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video2 &
v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video3 &

Mobilaz · May 23, 2019, 5:50am

Hello,

Were you able to isolate the issue ? We are still facing the same stability issue and we do not know what test we could perform to gather more information around the root cause ?

Do you need us to perform more tests ? If yes, could you describe them to me ?

Thanks in advance,

keith.wang · August 16, 2019, 2:05am

hi all:
Is this problem solved? We also encountered the same problem.

JerryChang · August 16, 2019, 2:49am

hello keith.wang,

we had fix the known issue of multiple cameras stream-off process.
this kernel fix did not include in JetPack-4.2.1 (l4t-r32.2) yet, please apply this patch (devtalk1050432_Aug16.tar.gz) manually for verification.
thanks
devtalk1050432_Aug16.tar.gz (2.29 KB)

keith.wang · August 16, 2019, 3:04am

hi JerryChang:
thank you