CSIMUX_STREAM video capture failure

I’m noticing an intermittent error with video capture that only seems to go away when I reinitialize my camera devices. About 80% of the time when I capture, it is successful. When it is successful, subsequent captures do not have any issues and I’m able to confirm my video input is what I expect it to be. However, when it fails, it’s a hard fail and I need to reinitialize my device in order to make another capture attempt. In the times that it fails, the errors in the kernel debug log include problems relating to CSIMUX_STREAM:

 kworker/1:5-5652  [001] ....   452.721844: rtcpu_vinotify_error: tstamp:14282534727 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:14282533779 data:0x00000489
 kworker/1:5-5652  [001] ....   452.721849: rtcpu_vinotify_error: tstamp:14282535120 tag:CSIMUX_STREAM channel:0x00 frame:0 vi_tstamp:14282534038 data:0x00010000
 kworker/1:5-5652  [001] ....   452.823843: rtcpu_vinotify_error: tstamp:14285784325 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:14285783620 data:0x00000489

The error I see reported to the console and dmesg when this happens is:

[  410.828750] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 0, err_data 64
[  413.386679] tegra194-vi5 15c10000.vi: no reply from camera processor
[  413.386853] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[  413.387030] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[  413.391772] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel

I do not normally see the CSIMUX_STREAM error appear otherwise, so I’m inclined to think it is related to the problem I’m encountering.

Any ideas what might be wrong? What does the “data:0x00010000” parameter represent?

Here’s my setup info:

  • 2x AR0233-based cameras, each equipped with a Maxim 9295A GMSL2 serializer
  • 1x Maxim 9296A GMSL2 Deserializer
  • AR0233 sensors output RGB888 @ 1920x1080
  • vi/isp/nvcsi rates are set to their max_rate. (See when get raw16 data through vc-id, corr_err: discarding frame 0, flags: 32, err_data 160)
  • Tracing is enabled. (Reference same link as above)
  • I’m using the following command to capture data frames: v4l2-ctl -d /dev/video2 --set-fmt-video=width=1920,height=1080,pixelformat=RGB3 --stream-mmap --stream-count=1 --stream-to=ar0233-0.raw

Thanks!
–tim

Check the wiki its’ SPURIOUS_DATA_STREAM_4.
Spurious data means VI sees some other packets before FS packet, as VI alwasy expect the 1st packet to be frame start

https://elinux.org/Jetson_TX2_Camera_BringUp

For reference, here are traces of a “good” capture and “bad” capture when I encountered the CSIMUX_STREAM failure. I’m still digging into what is happening in each instance, but wanted to post them in case you have any insight.

Good trace:

# tracer: nop
#
# entries-in-buffer/entries-written: 164/164   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
        v4l2-ctl-9650  [001] ....  1557.164775: tegra_channel_open: vi-output, jbar0233 2-0010
        v4l2-ctl-9650  [000] ....  1557.167619: tegra_channel_set_power: jbar0233 2-0010 : 0x1
        v4l2-ctl-9650  [000] ....  1557.167731: camera_common_s_power: status : 0x1
        v4l2-ctl-9650  [000] ....  1557.168599: tegra_channel_set_power: 15a00000.nvcsi--6 : 0x1
        v4l2-ctl-9650  [000] ....  1557.168607: csi_s_power: enable : 0x1
        v4l2-ctl-9650  [001] ....  1557.191974: tegra_channel_set_stream: enable : 0x1
        v4l2-ctl-9650  [001] ....  1557.201372: tegra_channel_set_stream: 15a00000.nvcsi--6 : 0x1
        v4l2-ctl-9650  [001] ....  1557.201421: csi_s_stream: enable : 0x1
        v4l2-ctl-9650  [001] ....  1557.201445: tegra_channel_set_stream: jbar0233 2-0010 : 0x1
        v4l2-ctl-9650  [001] ....  1557.201733: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt a
     kworker/0:1-8792  [000] ....  1557.233710: rtos_queue_send_from_isr_failed: tstamp:48799320344 queue:0x0bcb41f8
     kworker/0:1-8792  [000] ....  1557.233715: rtos_queue_send_from_isr_failed: tstamp:48799322851 queue:0x0bcb8a60
     kworker/0:1-8792  [000] ....  1557.233716: rtos_queue_send_from_isr_failed: tstamp:48799325344 queue:0x0bcba5e0
     kworker/0:1-8792  [000] ....  1557.233718: rtos_queue_send_from_isr_failed: tstamp:48799327805 queue:0x0bcbb3a0
     kworker/0:1-8792  [000] ....  1557.233719: rtos_queue_send_from_isr_failed: tstamp:48799330260 queue:0x0bcbc160
     kworker/0:1-8792  [000] ....  1557.233724: rtcpu_nvcsi_intr: tstamp:48799354680 class:GLOBAL type:PHY_INTR0 phy:2 cil:0 st:0 vc:0 status:0x00000001
     kworker/0:1-8792  [000] ....  1557.233762: rtos_queue_peek_from_isr_failed: tstamp:48799420605 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1557.233766: rtcpu_start: tstamp:48799422399
     kworker/0:1-8792  [000] ....  1557.233768: rtos_queue_send_from_isr_failed: tstamp:48799433689 queue:0x0bcb41f8
     kworker/0:1-8792  [000] ....  1557.233769: rtos_queue_send_from_isr_failed: tstamp:48799433850 queue:0x0bcb8a60
     kworker/0:1-8792  [000] ....  1557.233770: rtos_queue_send_from_isr_failed: tstamp:48799434011 queue:0x0bcba5e0
     kworker/0:1-8792  [000] ....  1557.233772: rtos_queue_send_from_isr_failed: tstamp:48799434168 queue:0x0bcbb3a0
     kworker/0:1-8792  [000] ....  1557.233773: rtos_queue_send_from_isr_failed: tstamp:48799434323 queue:0x0bcbc160
     kworker/0:1-8792  [000] ....  1557.233776: rtcpu_string: tstamp:48799434834 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:1-8792  [000] ....  1557.233791: rtcpu_string: tstamp:48799435028 id:0x04010000 str:"VM GOS[#0] addr=0xe4900000
"
     kworker/0:1-8792  [000] ....  1557.233796: rtcpu_string: tstamp:48799435293 id:0x04010000 str:"VM GOS[#1] addr=0xe4901000
"
     kworker/0:1-8792  [000] ....  1557.233802: rtcpu_string: tstamp:48799435573 id:0x04010000 str:"VM GOS[#2] addr=0xe4902000
"
     kworker/0:1-8792  [000] ....  1557.233806: rtcpu_string: tstamp:48799435816 id:0x04010000 str:"VM GOS[#3] addr=0xe4903000
"
     kworker/0:1-8792  [000] ....  1557.233810: rtcpu_string: tstamp:48799436057 id:0x04010000 str:"VM GOS[#4] addr=0xe4904000
"
     kworker/0:1-8792  [000] ....  1557.233813: rtcpu_string: tstamp:48799436298 id:0x04010000 str:"VM GOS[#5] addr=0xe4905000
"
     kworker/0:1-8792  [000] ....  1557.233817: rtcpu_string: tstamp:48799443994 id:0x04010000 str:"vi5_hwinit: firmware CL2018101701 protocol versi"
     kworker/0:1-8792  [000] ....  1557.233820: rtcpu_string: tstamp:48799444121 id:0x04010000 str:"on 2.2
"
     kworker/0:1-8792  [000] ....  1557.233824: rtos_queue_send_from_isr_failed: tstamp:48799462578 queue:0x0bcb41f8
     kworker/0:1-8792  [000] ....  1557.233825: rtos_queue_send_from_isr_failed: tstamp:48799462758 queue:0x0bcb8a60
     kworker/0:1-8792  [000] ....  1557.233827: rtos_queue_send_from_isr_failed: tstamp:48799462923 queue:0x0bcba5e0
     kworker/0:1-8792  [000] ....  1557.233828: rtos_queue_send_from_isr_failed: tstamp:48799463079 queue:0x0bcbb3a0
     kworker/0:1-8792  [000] ....  1557.233867: rtos_queue_send_from_isr_failed: tstamp:48799463234 queue:0x0bcbc160
     kworker/0:1-8792  [000] ....  1557.233869: rtcpu_string: tstamp:48799464125 id:0x04010000 str:"VI GOS[#0] set to VM GOS[4] base 0xe4904000
"
     kworker/0:1-8792  [000] ....  1557.233877: rtcpu_vinotify_event: tstamp:48799787669 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49754394368 data:0x10000000
     kworker/0:1-8792  [000] ....  1557.233879: rtcpu_vinotify_event: tstamp:48799787824 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49754398624 data:0x31000001
     kworker/0:1-8792  [000] ....  1557.233880: rtcpu_vinotify_event: tstamp:48799788018 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49754399712 data:0x07020001
     kworker/0:1-8792  [000] ....  1557.233882: rtcpu_vinotify_event: tstamp:48799788164 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49754439776 data:0x10000000
     kworker/0:1-8792  [000] ....  1557.233883: rtcpu_vinotify_event: tstamp:48799788352 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49754444064 data:0x31000002
     kworker/0:1-8792  [000] ....  1557.335682: rtcpu_vinotify_error: tstamp:48803281673 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48803280963 data:0x00000489
     kworker/0:1-8792  [000] ....  1557.386696: rtcpu_vinotify_event: tstamp:48803570099 tag:FS channel:0x01 frame:0 vi_tstamp:48803280273 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.386699: rtcpu_vinotify_event: tstamp:48803570293 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48803280963 data:0x00000489
     kworker/0:1-8792  [000] ....  1557.386701: rtcpu_vinotify_event: tstamp:48803570441 tag:FS channel:0x00 frame:0 vi_tstamp:48803318477 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.386702: rtcpu_vinotify_event: tstamp:48803570612 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:48803318477 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.386704: rtcpu_vinotify_event: tstamp:48803570760 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:48803319165 data:0x00000001
     kworker/0:1-8792  [000] ....  1557.386743: rtcpu_vinotify_event: tstamp:48803570952 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49877725664 data:0x08020001
     kworker/0:1-8792  [000] ....  1557.386746: rtcpu_vinotify_event: tstamp:48804024316 tag:FE channel:0x01 frame:0 vi_tstamp:48804022873 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.386748: rtcpu_vinotify_event: tstamp:48804024488 tag:FE channel:0x00 frame:0 vi_tstamp:48804023229 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.386749: rtcpu_vinotify_event: tstamp:48804024640 tag:CHANSEL_SHORT_FRAME channel:0x10 frame:0 vi_tstamp:48804023229 data:0x01000000
     kworker/0:1-8792  [000] ....  1557.386750: rtcpu_vinotify_event: tstamp:48804024851 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49900257408 data:0x01020001
     kworker/0:1-8792  [000] ....  1557.386752: rtcpu_vinotify_event: tstamp:48804025003 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:48804023229 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.386756: rtos_queue_send_from_isr_failed: tstamp:48804037113 queue:0x0bcb41f8
     kworker/0:1-8792  [000] ....  1557.386758: rtos_queue_send_from_isr_failed: tstamp:48804037276 queue:0x0bcb8a60
     kworker/0:1-8792  [000] ....  1557.386759: rtos_queue_send_from_isr_failed: tstamp:48804037441 queue:0x0bcba5e0
     kworker/0:1-8792  [000] ....  1557.386761: rtos_queue_send_from_isr_failed: tstamp:48804037599 queue:0x0bcbb3a0
     kworker/0:1-8792  [000] ....  1557.386788: rtos_queue_send_from_isr_failed: tstamp:48804037753 queue:0x0bcbc160
     kworker/0:1-8792  [000] ....  1557.386790: rtcpu_vinotify_event: tstamp:48804361073 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49900262656 data:0x07020002
     kworker/0:1-8792  [000] ....  1557.386791: rtcpu_vinotify_event: tstamp:48804361245 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49900356288 data:0x10000000
     kworker/0:1-8792  [000] ....  1557.386792: rtcpu_vinotify_event: tstamp:48804361436 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49900360576 data:0x31000003
     kworker/0:1-8792  [000] ....  1557.386796: rtos_queue_peek_from_isr_failed: tstamp:48804421008 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1557.437755: rtcpu_vinotify_error: tstamp:48806531411 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48806530116 data:0x00000489
 vi-output, jbar-9652  [003] ....  1557.456210: tegra_channel_capture_frame: sof:1561.808975168
 vi-output, jbar-9652  [003] ....  1557.456215: tegra_channel_capture_frame: eof:1561.832738240
     kworker/0:1-8792  [000] ....  1557.488718: rtcpu_vinotify_event: tstamp:48806767989 tag:FS channel:0x01 frame:0 vi_tstamp:48806500650 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.488724: rtcpu_vinotify_event: tstamp:48806768163 tag:FS channel:0x00 frame:0 vi_tstamp:48806500999 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.488725: rtcpu_vinotify_event: tstamp:48806768316 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:48806500999 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.488727: rtcpu_vinotify_event: tstamp:48806768504 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48806530116 data:0x00000489
     kworker/0:1-8792  [000] ....  1557.488728: rtcpu_vinotify_event: tstamp:48806768650 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:48806530474 data:0x00000001
     kworker/0:1-8792  [000] ....  1557.488729: rtcpu_vinotify_event: tstamp:48806768820 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49980487552 data:0x08020002
     kworker/0:1-8792  [000] ....  1557.488731: rtcpu_vinotify_event: tstamp:48807274134 tag:FE channel:0x01 frame:0 vi_tstamp:48807272712 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.488761: rtcpu_vinotify_event: tstamp:48807274308 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:48807273049 data:0x04370002
     kworker/0:1-8792  [000] ....  1557.488762: rtcpu_vinotify_event: tstamp:48807274457 tag:FE channel:0x00 frame:0 vi_tstamp:48807273067 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.488763: rtcpu_vinotify_event: tstamp:48807274628 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:48807273068 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.488765: rtcpu_vinotify_event: tstamp:48807274776 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:48807273070 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.488766: rtcpu_vinotify_event: tstamp:48807274982 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:50004252224 data:0x02020002
     kworker/0:1-8792  [000] ....  1557.488767: rtcpu_vinotify_event: tstamp:48807275127 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:50004253760 data:0x00020002
     kworker/0:1-8792  [000] ....  1557.488776: rtcpu_vinotify_event: tstamp:48807593346 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:50004257664 data:0x07020003
     kworker/0:1-8792  [000] ....  1557.539696: rtos_queue_send_from_isr_failed: tstamp:48808488497 queue:0x0bcb41f8
     kworker/0:1-8792  [000] ....  1557.539701: rtos_queue_send_from_isr_failed: tstamp:48808488667 queue:0x0bcb8a60
     kworker/0:1-8792  [000] ....  1557.539703: rtos_queue_send_from_isr_failed: tstamp:48808488832 queue:0x0bcba5e0
     kworker/0:1-8792  [000] ....  1557.539704: rtos_queue_send_from_isr_failed: tstamp:48808488990 queue:0x0bcbb3a0
     kworker/0:1-8792  [000] ....  1557.539706: rtos_queue_send_from_isr_failed: tstamp:48808489147 queue:0x0bcbc160
     kworker/0:1-8792  [000] ....  1557.539709: rtos_queue_peek_from_isr_failed: tstamp:48809421013 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1557.539724: rtcpu_vinotify_error: tstamp:48809782306 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48809781022 data:0x00000489
 vi-output, jbar-9652  [003] ....  1557.560226: tegra_channel_capture_frame: sof:1561.912984416
 vi-output, jbar-9652  [003] ....  1557.560234: tegra_channel_capture_frame: eof:1561.936755744
        v4l2-ctl-9650  [003] ....  1557.560362: tegra_channel_set_stream: enable : 0x0
        v4l2-ctl-9650  [003] ....  1557.560364: tegra_channel_set_stream: jbar0233 2-0010 : 0x0
        v4l2-ctl-9650  [003] ....  1557.560470: tegra_channel_set_stream: 15a00000.nvcsi--6 : 0x0
        v4l2-ctl-9650  [003] ....  1557.560474: csi_s_stream: enable : 0x0
        v4l2-ctl-9650  [000] ....  1557.577071: tegra_channel_close: vi-output, jbar0233 2-0010
        v4l2-ctl-9650  [000] ....  1557.580549: tegra_channel_set_power: jbar0233 2-0010 : 0x0
        v4l2-ctl-9650  [000] ....  1557.580609: camera_common_s_power: status : 0x0
        v4l2-ctl-9650  [000] ....  1557.580865: tegra_channel_set_power: 15a00000.nvcsi--6 : 0x0
        v4l2-ctl-9650  [000] ....  1557.580871: csi_s_power: enable : 0x0
     kworker/0:1-8792  [000] ....  1557.590692: rtcpu_vinotify_event: tstamp:48810000366 tag:FS channel:0x00 frame:0 vi_tstamp:48809751288 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.590697: rtcpu_vinotify_event: tstamp:48810000571 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:48809751289 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.590699: rtcpu_vinotify_event: tstamp:48810000721 tag:FS channel:0x01 frame:0 vi_tstamp:48809751413 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.590701: rtcpu_vinotify_event: tstamp:48810000911 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:48809780763 data:0x00000001
     kworker/0:1-8792  [000] ....  1557.590702: rtcpu_vinotify_event: tstamp:48810001079 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:50084496800 data:0x08020003
     kworker/0:1-8792  [000] ....  1557.590704: rtcpu_vinotify_event: tstamp:48810001267 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48809781022 data:0x00000489
     kworker/0:1-8792  [000] ....  1557.590705: rtcpu_vinotify_event: tstamp:48810001412 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:50084580384 data:0x10000000
     kworker/0:1-8792  [000] ....  1557.590706: rtcpu_vinotify_event: tstamp:48810001602 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:50084584640 data:0x31000004
     kworker/0:1-8792  [000] ....  1557.590708: rtcpu_vinotify_event: tstamp:48810524628 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:48810523341 data:0x04370002
     kworker/0:1-8792  [000] ....  1557.590709: rtcpu_vinotify_event: tstamp:48810524803 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:48810523362 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.590710: rtcpu_vinotify_event: tstamp:48810524952 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:50108259968 data:0x02020003
     kworker/0:1-8792  [000] ....  1557.590712: rtcpu_vinotify_event: tstamp:48810525139 tag:FE channel:0x00 frame:0 vi_tstamp:48810523617 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.590713: rtcpu_vinotify_event: tstamp:48810525306 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:48810523617 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.590717: rtcpu_vinotify_event: tstamp:48810525494 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:50108269440 data:0x00020003
     kworker/0:1-8792  [000] ....  1557.590718: rtcpu_vinotify_event: tstamp:48810525641 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:50108272384 data:0x07020004
     kworker/0:1-8792  [000] ....  1557.590733: rtos_queue_send_from_isr_failed: tstamp:48810537949 queue:0x0bcb41f8
     kworker/0:1-8792  [000] ....  1557.590735: rtos_queue_send_from_isr_failed: tstamp:48810538115 queue:0x0bcb8a60
     kworker/0:1-8792  [000] ....  1557.590736: rtos_queue_send_from_isr_failed: tstamp:48810538278 queue:0x0bcba5e0
     kworker/0:1-8792  [000] ....  1557.590738: rtos_queue_send_from_isr_failed: tstamp:48810538437 queue:0x0bcbb3a0
     kworker/0:1-8792  [000] ....  1557.590740: rtos_queue_send_from_isr_failed: tstamp:48810538597 queue:0x0bcbc160
     kworker/0:1-8792  [000] ....  1557.590778: rtcpu_vinotify_event: tstamp:48810831805 tag:FE channel:0x01 frame:0 vi_tstamp:48810523635 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.590780: rtos_queue_send_from_isr_failed: tstamp:48810853841 queue:0x0bcb41f8
     kworker/0:1-8792  [000] ....  1557.590781: rtos_queue_send_from_isr_failed: tstamp:48810854064 queue:0x0bcb8a60
     kworker/0:1-8792  [000] ....  1557.590782: rtos_queue_send_from_isr_failed: tstamp:48810854287 queue:0x0bcba5e0
     kworker/0:1-8792  [000] ....  1557.590784: rtos_queue_send_from_isr_failed: tstamp:48810854506 queue:0x0bcbb3a0
     kworker/0:1-8792  [000] ....  1557.590785: rtos_queue_send_from_isr_failed: tstamp:48810854725 queue:0x0bcbc160
     kworker/0:1-8792  [000] ....  1557.692686: rtos_queue_peek_from_isr_failed: tstamp:48814421049 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1557.845689: rtos_queue_peek_from_isr_failed: tstamp:48819421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1558.049685: rtos_queue_peek_from_isr_failed: tstamp:48824421044 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1558.202698: rtos_queue_peek_from_isr_failed: tstamp:48829421044 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1558.355680: rtos_queue_peek_from_isr_failed: tstamp:48834421044 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1558.508696: rtos_queue_peek_from_isr_failed: tstamp:48839421080 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1558.661692: rtos_queue_peek_from_isr_failed: tstamp:48844421009 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1558.814686: rtos_queue_peek_from_isr_failed: tstamp:48849421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1558.967684: rtos_queue_peek_from_isr_failed: tstamp:48854421043 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1559.171689: rtos_queue_peek_from_isr_failed: tstamp:48859421045 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1559.324686: rtos_queue_peek_from_isr_failed: tstamp:48864421043 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1559.477698: rtos_queue_peek_from_isr_failed: tstamp:48869421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1559.630713: rtos_queue_peek_from_isr_failed: tstamp:48874421080 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1559.783683: rtos_queue_peek_from_isr_failed: tstamp:48879421083 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1559.936686: rtos_queue_peek_from_isr_failed: tstamp:48884421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1560.089683: rtos_queue_peek_from_isr_failed: tstamp:48889421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1560.293686: rtos_queue_peek_from_isr_failed: tstamp:48894421039 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1560.446684: rtos_queue_peek_from_isr_failed: tstamp:48899421044 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1560.599701: rtos_queue_peek_from_isr_failed: tstamp:48904421039 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1560.752684: rtos_queue_peek_from_isr_failed: tstamp:48909421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1560.905682: rtos_queue_peek_from_isr_failed: tstamp:48914421084 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1561.058732: rtos_queue_peek_from_isr_failed: tstamp:48919421009 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1561.211683: rtos_queue_peek_from_isr_failed: tstamp:48924421036 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1561.415800: rtos_queue_peek_from_isr_failed: tstamp:48929421044 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1561.568680: rtos_queue_peek_from_isr_failed: tstamp:48934421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1561.721721: rtos_queue_peek_from_isr_failed: tstamp:48939421049 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1561.874688: rtos_queue_peek_from_isr_failed: tstamp:48944421044 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1562.027679: rtos_queue_peek_from_isr_failed: tstamp:48949421080 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1562.180692: rtos_queue_peek_from_isr_failed: tstamp:48954421084 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1562.333714: rtos_queue_peek_from_isr_failed: tstamp:48959421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1562.486705: rtos_queue_peek_from_isr_failed: tstamp:48964421039 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1562.690712: rtos_queue_peek_from_isr_failed: tstamp:48969421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1562.843698: rtos_queue_peek_from_isr_failed: tstamp:48974421043 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1562.996700: rtos_queue_peek_from_isr_failed: tstamp:48979421040 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1563.149695: rtos_queue_peek_from_isr_failed: tstamp:48984421039 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1563.302681: rtos_queue_peek_from_isr_failed: tstamp:48989421082 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1563.455764: rtos_queue_peek_from_isr_failed: tstamp:48994420988 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1563.608715: rtos_queue_peek_from_isr_failed: tstamp:48999421040 queue:0x0bcbcf78
     kworker/0:1-8792  [000] ....  1563.659696: rtos_queue_peek_from_isr_failed: tstamp:48999764314 queue:0x0bcbcf78

Bad trace:

# tracer: nop
#
# entries-in-buffer/entries-written: 383/383   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
        v4l2-ctl-9610  [001] ....  1526.882367: tegra_channel_open: vi-output, jbar0233 2-0010
        v4l2-ctl-9610  [000] ....  1526.884519: tegra_channel_set_power: jbar0233 2-0010 : 0x1
        v4l2-ctl-9610  [000] ....  1526.884561: camera_common_s_power: status : 0x1
        v4l2-ctl-9610  [000] ....  1526.885059: tegra_channel_set_power: 15a00000.nvcsi--6 : 0x1
        v4l2-ctl-9610  [000] ....  1526.885066: csi_s_power: enable : 0x1
        v4l2-ctl-9610  [000] ....  1526.908880: tegra_channel_set_stream: enable : 0x1
        v4l2-ctl-9610  [000] ....  1526.916645: tegra_channel_set_stream: 15a00000.nvcsi--6 : 0x1
        v4l2-ctl-9610  [000] ....  1526.916654: csi_s_stream: enable : 0x1
        v4l2-ctl-9610  [000] ....  1526.916696: tegra_channel_set_stream: jbar0233 2-0010 : 0x1
        v4l2-ctl-9610  [000] ....  1526.916971: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt a
     kworker/0:2-9517  [000] ....  1526.952717: rtos_queue_send_from_isr_failed: tstamp:47852921972 queue:0x0bcb41f8
     kworker/0:2-9517  [000] ....  1526.952721: rtos_queue_send_from_isr_failed: tstamp:47852924511 queue:0x0bcb8a60
     kworker/0:2-9517  [000] ....  1526.952723: rtos_queue_send_from_isr_failed: tstamp:47852927031 queue:0x0bcba5e0
     kworker/0:2-9517  [000] ....  1526.952724: rtos_queue_send_from_isr_failed: tstamp:47852929492 queue:0x0bcbb3a0
     kworker/0:2-9517  [000] ....  1526.952726: rtos_queue_send_from_isr_failed: tstamp:47852931946 queue:0x0bcbc160
     kworker/0:2-9517  [000] ....  1526.952730: rtcpu_nvcsi_intr: tstamp:47852935221 class:GLOBAL type:PHY_INTR0 phy:2 cil:0 st:0 vc:0 status:0x00000001
     kworker/0:2-9517  [000] ....  1526.952766: rtos_queue_peek_from_isr_failed: tstamp:47853009763 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1526.952771: rtcpu_start: tstamp:47853012359
     kworker/0:2-9517  [000] ....  1526.952773: rtos_queue_send_from_isr_failed: tstamp:47853024925 queue:0x0bcb41f8
     kworker/0:2-9517  [000] ....  1526.952774: rtos_queue_send_from_isr_failed: tstamp:47853025085 queue:0x0bcb8a60
     kworker/0:2-9517  [000] ....  1526.952775: rtos_queue_send_from_isr_failed: tstamp:47853025247 queue:0x0bcba5e0
     kworker/0:2-9517  [000] ....  1526.952777: rtos_queue_send_from_isr_failed: tstamp:47853025406 queue:0x0bcbb3a0
     kworker/0:2-9517  [000] ....  1526.952778: rtos_queue_send_from_isr_failed: tstamp:47853025563 queue:0x0bcbc160
     kworker/0:2-9517  [000] ....  1526.952781: rtcpu_string: tstamp:47853026058 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:2-9517  [000] ....  1526.952797: rtcpu_string: tstamp:47853026250 id:0x04010000 str:"VM GOS[#0] addr=0xe4900000
"
     kworker/0:2-9517  [000] ....  1526.952830: rtcpu_string: tstamp:47853026516 id:0x04010000 str:"VM GOS[#1] addr=0xe4901000
"
     kworker/0:2-9517  [000] ....  1526.952835: rtcpu_string: tstamp:47853026817 id:0x04010000 str:"VM GOS[#2] addr=0xe4902000
"
     kworker/0:2-9517  [000] ....  1526.952839: rtcpu_string: tstamp:47853027058 id:0x04010000 str:"VM GOS[#3] addr=0xe4903000
"
     kworker/0:2-9517  [000] ....  1526.952844: rtcpu_string: tstamp:47853027301 id:0x04010000 str:"VM GOS[#4] addr=0xe4904000
"
     kworker/0:2-9517  [000] ....  1526.952848: rtcpu_string: tstamp:47853027543 id:0x04010000 str:"VM GOS[#5] addr=0xe4905000
"
     kworker/0:2-9517  [000] ....  1526.952852: rtcpu_string: tstamp:47853035261 id:0x04010000 str:"vi5_hwinit: firmware CL2018101701 protocol versi"
     kworker/0:2-9517  [000] ....  1526.952854: rtcpu_string: tstamp:47853035388 id:0x04010000 str:"on 2.2
"
     kworker/0:2-9517  [000] ....  1526.952858: rtos_queue_send_from_isr_failed: tstamp:47853055575 queue:0x0bcb41f8
     kworker/0:2-9517  [000] ....  1526.952860: rtos_queue_send_from_isr_failed: tstamp:47853055756 queue:0x0bcb8a60
     kworker/0:2-9517  [000] ....  1526.952861: rtos_queue_send_from_isr_failed: tstamp:47853055921 queue:0x0bcba5e0
     kworker/0:2-9517  [000] ....  1526.952862: rtos_queue_send_from_isr_failed: tstamp:47853056077 queue:0x0bcbb3a0
     kworker/0:2-9517  [000] ....  1526.952864: rtos_queue_send_from_isr_failed: tstamp:47853056233 queue:0x0bcbc160
     kworker/0:2-9517  [000] ....  1526.952865: rtcpu_string: tstamp:47853057158 id:0x04010000 str:"VI GOS[#0] set to VM GOS[4] base 0xe4904000
"
     kworker/0:2-9517  [000] ....  1526.952872: rtcpu_vinotify_event: tstamp:47853378894 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49534143136 data:0x10000000
     kworker/0:2-9517  [000] ....  1526.952873: rtcpu_vinotify_event: tstamp:47853379051 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49534147392 data:0x31000001
     kworker/0:2-9517  [000] ....  1526.952875: rtcpu_vinotify_event: tstamp:47853379244 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49534148480 data:0x07020001
     kworker/0:2-9517  [000] ....  1526.952876: rtcpu_vinotify_event: tstamp:47853379392 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49534188736 data:0x10000000
     kworker/0:2-9517  [000] ....  1526.952877: rtcpu_vinotify_event: tstamp:47853379602 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:49534192992 data:0x31000002
     kworker/0:3-1736  [000] ....  1527.054703: rtcpu_vinotify_error: tstamp:47856781788 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47856781078 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.054707: rtcpu_vinotify_event: tstamp:47856817515 tag:FS channel:0x01 frame:0 vi_tstamp:47856780389 data:0x00000014
     kworker/0:3-1736  [000] ....  1527.054709: rtcpu_vinotify_event: tstamp:47856817687 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47856781078 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.054716: rtcpu_vinotify_event: tstamp:47857161429 tag:FS channel:0x00 frame:0 vi_tstamp:47856818701 data:0x00000014
     kworker/0:3-1736  [000] ....  1527.054717: rtcpu_vinotify_event: tstamp:47857161604 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:47856818701 data:0x00000000
     kworker/0:3-1736  [000] ....  1527.054719: rtcpu_vinotify_event: tstamp:47857161752 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:47856819390 data:0x00000001
     kworker/0:3-1736  [000] ....  1527.054720: rtcpu_vinotify_event: tstamp:47857161941 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49654503936 data:0x08020001
     kworker/0:2-9517  [000] ....  1527.105681: rtcpu_vinotify_event: tstamp:47857849139 tag:FE channel:0x01 frame:0 vi_tstamp:47857523026 data:0x00000024
     kworker/0:2-9517  [000] ....  1527.105686: rtos_queue_peek_from_isr_failed: tstamp:47858010168 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1527.156716: rtcpu_vinotify_error: tstamp:47860031381 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47860030693 data:0x00000489
     kworker/0:2-9517  [000] ....  1527.156720: rtcpu_vinotify_event: tstamp:47860069942 tag:FS channel:0x01 frame:0 vi_tstamp:47860001225 data:0x00000014
     kworker/0:2-9517  [000] ....  1527.156722: rtcpu_vinotify_event: tstamp:47860070094 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47860030693 data:0x00000489
     kworker/0:2-9517  [000] ....  1527.156723: rtcpu_vinotify_event: tstamp:47860070287 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:47860068774 data:0x04370002
     kworker/0:2-9517  [000] ....  1527.156725: rtcpu_vinotify_event: tstamp:47860070436 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:47860068795 data:0x00000000
     kworker/0:2-9517  [000] ....  1527.156728: rtos_queue_send_from_isr_failed: tstamp:47860084016 queue:0x0bcb41f8
     kworker/0:2-9517  [000] ....  1527.156730: rtos_queue_send_from_isr_failed: tstamp:47860084180 queue:0x0bcb8a60
     kworker/0:2-9517  [000] ....  1527.156732: rtos_queue_send_from_isr_failed: tstamp:47860084345 queue:0x0bcba5e0
     kworker/0:2-9517  [000] ....  1527.156733: rtos_queue_send_from_isr_failed: tstamp:47860084502 queue:0x0bcbb3a0
     kworker/0:2-9517  [000] ....  1527.156734: rtos_queue_send_from_isr_failed: tstamp:47860084658 queue:0x0bcbc160
     kworker/0:2-9517  [000] ....  1527.156736: rtcpu_vinotify_event: tstamp:47860393710 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49758485664 data:0x02020001
     kworker/0:2-9517  [000] ....  1527.156737: rtcpu_vinotify_event: tstamp:47860393899 tag:CHANSEL_FAULT channel:0x23 frame:0 vi_tstamp:47860069221 data:0x04380040
     kworker/0:2-9517  [000] ....  1527.156738: rtcpu_vinotify_event: tstamp:47860394091 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:49758500192 data:0x01020001
     kworker/0:3-1736  [000] ....  1527.207706: rtcpu_vinotify_event: tstamp:47861081403 tag:FE channel:0x01 frame:0 vi_tstamp:47860773551 data:0x00000024
     kworker/0:3-1736  [000] ....  1527.258695: rtos_queue_peek_from_isr_failed: tstamp:47863010165 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1527.258700: rtcpu_vinotify_error: tstamp:47863281957 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47863281250 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.258702: rtcpu_vinotify_event: tstamp:47863488479 tag:FS channel:0x01 frame:0 vi_tstamp:47863251528 data:0x00000014
     kworker/0:3-1736  [000] ....  1527.258739: rtcpu_vinotify_event: tstamp:47863488628 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47863281250 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.309698: rtcpu_vinotify_event: tstamp:47864176184 tag:FE channel:0x01 frame:0 vi_tstamp:47864023841 data:0x00000024
     kworker/0:3-1736  [000] ....  1527.360690: rtcpu_vinotify_error: tstamp:47866531785 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47866531078 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.360696: rtcpu_vinotify_event: tstamp:47866583259 tag:FS channel:0x01 frame:0 vi_tstamp:47866501368 data:0x00000014
     kworker/0:3-1736  [000] ....  1527.360697: rtcpu_vinotify_event: tstamp:47866583481 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47866531078 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.411704: rtcpu_vinotify_event: tstamp:47867614864 tag:FE channel:0x01 frame:0 vi_tstamp:47867273672 data:0x00000024
     kworker/0:3-1736  [000] ....  1527.411710: rtos_queue_peek_from_isr_failed: tstamp:47868010164 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1527.462719: rtcpu_vinotify_error: tstamp:47869782074 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47869781369 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.513704: rtcpu_vinotify_event: tstamp:47870021804 tag:FS channel:0x01 frame:0 vi_tstamp:47869751669 data:0x00000014
     kworker/0:3-1736  [000] ....  1527.513709: rtcpu_vinotify_event: tstamp:47870021957 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47869781369 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.513711: rtcpu_vinotify_event: tstamp:47870709653 tag:FE channel:0x01 frame:0 vi_tstamp:47870523962 data:0x00000024
     kworker/0:3-1736  [000] ....  1527.564697: rtos_queue_peek_from_isr_failed: tstamp:47873010164 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1527.564701: rtcpu_vinotify_error: tstamp:47873031227 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47873030523 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.564703: rtcpu_vinotify_event: tstamp:47873116626 tag:FS channel:0x01 frame:0 vi_tstamp:47873001047 data:0x00000014
     kworker/0:3-1736  [000] ....  1527.564704: rtcpu_vinotify_event: tstamp:47873116798 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47873030523 data:0x00000489
     kworker/0:3-1736  [000] ....  1527.615727: rtcpu_vinotify_event: tstamp:47873773713 tag:FE channel:0x01 frame:0 vi_tstamp:47873773122 data:0x00000024
     kworker/0:3-1736  [000] ....  1527.666703: rtcpu_vinotify_event: tstamp:47876280156 tag:FS channel:0x01 frame:0 vi_tstamp:47876250895 data:0x00000014
     kworker/0:3-1736  [000] ....  1527.666707: rtcpu_vinotify_error: tstamp:47876281060 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47876280362 data:0x00000489
     kworker/0:2-9517  [000] ....  1527.717709: rtcpu_vinotify_event: tstamp:47876624033 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47876280362 data:0x00000489
     kworker/0:2-9517  [000] ....  1527.717712: rtcpu_vinotify_event: tstamp:47877311738 tag:FE channel:0x01 frame:0 vi_tstamp:47877022959 data:0x00000024
     kworker/0:2-9517  [000] ....  1527.768696: rtos_queue_peek_from_isr_failed: tstamp:47878010146 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1527.768703: rtcpu_vinotify_error: tstamp:47879532031 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47879531326 data:0x00000489
     kworker/0:2-9517  [000] ....  1527.819704: rtcpu_vinotify_event: tstamp:47879718815 tag:FS channel:0x01 frame:0 vi_tstamp:47879501650 data:0x00000014
     kworker/0:2-9517  [000] ....  1527.819707: rtcpu_vinotify_event: tstamp:47879718990 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47879531326 data:0x00000489
     kworker/0:2-9517  [000] ....  1527.819708: rtcpu_vinotify_event: tstamp:47880406523 tag:FE channel:0x01 frame:0 vi_tstamp:47880273921 data:0x00000024
     kworker/0:2-9517  [000] ....  1527.921693: rtcpu_vinotify_error: tstamp:47882781398 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47882780694 data:0x00000489
     kworker/0:2-9517  [000] ....  1527.921697: rtcpu_vinotify_event: tstamp:47882813601 tag:FS channel:0x01 frame:0 vi_tstamp:47882751038 data:0x00000014
     kworker/0:2-9517  [000] ....  1527.921699: rtcpu_vinotify_event: tstamp:47882813749 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47882780694 data:0x00000489
     kworker/0:2-9517  [000] ....  1527.921701: rtos_queue_peek_from_isr_failed: tstamp:47883010162 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1527.921703: rtcpu_vinotify_event: tstamp:47883845205 tag:FE channel:0x01 frame:0 vi_tstamp:47883523289 data:0x00000024
     kworker/0:2-9517  [000] ....  1528.023695: rtcpu_vinotify_error: tstamp:47886031691 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47886030985 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.023698: rtcpu_vinotify_event: tstamp:47886252181 tag:FS channel:0x01 frame:0 vi_tstamp:47886001339 data:0x00000014
     kworker/0:2-9517  [000] ....  1528.023700: rtcpu_vinotify_event: tstamp:47886252351 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47886030985 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.023702: rtcpu_vinotify_event: tstamp:47886939995 tag:FE channel:0x01 frame:0 vi_tstamp:47886773580 data:0x00000024
     kworker/0:2-9517  [000] ....  1528.074697: rtos_queue_peek_from_isr_failed: tstamp:47888010168 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1528.125710: rtcpu_vinotify_error: tstamp:47889281350 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47889280644 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.125714: rtcpu_vinotify_event: tstamp:47889346971 tag:FS channel:0x01 frame:0 vi_tstamp:47889251178 data:0x00000014
     kworker/0:2-9517  [000] ....  1528.125716: rtcpu_vinotify_event: tstamp:47889347120 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47889280644 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.125718: rtcpu_vinotify_event: tstamp:47890034681 tag:FE channel:0x01 frame:0 vi_tstamp:47890023241 data:0x00000024
     kworker/0:3-1736  [000] ....  1528.227689: rtcpu_vinotify_event: tstamp:47892501604 tag:FS channel:0x01 frame:0 vi_tstamp:47892501018 data:0x00000014
     kworker/0:3-1736  [000] ....  1528.227694: rtcpu_vinotify_error: tstamp:47892531358 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47892530652 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.227696: rtcpu_vinotify_event: tstamp:47892820060 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47892530652 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.227732: rtos_queue_peek_from_isr_failed: tstamp:47893010164 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1528.227734: rtcpu_vinotify_event: tstamp:47893507766 tag:FE channel:0x01 frame:0 vi_tstamp:47893273246 data:0x00000024
     kworker/0:3-1736  [000] ....  1528.329701: rtcpu_vinotify_error: tstamp:47895781647 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47895780943 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.329704: rtcpu_vinotify_event: tstamp:47895914842 tag:FS channel:0x01 frame:0 vi_tstamp:47895751319 data:0x00000014
     kworker/0:3-1736  [000] ....  1528.329706: rtcpu_vinotify_event: tstamp:47895914990 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47895780943 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.329741: rtcpu_vinotify_event: tstamp:47896602552 tag:FE channel:0x01 frame:0 vi_tstamp:47896523537 data:0x00000024
     kworker/0:3-1736  [000] ....  1528.380721: rtos_queue_peek_from_isr_failed: tstamp:47898010165 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1528.431693: rtcpu_vinotify_event: tstamp:47899009558 tag:FS channel:0x01 frame:0 vi_tstamp:47899001159 data:0x00000014
     kworker/0:3-1736  [000] ....  1528.431695: rtcpu_vinotify_error: tstamp:47899031479 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47899030772 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.431697: rtcpu_vinotify_event: tstamp:47899353433 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47899030772 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.431698: rtcpu_vinotify_event: tstamp:47900041144 tag:FE channel:0x01 frame:0 vi_tstamp:47899773367 data:0x00000024
     kworker/0:3-1736  [000] ....  1528.533700: rtcpu_vinotify_error: tstamp:47902281774 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47902281070 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.533704: rtcpu_vinotify_event: tstamp:47902448216 tag:FS channel:0x01 frame:0 vi_tstamp:47902251460 data:0x00000014
     kworker/0:3-1736  [000] ....  1528.533706: rtcpu_vinotify_event: tstamp:47902448364 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47902281070 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.533708: rtos_queue_peek_from_isr_failed: tstamp:47903010162 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1528.533710: rtcpu_vinotify_event: tstamp:47903135926 tag:FE channel:0x01 frame:0 vi_tstamp:47903023665 data:0x00000024
     kworker/0:3-1736  [000] ....  1528.635712: rtcpu_vinotify_error: tstamp:47905531480 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47905530776 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.635717: rtcpu_vinotify_event: tstamp:47905542997 tag:FS channel:0x01 frame:0 vi_tstamp:47905501307 data:0x00000014
     kworker/0:3-1736  [000] ....  1528.635719: rtcpu_vinotify_event: tstamp:47905543185 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47905530776 data:0x00000489
     kworker/0:3-1736  [000] ....  1528.635721: rtcpu_vinotify_event: tstamp:47906574615 tag:FE channel:0x01 frame:0 vi_tstamp:47906273372 data:0x00000024
     kworker/0:2-9517  [000] ....  1528.686695: rtos_queue_peek_from_isr_failed: tstamp:47908010164 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1528.737696: rtcpu_vinotify_error: tstamp:47908781901 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47908781194 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.737699: rtcpu_vinotify_event: tstamp:47908782209 tag:FS channel:0x01 frame:0 vi_tstamp:47908751610 data:0x00000014
     kworker/0:2-9517  [000] ....  1528.737701: rtcpu_vinotify_event: tstamp:47908782359 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47908781194 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.737702: rtcpu_vinotify_event: tstamp:47909531810 tag:FE channel:0x01 frame:0 vi_tstamp:47909523786 data:0x00000024
     kworker/0:2-9517  [000] ....  1528.839701: rtcpu_vinotify_error: tstamp:47912031266 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47912030562 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.839704: rtcpu_vinotify_event: tstamp:47912282682 tag:FS channel:0x01 frame:0 vi_tstamp:47912000989 data:0x00000014
     kworker/0:2-9517  [000] ....  1528.839706: rtcpu_vinotify_event: tstamp:47912282853 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47912030562 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.839707: rtcpu_vinotify_event: tstamp:47912970388 tag:FE channel:0x01 frame:0 vi_tstamp:47912773154 data:0x00000024
     kworker/0:2-9517  [000] ....  1528.839710: rtos_queue_peek_from_isr_failed: tstamp:47913010162 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1528.941729: rtcpu_vinotify_error: tstamp:47915281012 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47915280305 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.941735: rtcpu_vinotify_event: tstamp:47915377469 tag:FS channel:0x01 frame:0 vi_tstamp:47915250828 data:0x00000014
     kworker/0:2-9517  [000] ....  1528.941737: rtcpu_vinotify_event: tstamp:47915377617 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47915280305 data:0x00000489
     kworker/0:2-9517  [000] ....  1528.941738: rtcpu_vinotify_event: tstamp:47916065177 tag:FE channel:0x01 frame:0 vi_tstamp:47916022901 data:0x00000024
     kworker/0:2-9517  [000] ....  1529.045693: rtos_queue_peek_from_isr_failed: tstamp:47918010164 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1529.045699: rtcpu_vinotify_error: tstamp:47918531773 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47918531068 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.045701: rtcpu_vinotify_event: tstamp:47918816149 tag:FS channel:0x01 frame:0 vi_tstamp:47918501590 data:0x00000014
     kworker/0:2-9517  [000] ....  1529.045702: rtcpu_vinotify_event: tstamp:47918816320 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47918531068 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.097749: rtcpu_vinotify_event: tstamp:47919503862 tag:FE channel:0x01 frame:0 vi_tstamp:47919273663 data:0x00000024
     kworker/0:2-9517  [000] ....  1529.148695: rtcpu_vinotify_error: tstamp:47921781149 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47921780444 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.148731: rtcpu_vinotify_event: tstamp:47921910833 tag:FS channel:0x01 frame:0 vi_tstamp:47921750969 data:0x00000014
     kworker/0:2-9517  [000] ....  1529.148733: rtcpu_vinotify_event: tstamp:47921910981 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47921780444 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.148734: rtcpu_vinotify_event: tstamp:47922598641 tag:FE channel:0x01 frame:0 vi_tstamp:47922523042 data:0x00000024
     kworker/0:3-1736  [000] ....  1529.201753: rtos_queue_peek_from_isr_failed: tstamp:47923010165 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1529.252690: rtcpu_vinotify_event: tstamp:47925005641 tag:FS channel:0x01 frame:0 vi_tstamp:47925000816 data:0x00000014
     kworker/0:3-1736  [000] ....  1529.252693: rtcpu_vinotify_error: tstamp:47925030988 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47925030284 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.252731: rtcpu_vinotify_event: tstamp:47925349517 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47925030284 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.252733: rtcpu_vinotify_event: tstamp:47925773470 tag:FE channel:0x01 frame:0 vi_tstamp:47925772882 data:0x00000024
     kworker/0:3-1736  [000] ....  1529.356719: rtos_queue_peek_from_isr_failed: tstamp:47928010166 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1529.356727: rtcpu_vinotify_event: tstamp:47928272327 tag:FS channel:0x01 frame:0 vi_tstamp:47928251119 data:0x00000014
     kworker/0:3-1736  [000] ....  1529.356729: rtcpu_vinotify_error: tstamp:47928281335 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47928280640 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.356731: rtcpu_vinotify_event: tstamp:47928616202 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47928280640 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.409779: rtcpu_vinotify_event: tstamp:47929303911 tag:FE channel:0x01 frame:0 vi_tstamp:47929023241 data:0x00000024
     kworker/0:3-1736  [000] ....  1529.460705: rtcpu_vinotify_error: tstamp:47931531128 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47931530424 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.460708: rtcpu_vinotify_event: tstamp:47931710987 tag:FS channel:0x01 frame:0 vi_tstamp:47931500960 data:0x00000014
     kworker/0:3-1736  [000] ....  1529.460710: rtcpu_vinotify_event: tstamp:47931711158 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47931530424 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.460712: rtcpu_vinotify_event: tstamp:47932398697 tag:FE channel:0x01 frame:0 vi_tstamp:47932273282 data:0x00000024
     kworker/0:3-1736  [000] ....  1529.513723: rtos_queue_peek_from_isr_failed: tstamp:47933010165 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1529.564688: rtcpu_vinotify_error: tstamp:47934781473 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47934780768 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.564691: rtcpu_vinotify_event: tstamp:47934805769 tag:FS channel:0x01 frame:0 vi_tstamp:47934751260 data:0x00000014
     kworker/0:3-1736  [000] ....  1529.564693: rtcpu_vinotify_event: tstamp:47934805917 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47934780768 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.617743: rtcpu_vinotify_event: tstamp:47935837382 tag:FE channel:0x01 frame:0 vi_tstamp:47935523363 data:0x00000024
     kworker/0:3-1736  [000] ....  1529.668688: rtos_queue_peek_from_isr_failed: tstamp:47938010162 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1529.668692: rtcpu_vinotify_error: tstamp:47938031279 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47938030575 data:0x00000489
     kworker/0:3-1736  [000] ....  1529.668694: rtcpu_vinotify_event: tstamp:47938244353 tag:FS channel:0x01 frame:0 vi_tstamp:47938001100 data:0x00000014
     kworker/0:3-1736  [000] ....  1529.668695: rtcpu_vinotify_event: tstamp:47938244523 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47938030575 data:0x00000489
 vi-output, jbar-9612  [000] ....  1529.698593: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt a
     kworker/0:2-9517  [000] ....  1529.721727: rtcpu_vinotify_event: tstamp:47938932168 tag:FE channel:0x01 frame:0 vi_tstamp:47938773430 data:0x00000024
     kworker/0:2-9517  [000] ....  1529.721732: rtos_queue_send_from_isr_failed: tstamp:47939780235 queue:0x0bcb41f8
     kworker/0:2-9517  [000] ....  1529.721734: rtos_queue_send_from_isr_failed: tstamp:47939780397 queue:0x0bcb8a60
     kworker/0:2-9517  [000] ....  1529.721735: rtos_queue_send_from_isr_failed: tstamp:47939780559 queue:0x0bcba5e0
     kworker/0:2-9517  [000] ....  1529.721737: rtos_queue_send_from_isr_failed: tstamp:47939780717 queue:0x0bcbb3a0
     kworker/0:2-9517  [000] ....  1529.721738: rtos_queue_send_from_isr_failed: tstamp:47939780874 queue:0x0bcbc160
     kworker/0:2-9517  [000] ....  1529.721739: rtos_queue_send_from_isr_failed: tstamp:47939847882 queue:0x0bcb41f8
     kworker/0:2-9517  [000] ....  1529.721741: rtos_queue_send_from_isr_failed: tstamp:47939848042 queue:0x0bcb8a60
     kworker/0:2-9517  [000] ....  1529.721772: rtos_queue_send_from_isr_failed: tstamp:47939848203 queue:0x0bcba5e0
     kworker/0:2-9517  [000] ....  1529.721774: rtos_queue_send_from_isr_failed: tstamp:47939848360 queue:0x0bcbb3a0
     kworker/0:2-9517  [000] ....  1529.721775: rtos_queue_send_from_isr_failed: tstamp:47939848517 queue:0x0bcbc160
     kworker/0:2-9517  [000] ....  1529.721776: rtos_queue_send_from_isr_failed: tstamp:47939858592 queue:0x0bcb41f8
     kworker/0:2-9517  [000] ....  1529.721778: rtos_queue_send_from_isr_failed: tstamp:47939858752 queue:0x0bcb8a60
     kworker/0:2-9517  [000] ....  1529.721779: rtos_queue_send_from_isr_failed: tstamp:47939858912 queue:0x0bcba5e0
     kworker/0:2-9517  [000] ....  1529.721780: rtos_queue_send_from_isr_failed: tstamp:47939859069 queue:0x0bcbb3a0
     kworker/0:2-9517  [000] ....  1529.721782: rtos_queue_send_from_isr_failed: tstamp:47939859226 queue:0x0bcbc160
     kworker/0:2-9517  [000] ....  1529.721786: rtcpu_string: tstamp:47939859687 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:2-9517  [000] ....  1529.721799: rtcpu_string: tstamp:47939859877 id:0x04010000 str:"VM GOS[#0] addr=0xe4900000
"
     kworker/0:2-9517  [000] ....  1529.721804: rtcpu_string: tstamp:47939860143 id:0x04010000 str:"VM GOS[#1] addr=0xe4901000
"
     kworker/0:2-9517  [000] ....  1529.721808: rtcpu_string: tstamp:47939860426 id:0x04010000 str:"VM GOS[#2] addr=0xe4902000
"
     kworker/0:2-9517  [000] ....  1529.721815: rtcpu_string: tstamp:47939860672 id:0x04010000 str:"VM GOS[#3] addr=0xe4903000
"
     kworker/0:2-9517  [000] ....  1529.721819: rtcpu_string: tstamp:47939860913 id:0x04010000 str:"VM GOS[#4] addr=0xe4904000
"
     kworker/0:2-9517  [000] ....  1529.721823: rtcpu_string: tstamp:47939861156 id:0x04010000 str:"VM GOS[#5] addr=0xe4905000
"
     kworker/0:2-9517  [000] ....  1529.721827: rtos_queue_send_from_isr_failed: tstamp:47939873158 queue:0x0bcb41f8
     kworker/0:2-9517  [000] ....  1529.721829: rtos_queue_send_from_isr_failed: tstamp:47939873337 queue:0x0bcb8a60
     kworker/0:2-9517  [000] ....  1529.721863: rtos_queue_send_from_isr_failed: tstamp:47939873500 queue:0x0bcba5e0
     kworker/0:2-9517  [000] ....  1529.721864: rtos_queue_send_from_isr_failed: tstamp:47939873656 queue:0x0bcbb3a0
     kworker/0:2-9517  [000] ....  1529.721865: rtos_queue_send_from_isr_failed: tstamp:47939873813 queue:0x0bcbc160
     kworker/0:2-9517  [000] ....  1529.721867: rtcpu_vinotify_event: tstamp:47939875630 tag:CHANSEL_FAULT_FE channel:0x10 frame:0 vi_tstamp:47939781384 data:0x00000000
     kworker/0:2-9517  [000] ....  1529.721868: rtcpu_vinotify_event: tstamp:47939875790 tag:CHANSEL_LOAD_FRAMED channel:0x10 frame:0 vi_tstamp:47939781384 data:0x38000002
     kworker/0:2-9517  [000] ....  1529.721870: rtcpu_vinotify_event: tstamp:47939875968 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:47939781384 data:0x00000000
     kworker/0:2-9517  [000] ....  1529.721871: rtcpu_vinotify_event: tstamp:47939876117 tag:FE channel:0x00 frame:0 vi_tstamp:47939874962 data:0x00000024
     kworker/0:2-9517  [000] ....  1529.721872: rtcpu_vinotify_event: tstamp:47940204435 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:48017336128 data:0x10000000
     kworker/0:2-9517  [000] ....  1529.721874: rtcpu_vinotify_event: tstamp:47940204588 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:48017340384 data:0x31000001
     kworker/0:2-9517  [000] ....  1529.721875: rtcpu_vinotify_event: tstamp:47940204781 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:48017341472 data:0x07020001
     kworker/0:2-9517  [000] ....  1529.721876: rtcpu_vinotify_event: tstamp:47940204928 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:48017407872 data:0x10000000
     kworker/0:2-9517  [000] ....  1529.721878: rtcpu_vinotify_event: tstamp:47940205117 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:48017412160 data:0x31000002
     kworker/0:2-9517  [000] ....  1529.772751: rtcpu_vinotify_error: tstamp:47941281358 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47941280415 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.772756: rtcpu_vinotify_error: tstamp:47941281783 tag:CSIMUX_STREAM channel:0x00 frame:0 vi_tstamp:47941280673 data:0x00010000
     kworker/0:2-9517  [000] ....  1529.772758: rtcpu_vinotify_event: tstamp:47941579860 tag:FS channel:0x01 frame:0 vi_tstamp:47941250940 data:0x00000014
     kworker/0:2-9517  [000] ....  1529.772760: rtcpu_vinotify_event: tstamp:47941580052 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47941280415 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.772761: rtcpu_vinotify_event: tstamp:47941580205 tag:CSIMUX_STREAM channel:0x00 frame:0 vi_tstamp:47941280673 data:0x00010000
     kworker/0:2-9517  [000] ....  1529.825757: rtcpu_vinotify_event: tstamp:47942267560 tag:FE channel:0x01 frame:0 vi_tstamp:47942023270 data:0x00000024
     kworker/0:2-9517  [000] ....  1529.825763: rtos_queue_peek_from_isr_failed: tstamp:47943010164 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1529.876714: rtcpu_vinotify_error: tstamp:47944531424 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47944530715 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.876717: rtcpu_vinotify_event: tstamp:47944674616 tag:FS channel:0x01 frame:0 vi_tstamp:47944501240 data:0x00000014
     kworker/0:2-9517  [000] ....  1529.876719: rtcpu_vinotify_event: tstamp:47944674789 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47944530715 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.876720: rtcpu_vinotify_event: tstamp:47945362321 tag:FE channel:0x01 frame:0 vi_tstamp:47945273571 data:0x00000024
     kworker/0:2-9517  [000] ....  1529.980709: rtcpu_vinotify_event: tstamp:47947769401 tag:FS channel:0x01 frame:0 vi_tstamp:47947751087 data:0x00000014
     kworker/0:2-9517  [000] ....  1529.980712: rtcpu_vinotify_error: tstamp:47947781258 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47947780556 data:0x00000489
     kworker/0:2-9517  [000] ....  1529.980716: rtos_queue_peek_from_isr_failed: tstamp:47948010164 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1529.980718: rtcpu_vinotify_event: tstamp:47948113278 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47947780556 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.033734: rtcpu_vinotify_event: tstamp:47948800983 tag:FE channel:0x01 frame:0 vi_tstamp:47948523152 data:0x00000024
     kworker/0:2-9517  [000] ....  1530.084701: rtcpu_vinotify_event: tstamp:47951001983 tag:FS channel:0x01 frame:0 vi_tstamp:47951001391 data:0x00000014
     kworker/0:2-9517  [000] ....  1530.084703: rtcpu_vinotify_error: tstamp:47951031565 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47951030857 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.084705: rtcpu_vinotify_event: tstamp:47951345607 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47951030857 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.137807: rtcpu_vinotify_event: tstamp:47952033312 tag:FE channel:0x01 frame:0 vi_tstamp:47951773453 data:0x00000024
     kworker/0:2-9517  [000] ....  1530.137813: rtos_queue_peek_from_isr_failed: tstamp:47953010166 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1530.188689: rtcpu_vinotify_error: tstamp:47954281402 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47954280697 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.188695: rtcpu_vinotify_event: tstamp:47954440362 tag:FS channel:0x01 frame:0 vi_tstamp:47954251231 data:0x00000014
     kworker/0:2-9517  [000] ....  1530.188696: rtcpu_vinotify_event: tstamp:47954440511 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47954280697 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.188698: rtcpu_vinotify_event: tstamp:47955128067 tag:FE channel:0x01 frame:0 vi_tstamp:47955023551 data:0x00000024
     kworker/0:3-1736  [000] ....  1530.292692: rtcpu_vinotify_error: tstamp:47957531245 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47957530537 data:0x00000489
     kworker/0:3-1736  [000] ....  1530.292696: rtcpu_vinotify_event: tstamp:47957535118 tag:FS channel:0x01 frame:0 vi_tstamp:47957501070 data:0x00000014
     kworker/0:3-1736  [000] ....  1530.292698: rtcpu_vinotify_event: tstamp:47957535288 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47957530537 data:0x00000489
     kworker/0:3-1736  [000] ....  1530.292702: rtos_queue_peek_from_isr_failed: tstamp:47958010164 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1530.345735: rtcpu_vinotify_event: tstamp:47958566705 tag:FE channel:0x01 frame:0 vi_tstamp:47958273399 data:0x00000024
     kworker/0:3-1736  [000] ....  1530.396696: rtcpu_vinotify_error: tstamp:47960781091 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47960780387 data:0x00000489
     kworker/0:3-1736  [000] ....  1530.396700: rtcpu_vinotify_event: tstamp:47960973762 tag:FS channel:0x01 frame:0 vi_tstamp:47960750910 data:0x00000014
     kworker/0:3-1736  [000] ....  1530.396702: rtcpu_vinotify_event: tstamp:47960973911 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47960780387 data:0x00000489
     kworker/0:3-1736  [000] ....  1530.396704: rtcpu_vinotify_event: tstamp:47961661468 tag:FE channel:0x01 frame:0 vi_tstamp:47961523240 data:0x00000024
     kworker/0:3-1736  [000] ....  1530.449784: rtos_queue_peek_from_isr_failed: tstamp:47963010165 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1530.500696: rtcpu_vinotify_error: tstamp:47964031393 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47964030688 data:0x00000489
     kworker/0:3-1736  [000] ....  1530.500701: rtcpu_vinotify_event: tstamp:47964068544 tag:FS channel:0x01 frame:0 vi_tstamp:47964001210 data:0x00000014
     kworker/0:3-1736  [000] ....  1530.500703: rtcpu_vinotify_event: tstamp:47964068715 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47964030688 data:0x00000489
     kworker/0:3-1736  [000] ....  1530.553746: rtcpu_vinotify_event: tstamp:47965100147 tag:FE channel:0x01 frame:0 vi_tstamp:47964773282 data:0x00000024
     kworker/0:3-1736  [000] ....  1530.604696: rtcpu_vinotify_error: tstamp:47967281232 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47967280527 data:0x00000489
     kworker/0:3-1736  [000] ....  1530.604700: rtcpu_vinotify_event: tstamp:47967281543 tag:FS channel:0x01 frame:0 vi_tstamp:47967251050 data:0x00000014
     kworker/0:3-1736  [000] ....  1530.604702: rtcpu_vinotify_event: tstamp:47967281694 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47967280527 data:0x00000489
     kworker/0:3-1736  [000] ....  1530.604705: rtos_queue_peek_from_isr_failed: tstamp:47968010164 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1530.604708: rtcpu_vinotify_event: tstamp:47968057325 tag:FE channel:0x01 frame:0 vi_tstamp:47968023381 data:0x00000024
     kworker/0:2-9517  [000] ....  1530.708691: rtcpu_vinotify_error: tstamp:47970531534 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47970530828 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.708695: rtcpu_vinotify_event: tstamp:47970808274 tag:FS channel:0x01 frame:0 vi_tstamp:47970501359 data:0x00000014
     kworker/0:2-9517  [000] ....  1530.708696: rtcpu_vinotify_event: tstamp:47970808447 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47970530828 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.761738: rtcpu_vinotify_event: tstamp:47971495982 tag:FE channel:0x01 frame:0 vi_tstamp:47971273423 data:0x00000024
     kworker/0:2-9517  [000] ....  1530.761742: rtos_queue_peek_from_isr_failed: tstamp:47973010164 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1530.812689: rtcpu_vinotify_error: tstamp:47973780913 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47973780207 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.812692: rtcpu_vinotify_event: tstamp:47973903037 tag:FS channel:0x01 frame:0 vi_tstamp:47973750739 data:0x00000014
     kworker/0:2-9517  [000] ....  1530.812694: rtcpu_vinotify_event: tstamp:47973903186 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47973780207 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.812696: rtcpu_vinotify_event: tstamp:47974590721 tag:FE channel:0x01 frame:0 vi_tstamp:47974523062 data:0x00000024
     kworker/0:2-9517  [000] ....  1530.916695: rtcpu_vinotify_error: tstamp:47977031680 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47977030970 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.916698: rtcpu_vinotify_event: tstamp:47977341678 tag:FS channel:0x01 frame:0 vi_tstamp:47977001502 data:0x00000014
     kworker/0:2-9517  [000] ....  1530.916733: rtcpu_vinotify_event: tstamp:47977341849 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47977030970 data:0x00000489
     kworker/0:2-9517  [000] ....  1530.969804: rtos_queue_peek_from_isr_failed: tstamp:47978010169 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1530.969810: rtcpu_vinotify_event: tstamp:47978029379 tag:FE channel:0x01 frame:0 vi_tstamp:47977773822 data:0x00000024
     kworker/0:2-9517  [000] ....  1531.020691: rtcpu_vinotify_error: tstamp:47980281512 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47980280807 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.020696: rtcpu_vinotify_event: tstamp:47980436456 tag:FS channel:0x01 frame:0 vi_tstamp:47980251341 data:0x00000014
     kworker/0:2-9517  [000] ....  1531.020698: rtcpu_vinotify_event: tstamp:47980436606 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47980280807 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.020700: rtcpu_vinotify_event: tstamp:47981124161 tag:FE channel:0x01 frame:0 vi_tstamp:47981023663 data:0x00000024
     kworker/0:2-9517  [000] ....  1531.124695: rtos_queue_peek_from_isr_failed: tstamp:47983010165 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1531.124699: rtcpu_vinotify_error: tstamp:47983531359 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47983530654 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.124702: rtcpu_vinotify_event: tstamp:47983531651 tag:FS channel:0x01 frame:0 vi_tstamp:47983501181 data:0x00000014
     kworker/0:2-9517  [000] ....  1531.124703: rtcpu_vinotify_event: tstamp:47983531822 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47983530654 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.124705: rtcpu_vinotify_event: tstamp:47984274099 tag:FE channel:0x01 frame:0 vi_tstamp:47984273512 data:0x00000024
     kworker/0:3-1736  [000] ....  1531.228689: rtcpu_vinotify_event: tstamp:47986763535 tag:FS channel:0x01 frame:0 vi_tstamp:47986751021 data:0x00000014
     kworker/0:3-1736  [000] ....  1531.228694: rtcpu_vinotify_error: tstamp:47986781192 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47986780496 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.228726: rtcpu_vinotify_event: tstamp:47987107426 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47986780496 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.281774: rtcpu_vinotify_event: tstamp:47987795131 tag:FE channel:0x01 frame:0 vi_tstamp:47987523352 data:0x00000024
     kworker/0:3-1736  [000] ....  1531.281779: rtos_queue_peek_from_isr_failed: tstamp:47988010166 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1531.332716: rtcpu_vinotify_error: tstamp:47990031040 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47990030336 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.332720: rtcpu_vinotify_event: tstamp:47990202185 tag:FS channel:0x01 frame:0 vi_tstamp:47990000860 data:0x00000014
     kworker/0:3-1736  [000] ....  1531.332721: rtcpu_vinotify_event: tstamp:47990202356 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47990030336 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.332723: rtcpu_vinotify_event: tstamp:47990889855 tag:FE channel:0x01 frame:0 vi_tstamp:47990773193 data:0x00000024
     kworker/0:3-1736  [000] ....  1531.436697: rtos_queue_peek_from_isr_failed: tstamp:47993010166 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1531.436702: rtcpu_vinotify_error: tstamp:47993281803 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47993281098 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.436704: rtcpu_vinotify_event: tstamp:47993296949 tag:FS channel:0x01 frame:0 vi_tstamp:47993251630 data:0x00000014
     kworker/0:3-1736  [000] ....  1531.436707: rtcpu_vinotify_event: tstamp:47993297098 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47993281098 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.489707: rtcpu_vinotify_event: tstamp:47994328556 tag:FE channel:0x01 frame:0 vi_tstamp:47994023952 data:0x00000024
     kworker/0:3-1736  [000] ....  1531.540702: rtcpu_vinotify_error: tstamp:47996531180 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47996530476 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.540705: rtcpu_vinotify_event: tstamp:47996735608 tag:FS channel:0x01 frame:0 vi_tstamp:47996501011 data:0x00000014
     kworker/0:3-1736  [000] ....  1531.540707: rtcpu_vinotify_event: tstamp:47996735799 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47996530476 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.593735: rtcpu_vinotify_event: tstamp:47997423316 tag:FE channel:0x01 frame:0 vi_tstamp:47997273331 data:0x00000024
     kworker/0:3-1736  [000] ....  1531.593738: rtos_queue_peek_from_isr_failed: tstamp:47998010164 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1531.644695: rtcpu_vinotify_error: tstamp:47999781020 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47999780316 data:0x00000489
     kworker/0:3-1736  [000] ....  1531.644698: rtcpu_vinotify_event: tstamp:47999830376 tag:FS channel:0x01 frame:0 vi_tstamp:47999750850 data:0x00000014
     kworker/0:3-1736  [000] ....  1531.644699: rtcpu_vinotify_event: tstamp:47999830545 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:47999780316 data:0x00000489
        v4l2-ctl-9610  [001] ....  1531.673022: tegra_channel_close: vi-output, jbar0233 2-0010
     kworker/0:3-1736  [000] ....  1531.697731: rtcpu_vinotify_event: tstamp:48000861879 tag:FE channel:0x01 frame:0 vi_tstamp:48000523171 data:0x00000024
     kworker/0:2-9517  [000] ....  1531.748709: rtcpu_vinotify_event: tstamp:48003001742 tag:FS channel:0x01 frame:0 vi_tstamp:48003001151 data:0x00000014
     kworker/0:2-9517  [000] ....  1531.748715: rtos_queue_peek_from_isr_failed: tstamp:48003010160 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1531.748717: rtcpu_vinotify_error: tstamp:48003031336 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48003030625 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.748719: rtcpu_vinotify_event: tstamp:48003337734 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48003030625 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.801777: rtcpu_vinotify_event: tstamp:48004025440 tag:FE channel:0x01 frame:0 vi_tstamp:48003773479 data:0x00000024
     kworker/0:2-9517  [000] ....  1531.852699: rtcpu_vinotify_error: tstamp:48006281171 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48006280466 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.852702: rtcpu_vinotify_event: tstamp:48006432495 tag:FS channel:0x01 frame:0 vi_tstamp:48006250991 data:0x00000014
     kworker/0:2-9517  [000] ....  1531.852704: rtcpu_vinotify_event: tstamp:48006432643 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48006280466 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.852706: rtcpu_vinotify_event: tstamp:48007120201 tag:FE channel:0x01 frame:0 vi_tstamp:48007023321 data:0x00000024
     kworker/0:2-9517  [000] ....  1531.905711: rtos_queue_peek_from_isr_failed: tstamp:48008010165 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1531.956694: rtcpu_vinotify_event: tstamp:48009527286 tag:FS channel:0x01 frame:0 vi_tstamp:48009500830 data:0x00000014
     kworker/0:2-9517  [000] ....  1531.956696: rtcpu_vinotify_error: tstamp:48009531004 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48009530307 data:0x00000489
     kworker/0:2-9517  [000] ....  1531.956698: rtcpu_vinotify_event: tstamp:48009871160 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48009530307 data:0x00000489
     kworker/0:2-9517  [000] ....  1532.009749: rtcpu_vinotify_event: tstamp:48010558866 tag:FE channel:0x01 frame:0 vi_tstamp:48010272947 data:0x00000024
     kworker/0:2-9517  [000] ....  1532.060687: rtcpu_vinotify_error: tstamp:48012780852 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48012780147 data:0x00000489
     kworker/0:2-9517  [000] ....  1532.060690: rtcpu_vinotify_event: tstamp:48012965921 tag:FS channel:0x01 frame:0 vi_tstamp:48012750678 data:0x00000014
     kworker/0:2-9517  [000] ....  1532.060692: rtcpu_vinotify_event: tstamp:48012966070 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48012780147 data:0x00000489
     kworker/0:2-9517  [000] ....  1532.060694: rtos_queue_peek_from_isr_failed: tstamp:48013010162 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1532.060695: rtcpu_vinotify_event: tstamp:48013653625 tag:FE channel:0x01 frame:0 vi_tstamp:48013523000 data:0x00000024
     kworker/0:2-9517  [000] ....  1532.164716: rtcpu_vinotify_error: tstamp:48016031411 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48016030706 data:0x00000489
     kworker/0:2-9517  [000] ....  1532.164722: rtcpu_vinotify_event: tstamp:48016060685 tag:FS channel:0x01 frame:0 vi_tstamp:48016000981 data:0x00000014
     kworker/0:2-9517  [000] ....  1532.164723: rtcpu_vinotify_event: tstamp:48016060856 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48016030706 data:0x00000489
     kworker/0:3-1736  [000] ....  1532.217749: rtcpu_vinotify_event: tstamp:48017092286 tag:FE channel:0x01 frame:0 vi_tstamp:48016773302 data:0x00000024
     kworker/0:3-1736  [000] ....  1532.217756: rtos_queue_peek_from_isr_failed: tstamp:48018010164 queue:0x0bcbcf78
        v4l2-ctl-9610  [000] ....  1532.256123: tegra_channel_set_stream: enable : 0x0
        v4l2-ctl-9610  [000] ....  1532.256126: tegra_channel_set_stream: jbar0233 2-0010 : 0x0
        v4l2-ctl-9610  [000] ....  1532.256226: tegra_channel_set_stream: 15a00000.nvcsi--6 : 0x0
        v4l2-ctl-9610  [000] ....  1532.256231: csi_s_stream: enable : 0x0
     kworker/0:3-1736  [000] ....  1532.268704: rtcpu_vinotify_error: tstamp:48019281702 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48019280997 data:0x00000489
     kworker/0:3-1736  [000] ....  1532.268709: rtcpu_vinotify_event: tstamp:48019282031 tag:FS channel:0x01 frame:0 vi_tstamp:48019251281 data:0x00000014
     kworker/0:3-1736  [000] ....  1532.268711: rtcpu_vinotify_event: tstamp:48019282180 tag:CHANSEL_NOMATCH channel:0x50 frame:0 vi_tstamp:48019280997 data:0x00000489
     kworker/0:3-1736  [000] ....  1532.268716: rtos_queue_send_from_isr_failed: tstamp:48019780422 queue:0x0bcb41f8
     kworker/0:3-1736  [000] ....  1532.268747: rtos_queue_send_from_isr_failed: tstamp:48019780585 queue:0x0bcb8a60
     kworker/0:3-1736  [000] ....  1532.268748: rtos_queue_send_from_isr_failed: tstamp:48019780747 queue:0x0bcba5e0
     kworker/0:3-1736  [000] ....  1532.268758: rtos_queue_send_from_isr_failed: tstamp:48019780904 queue:0x0bcbb3a0
     kworker/0:3-1736  [000] ....  1532.268760: rtos_queue_send_from_isr_failed: tstamp:48019781060 queue:0x0bcbc160
     kworker/0:3-1736  [000] ....  1532.268761: rtos_queue_send_from_isr_failed: tstamp:48020110703 queue:0x0bcb41f8
     kworker/0:3-1736  [000] ....  1532.268762: rtos_queue_send_from_isr_failed: tstamp:48020110922 queue:0x0bcb8a60
     kworker/0:3-1736  [000] ....  1532.268764: rtos_queue_send_from_isr_failed: tstamp:48020111144 queue:0x0bcba5e0
     kworker/0:3-1736  [000] ....  1532.268765: rtos_queue_send_from_isr_failed: tstamp:48020111363 queue:0x0bcbb3a0
     kworker/0:3-1736  [000] ....  1532.268766: rtos_queue_send_from_isr_failed: tstamp:48020111581 queue:0x0bcbc160
        v4l2-ctl-9610  [000] ....  1532.273160: tegra_channel_set_power: jbar0233 2-0010 : 0x0
        v4l2-ctl-9610  [000] ....  1532.273194: camera_common_s_power: status : 0x0
        v4l2-ctl-9610  [000] ....  1532.273360: tegra_channel_set_power: 15a00000.nvcsi--6 : 0x0
        v4l2-ctl-9610  [000] ....  1532.273367: csi_s_power: enable : 0x0
     kworker/0:3-1736  [000] ....  1532.370743: rtos_queue_peek_from_isr_failed: tstamp:48023010195 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1532.523749: rtos_queue_peek_from_isr_failed: tstamp:48028010192 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1532.727695: rtos_queue_peek_from_isr_failed: tstamp:48033010196 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1532.880680: rtos_queue_peek_from_isr_failed: tstamp:48038010196 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1533.033693: rtos_queue_peek_from_isr_failed: tstamp:48043010192 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1533.186691: rtos_queue_peek_from_isr_failed: tstamp:48048010235 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1533.339699: rtos_queue_peek_from_isr_failed: tstamp:48053010238 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1533.492704: rtos_queue_peek_from_isr_failed: tstamp:48058010196 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1533.645691: rtos_queue_peek_from_isr_failed: tstamp:48063010196 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1533.849706: rtos_queue_peek_from_isr_failed: tstamp:48068010192 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1534.002803: rtos_queue_peek_from_isr_failed: tstamp:48073010192 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1534.155762: rtos_queue_peek_from_isr_failed: tstamp:48078010174 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1534.308704: rtos_queue_peek_from_isr_failed: tstamp:48083010192 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1534.461701: rtos_queue_peek_from_isr_failed: tstamp:48088010199 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1534.614759: rtos_queue_peek_from_isr_failed: tstamp:48093010192 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1534.767702: rtos_queue_peek_from_isr_failed: tstamp:48098010192 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1534.920686: rtos_queue_peek_from_isr_failed: tstamp:48103010196 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1535.124697: rtos_queue_peek_from_isr_failed: tstamp:48108010192 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1535.277727: rtos_queue_peek_from_isr_failed: tstamp:48113010194 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1535.430690: rtos_queue_peek_from_isr_failed: tstamp:48118010196 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1535.583690: rtos_queue_peek_from_isr_failed: tstamp:48123010236 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1535.736683: rtos_queue_peek_from_isr_failed: tstamp:48128010234 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1535.889710: rtos_queue_peek_from_isr_failed: tstamp:48133010194 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1536.042711: rtos_queue_peek_from_isr_failed: tstamp:48138010196 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1536.246746: rtos_queue_peek_from_isr_failed: tstamp:48143010192 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1536.399702: rtos_queue_peek_from_isr_failed: tstamp:48148010199 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1536.552732: rtos_queue_peek_from_isr_failed: tstamp:48153010194 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1536.705682: rtos_queue_peek_from_isr_failed: tstamp:48158010196 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1536.858688: rtos_queue_peek_from_isr_failed: tstamp:48163010236 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1537.011761: rtos_queue_peek_from_isr_failed: tstamp:48168010165 queue:0x0bcbcf78
     kworker/0:2-9517  [000] ....  1537.164698: rtos_queue_peek_from_isr_failed: tstamp:48173010194 queue:0x0bcbcf78
     kworker/0:3-1736  [000] ....  1537.317704: rtos_queue_peek_from_isr_failed: tstamp:48176420774 queue:0x0bcbcf78

Thanks!

Not sure how relevant it is, but in the “good” trace, I see FS for channels 0 & 1 followed closely by FE for both. In the “bad” trace, I see FS for channels 0 & 1, but then the next FE for channel 0 doesn’t appear for a couple of seconds. Not sure what is up with that or if it is completely relevant. I’m also seeing a much larger number of FS/FE for channel 1 in the bad trace.

Channel 0 from ‘good’ trace:

tescott@linux-black-beast:~/troubleshoot$ grep "channel:0x00" trace_good.txt 
     kworker/0:1-8792  [000] ....  1557.386701: rtcpu_vinotify_event: tstamp:48803570441 tag:FS channel:0x00 frame:0 vi_tstamp:48803318477 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.386702: rtcpu_vinotify_event: tstamp:48803570612 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:48803318477 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.386748: rtcpu_vinotify_event: tstamp:48804024488 tag:FE channel:0x00 frame:0 vi_tstamp:48804023229 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.386752: rtcpu_vinotify_event: tstamp:48804025003 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:48804023229 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.488724: rtcpu_vinotify_event: tstamp:48806768163 tag:FS channel:0x00 frame:0 vi_tstamp:48806500999 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.488725: rtcpu_vinotify_event: tstamp:48806768316 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:48806500999 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.488762: rtcpu_vinotify_event: tstamp:48807274457 tag:FE channel:0x00 frame:0 vi_tstamp:48807273067 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.488763: rtcpu_vinotify_event: tstamp:48807274628 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:48807273068 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.590692: rtcpu_vinotify_event: tstamp:48810000366 tag:FS channel:0x00 frame:0 vi_tstamp:48809751288 data:0x00000014
     kworker/0:1-8792  [000] ....  1557.590697: rtcpu_vinotify_event: tstamp:48810000571 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:48809751289 data:0x00000000
     kworker/0:1-8792  [000] ....  1557.590712: rtcpu_vinotify_event: tstamp:48810525139 tag:FE channel:0x00 frame:0 vi_tstamp:48810523617 data:0x00000024
     kworker/0:1-8792  [000] ....  1557.590713: rtcpu_vinotify_event: tstamp:48810525306 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:48810523617 data:0x00000000

Channel 0 from ‘bad’ trace:

tescott@linux-black-beast:~/troubleshoot$ grep "channel:0x00" trace_bad.txt 
     kworker/0:3-1736  [000] ....  1527.054716: rtcpu_vinotify_event: tstamp:47857161429 tag:FS channel:0x00 frame:0 vi_tstamp:47856818701 data:0x00000014
     kworker/0:3-1736  [000] ....  1527.054717: rtcpu_vinotify_event: tstamp:47857161604 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:47856818701 data:0x00000000
     kworker/0:2-9517  [000] ....  1529.721870: rtcpu_vinotify_event: tstamp:47939875968 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:47939781384 data:0x00000000
     kworker/0:2-9517  [000] ....  1529.721871: rtcpu_vinotify_event: tstamp:47939876117 tag:FE channel:0x00 frame:0 vi_tstamp:47939874962 data:0x00000024
     kworker/0:2-9517  [000] ....  1529.772756: rtcpu_vinotify_error: tstamp:47941281783 tag:CSIMUX_STREAM channel:0x00 frame:0 vi_tstamp:47941280673 data:0x00010000
     kworker/0:2-9517  [000] ....  1529.772761: rtcpu_vinotify_event: tstamp:47941580205 tag:CSIMUX_STREAM channel:0x00 frame:0 vi_tstamp:47941280673 data:0x00010000

FYI–

When this error occurs, I’m seeing that my serializer is indicating CSI problems. I’ve seen early packet termination as well as ECC errors when it crops up. It’s looking like it may be an issue upstream – potentially originating at the camera sensor.

I’ll keep you posted.

Thanks,
–tim

I’m convinced this is an issue upstream with our video pipeline. If anything changes or I have updates, I’ll post them. Otherwise, I’ll consider this issue resolved.

–tim