Problem with camera in CSI G port on xavier NX

Hi all,

We ran into strange problem with camera in CSI G port on Xavier NX. Our carrier board is designed in 6x2Lines CSI camera configuration. All these ports are tested with imx219/imx477 cameras and there is no problem (we are able to access streams from cameras and display them on hdmi screen), so everything seems OK on the hardware side. Now we are trying to configure all ports to ov9281 for some specific application. We wrote device tree exactly according to working one for imx219. In 5 csi ports cameras working as expected without any problems, but in CSI G port we are not able to access video stream. Camera in this port is successfully registered after system start and we are sure that i2c communication works fine, and sensor is powered on.

 

I am enclosing dmesg outputs, kernel trace logs and relevant parts from device tree :

 

dmesg output :

gstreamer launch command:
gst-launch-1.0 v4l2src device="/dev/video0" ! "video/x-raw,width=1280,height=800,format=(string)GRAY8" ! videoconvert ! videoscale ! "video/x-raw,width=1280,height=800" ! xvimagesink sync=false




[   77.914942] tegra194-vi5 15c10000.vi: no reply from camera processor
[   77.915099] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[   77.915272] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[   77.917702] tegra-capture-ivc ivc-bc00000.rtcpu:ivccontrol@3: No callback found for msg id: 0x39
[   77.917917] tegra-capture-ivc ivc-bc00000.rtcpu:ivccontrol@3: No callback found for msg id: 0x41
[   77.918072] tegra-capture-ivc ivc-bc00000.rtcpu:ivccontrol@3: No callback found for msg id: 0x37
[   77.918241] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[   80.474881] tegra194-vi5 15c10000.vi: no reply from camera processor
[   80.475052] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[   80.475207] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[   80.478432] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[   83.034893] tegra194-vi5 15c10000.vi: no reply from camera processor
[   83.035064] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[   83.035223] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[   83.038451] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[   85.594892] tegra194-vi5 15c10000.vi: no reply from camera processor
[   85.595076] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms

kernel trace logs:

SETUP :
------------------------------------------------------------------------------------------------
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace

echo file vi2_fops.c +p > /sys/kernel/debug/dynamic_debug/control
echo file csi2_fops.c +p > /sys/kernel/debug/dynamic_debug/control

echo file vi4_fops.c +p > /sys/kernel/debug/dynamic_debug/control
echo file csi.c +p > /sys/kernel/debug/dynamic_debug/control
echo file csi4_fops.c +p > /sys/kernel/debug/dynamic_debug/control
echo file nvcsi.c +p > /sys/kernel/debug/dynamic_debug/control

cat /sys/kernel/debug/tracing/trace



OUTPUT :
--------------------------------------------------------------------------------------------------

root@ubuntu:/home/av_discovery# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 309/309   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
  gst-launch-1.0-8670  [000] ....   659.147708: tegra_channel_open: vi-output, ov9281 35-0060
  gst-launch-1.0-8670  [000] ....   659.149053: tegra_channel_set_power: ov9281 35-0060 : 0x1
  gst-launch-1.0-8670  [000] ....   659.149063: camera_common_s_power: status : 0x1
  gst-launch-1.0-8670  [000] ....   659.172484: tegra_channel_set_power: 15a00000.nvcsi--1 : 0x1
  gst-launch-1.0-8670  [000] ....   659.172490: csi_s_power: enable : 0x1
  gst-launch-1.0-8670  [000] ....   659.174290: tegra_channel_close: vi-output, ov9281 35-0060
  gst-launch-1.0-8670  [000] ....   659.174297: tegra_channel_set_power: ov9281 35-0060 : 0x0
  gst-launch-1.0-8670  [000] ....   659.174305: camera_common_s_power: status : 0x0
  gst-launch-1.0-8670  [000] ....   659.174638: tegra_channel_set_power: 15a00000.nvcsi--1 : 0x0
  gst-launch-1.0-8670  [000] ....   659.174642: csi_s_power: enable : 0x0
  gst-launch-1.0-8670  [000] ....   659.216443: tegra_channel_open: vi-output, ov9281 35-0060
  gst-launch-1.0-8670  [000] ....   659.216459: tegra_channel_set_power: ov9281 35-0060 : 0x1
  gst-launch-1.0-8670  [000] ....   659.216469: camera_common_s_power: status : 0x1
  gst-launch-1.0-8670  [000] ....   659.239930: tegra_channel_set_power: 15a00000.nvcsi--1 : 0x1
  gst-launch-1.0-8670  [000] ....   659.239942: csi_s_power: enable : 0x1
    v4l2src0:src-8672  [002] ....   659.252120: tegra_channel_capture_setup: vnc_id 0 W 1280 H 800 fmt 5
    v4l2src0:src-8672  [000] ....   659.257223: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-8672  [002] ....   659.260915: tegra_channel_set_stream: 15a00000.nvcsi--1 : 0x1
    v4l2src0:src-8672  [002] ....   659.260919: csi_s_stream: enable : 0x1
    v4l2src0:src-8672  [002] ....   659.260953: tegra_channel_set_stream: ov9281 35-0060 : 0x1
     kworker/0:4-5982  [000] ....   659.278024: rtos_queue_peek_from_isr_failed: tstamp:20966701432 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   659.278030: rtcpu_start: tstamp:20966704194
     kworker/0:4-5982  [000] ....   659.278032: rtos_queue_send_from_isr_failed: tstamp:20966715611 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   659.278033: rtos_queue_send_from_isr_failed: tstamp:20966715763 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   659.278034: rtos_queue_send_from_isr_failed: tstamp:20966715912 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   659.278035: rtos_queue_send_from_isr_failed: tstamp:20966716057 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   659.278035: rtos_queue_send_from_isr_failed: tstamp:20966716205 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   659.278037: rtcpu_string: tstamp:20966716652 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:4-5982  [000] ....   659.278081: rtcpu_string: tstamp:20966716830 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/0:4-5982  [000] ....   659.278109: rtcpu_string: tstamp:20966717084 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/0:4-5982  [000] ....   659.278116: rtcpu_string: tstamp:20966717351 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/0:4-5982  [000] ....   659.278122: rtcpu_string: tstamp:20966717584 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/0:4-5982  [000] ....   659.278127: rtcpu_string: tstamp:20966717817 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/0:4-5982  [000] ....   659.278132: rtcpu_string: tstamp:20966718049 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/0:4-5982  [000] ....   659.278138: rtcpu_string: tstamp:20966726764 id:0x04010000 str:"vi5_hwinit: firmware CL2018101701 protocol vers"
     kworker/0:4-5982  [000] ....   659.278139: rtcpu_string: tstamp:20966726881 id:0x04010000 str:"on 2.2
"
     kworker/0:4-5982  [000] ....   659.278145: rtos_queue_send_from_isr_failed: tstamp:20966740325 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   659.278146: rtos_queue_send_from_isr_failed: tstamp:20966740495 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   659.278146: rtos_queue_send_from_isr_failed: tstamp:20966740650 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   659.278147: rtos_queue_send_from_isr_failed: tstamp:20966740795 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   659.278148: rtos_queue_send_from_isr_failed: tstamp:20966740941 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   659.278149: rtcpu_string: tstamp:20966741806 id:0x04010000 str:"VI GOS[#0] set to VM GOS[4] base 0xc2104000
"
     kworker/0:4-5982  [000] ....   659.278154: rtos_queue_send_from_isr_failed: tstamp:20966858347 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   659.278174: rtos_queue_send_from_isr_failed: tstamp:20966858496 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   659.278175: rtos_queue_send_from_isr_failed: tstamp:20966858648 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   659.278176: rtos_queue_send_from_isr_failed: tstamp:20966858793 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   659.278177: rtos_queue_send_from_isr_failed: tstamp:20966858939 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   659.278179: rtcpu_vinotify_event: tstamp:20967237996 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:22395714848 data:0x10000000
     kworker/0:4-5982  [000] ....   659.278180: rtcpu_vinotify_event: tstamp:20967238142 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:22395723232 data:0x31000001
     kworker/0:4-5982  [000] ....   659.278181: rtcpu_vinotify_event: tstamp:20967238306 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:22395725376 data:0x07020001
     kworker/0:4-5982  [000] ....   659.278181: rtcpu_vinotify_event: tstamp:20967238443 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:22395759232 data:0x10000000
     kworker/0:4-5982  [000] ....   659.278182: rtcpu_vinotify_event: tstamp:20967238600 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:22395767648 data:0x31000002
     kworker/0:4-5982  [000] ....   659.446055: rtos_queue_peek_from_isr_failed: tstamp:20971701795 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   659.613991: rtos_queue_peek_from_isr_failed: tstamp:20976701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   659.781994: rtos_queue_peek_from_isr_failed: tstamp:20981701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   659.949974: rtos_queue_peek_from_isr_failed: tstamp:20986701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   660.061976: rtos_queue_peek_from_isr_failed: tstamp:20991701795 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   660.230048: rtos_queue_peek_from_isr_failed: tstamp:20996701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   660.397976: rtos_queue_peek_from_isr_failed: tstamp:21001701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   660.565978: rtos_queue_peek_from_isr_failed: tstamp:21006701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   660.733978: rtos_queue_peek_from_isr_failed: tstamp:21011701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   660.901981: rtos_queue_peek_from_isr_failed: tstamp:21016701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   661.069982: rtos_queue_peek_from_isr_failed: tstamp:21021701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   661.182066: rtos_queue_peek_from_isr_failed: tstamp:21026701790 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   661.350007: rtos_queue_peek_from_isr_failed: tstamp:21031701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   661.517974: rtos_queue_peek_from_isr_failed: tstamp:21036701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   661.685965: rtos_queue_peek_from_isr_failed: tstamp:21041701793 queue:0x0bcbcf78
 vi-output, ov92-8675  [000] ....   661.848342: tegra_channel_capture_setup: vnc_id 0 W 1280 H 800 fmt 5
     kworker/0:4-5982  [000] ....   661.854034: rtos_queue_peek_from_isr_failed: tstamp:21046701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   661.854040: rtos_queue_send_from_isr_failed: tstamp:21047655637 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   661.854042: rtos_queue_send_from_isr_failed: tstamp:21047655789 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   661.854043: rtos_queue_send_from_isr_failed: tstamp:21047655939 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   661.854044: rtos_queue_send_from_isr_failed: tstamp:21047656087 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   661.854045: rtos_queue_send_from_isr_failed: tstamp:21047656232 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   661.854046: rtos_queue_send_from_isr_failed: tstamp:21047709579 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   661.854048: rtos_queue_send_from_isr_failed: tstamp:21047709728 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   661.854049: rtos_queue_send_from_isr_failed: tstamp:21047709879 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   661.854050: rtos_queue_send_from_isr_failed: tstamp:21047710026 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   661.854103: rtos_queue_send_from_isr_failed: tstamp:21047710173 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   661.854106: rtcpu_string: tstamp:21047716994 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:4-5982  [000] ....   661.854138: rtcpu_string: tstamp:21047717175 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/0:4-5982  [000] ....   661.854148: rtcpu_string: tstamp:21047717431 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/0:4-5982  [000] ....   661.854157: rtcpu_string: tstamp:21047717696 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/0:4-5982  [000] ....   661.854165: rtcpu_string: tstamp:21047717932 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/0:4-5982  [000] ....   661.854173: rtcpu_string: tstamp:21047718164 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/0:4-5982  [000] ....   661.854180: rtcpu_string: tstamp:21047718398 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/0:4-5982  [000] ....   661.909983: rtos_queue_send_from_isr_failed: tstamp:21048048020 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   661.909989: rtos_queue_send_from_isr_failed: tstamp:21048048173 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   661.909990: rtos_queue_send_from_isr_failed: tstamp:21048048325 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   661.909992: rtos_queue_send_from_isr_failed: tstamp:21048048472 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   661.909993: rtos_queue_send_from_isr_failed: tstamp:21048048617 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   661.909996: rtcpu_vinotify_event: tstamp:21048076371 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:20702644384 data:0x10000000
     kworker/0:4-5982  [000] ....   661.909997: rtcpu_vinotify_event: tstamp:21048076538 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:20702673760 data:0x31000001
     kworker/0:4-5982  [000] ....   661.909998: rtcpu_vinotify_event: tstamp:21048076678 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:20702682336 data:0x07020001
     kworker/0:4-5982  [000] ....   661.910000: rtos_queue_send_from_isr_failed: tstamp:21048379781 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   661.910001: rtos_queue_send_from_isr_failed: tstamp:21048379934 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   661.910002: rtos_queue_send_from_isr_failed: tstamp:21048380085 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   661.910004: rtos_queue_send_from_isr_failed: tstamp:21048380231 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   661.910005: rtos_queue_send_from_isr_failed: tstamp:21048380379 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   661.910006: rtcpu_vinotify_event: tstamp:21048618522 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:20713245216 data:0x10000000
     kworker/0:4-5982  [000] ....   661.910007: rtcpu_vinotify_event: tstamp:21048618665 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:20713274912 data:0x31000002
     kworker/0:4-5982  [000] ....   662.021965: rtos_queue_peek_from_isr_failed: tstamp:21051701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   662.189999: rtos_queue_peek_from_isr_failed: tstamp:21056701794 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   662.301960: rtos_queue_peek_from_isr_failed: tstamp:21061701794 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   662.469964: rtos_queue_peek_from_isr_failed: tstamp:21066701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   662.637962: rtos_queue_peek_from_isr_failed: tstamp:21071701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   662.805966: rtos_queue_peek_from_isr_failed: tstamp:21076701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   662.973961: rtos_queue_peek_from_isr_failed: tstamp:21081701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   663.141964: rtos_queue_peek_from_isr_failed: tstamp:21086701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   663.309964: rtos_queue_peek_from_isr_failed: tstamp:21091701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   663.421966: rtos_queue_peek_from_isr_failed: tstamp:21096701796 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   663.589970: rtos_queue_peek_from_isr_failed: tstamp:21101701790 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   663.757965: rtos_queue_peek_from_isr_failed: tstamp:21106701790 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   663.929962: rtos_queue_peek_from_isr_failed: tstamp:21111701790 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   664.097966: rtos_queue_peek_from_isr_failed: tstamp:21116701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   664.265962: rtos_queue_peek_from_isr_failed: tstamp:21121701794 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   664.377985: rtos_queue_peek_from_isr_failed: tstamp:21126701791 queue:0x0bcbcf78
 vi-output, ov92-8675  [000] ....   664.408093: tegra_channel_capture_setup: vnc_id 0 W 1280 H 800 fmt 5
     kworker/0:2-8612  [000] ....   664.434331: rtos_queue_send_from_isr_failed: tstamp:21127657270 queue:0x0bcb41f8
     kworker/0:2-8612  [000] ....   664.434337: rtos_queue_send_from_isr_failed: tstamp:21127657421 queue:0x0bcb8a60
     kworker/0:2-8612  [000] ....   664.434338: rtos_queue_send_from_isr_failed: tstamp:21127657574 queue:0x0bcba5e0
     kworker/0:2-8612  [000] ....   664.434339: rtos_queue_send_from_isr_failed: tstamp:21127657721 queue:0x0bcbb3a0
     kworker/0:2-8612  [000] ....   664.434340: rtos_queue_send_from_isr_failed: tstamp:21127657867 queue:0x0bcbc160
     kworker/0:2-8612  [000] ....   664.434342: rtos_queue_send_from_isr_failed: tstamp:21127703767 queue:0x0bcb41f8
     kworker/0:2-8612  [000] ....   664.434343: rtos_queue_send_from_isr_failed: tstamp:21127703919 queue:0x0bcb8a60
     kworker/0:2-8612  [000] ....   664.434344: rtos_queue_send_from_isr_failed: tstamp:21127704070 queue:0x0bcba5e0
     kworker/0:2-8612  [000] ....   664.434345: rtos_queue_send_from_isr_failed: tstamp:21127704217 queue:0x0bcbb3a0
     kworker/0:2-8612  [000] ....   664.434346: rtos_queue_send_from_isr_failed: tstamp:21127704365 queue:0x0bcbc160
     kworker/0:2-8612  [000] ....   664.434349: rtcpu_string: tstamp:21127711042 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:2-8612  [000] ....   664.434393: rtcpu_string: tstamp:21127711222 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/0:2-8612  [000] ....   664.434410: rtcpu_string: tstamp:21127711476 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/0:2-8612  [000] ....   664.434425: rtcpu_string: tstamp:21127711741 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/0:2-8612  [000] ....   664.434440: rtcpu_string: tstamp:21127711975 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/0:2-8612  [000] ....   664.434454: rtcpu_string: tstamp:21127712207 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/0:2-8612  [000] ....   664.434502: rtcpu_string: tstamp:21127712440 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/0:2-8612  [000] ....   664.434518: rtos_queue_send_from_isr_failed: tstamp:21127807567 queue:0x0bcb41f8
     kworker/0:2-8612  [000] ....   664.434519: rtos_queue_send_from_isr_failed: tstamp:21127807719 queue:0x0bcb8a60
     kworker/0:2-8612  [000] ....   664.434520: rtos_queue_send_from_isr_failed: tstamp:21127807871 queue:0x0bcba5e0
     kworker/0:2-8612  [000] ....   664.434522: rtos_queue_send_from_isr_failed: tstamp:21127808019 queue:0x0bcbb3a0
     kworker/0:2-8612  [000] ....   664.434523: rtos_queue_send_from_isr_failed: tstamp:21127808163 queue:0x0bcbc160
     kworker/0:2-8612  [000] ....   664.434525: rtcpu_vinotify_event: tstamp:21127829561 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:23254952800 data:0x10000000
     kworker/0:2-8612  [000] ....   664.434526: rtcpu_vinotify_event: tstamp:21127829705 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:23254982176 data:0x31000001
     kworker/0:2-8612  [000] ....   664.434528: rtcpu_vinotify_event: tstamp:21127829867 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:23254990816 data:0x07020001
     kworker/0:2-8612  [000] ....   664.434529: rtos_queue_send_from_isr_failed: tstamp:21128028618 queue:0x0bcb41f8
     kworker/0:2-8612  [000] ....   664.434530: rtos_queue_send_from_isr_failed: tstamp:21128028767 queue:0x0bcb8a60
     kworker/0:2-8612  [000] ....   664.434531: rtos_queue_send_from_isr_failed: tstamp:21128028919 queue:0x0bcba5e0
     kworker/0:2-8612  [000] ....   664.434533: rtos_queue_send_from_isr_failed: tstamp:21128029064 queue:0x0bcbb3a0
     kworker/0:2-8612  [000] ....   664.434534: rtos_queue_send_from_isr_failed: tstamp:21128029211 queue:0x0bcbc160
     kworker/0:2-8612  [000] ....   664.434535: rtcpu_vinotify_event: tstamp:21128371804 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:23262023808 data:0x10000000
     kworker/0:2-8612  [000] ....   664.434536: rtcpu_vinotify_event: tstamp:21128371969 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:23262053536 data:0x31000002
     kworker/0:2-8612  [000] ....   664.545971: rtos_queue_peek_from_isr_failed: tstamp:21131701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   664.713967: rtos_queue_peek_from_isr_failed: tstamp:21136701795 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   664.881960: rtos_queue_peek_from_isr_failed: tstamp:21141701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   665.049963: rtos_queue_peek_from_isr_failed: tstamp:21146701795 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   665.217979: rtos_queue_peek_from_isr_failed: tstamp:21151701755 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   665.385969: rtos_queue_peek_from_isr_failed: tstamp:21156701793 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   665.497966: rtos_queue_peek_from_isr_failed: tstamp:21161701793 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   665.665964: rtos_queue_peek_from_isr_failed: tstamp:21166701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   665.833967: rtos_queue_peek_from_isr_failed: tstamp:21171701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   666.001967: rtos_queue_peek_from_isr_failed: tstamp:21176701793 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   666.169964: rtos_queue_peek_from_isr_failed: tstamp:21181701795 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   666.337967: rtos_queue_peek_from_isr_failed: tstamp:21186701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   666.505964: rtos_queue_peek_from_isr_failed: tstamp:21191701793 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   666.617965: rtos_queue_peek_from_isr_failed: tstamp:21196701795 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   666.785970: rtos_queue_peek_from_isr_failed: tstamp:21201701804 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   666.953973: rtos_queue_peek_from_isr_failed: tstamp:21206701793 queue:0x0bcbcf78
 vi-output, ov92-8675  [000] ....   666.968077: tegra_channel_capture_setup: vnc_id 0 W 1280 H 800 fmt 5
     kworker/0:2-8612  [000] ....   667.009995: rtos_queue_send_from_isr_failed: tstamp:21207654044 queue:0x0bcb41f8
     kworker/0:2-8612  [000] ....   667.010001: rtos_queue_send_from_isr_failed: tstamp:21207654198 queue:0x0bcb8a60
     kworker/0:2-8612  [000] ....   667.010002: rtos_queue_send_from_isr_failed: tstamp:21207654350 queue:0x0bcba5e0
     kworker/0:2-8612  [000] ....   667.010004: rtos_queue_send_from_isr_failed: tstamp:21207654497 queue:0x0bcbb3a0
     kworker/0:2-8612  [000] ....   667.010005: rtos_queue_send_from_isr_failed: tstamp:21207654645 queue:0x0bcbc160
     kworker/0:2-8612  [000] ....   667.010006: rtos_queue_send_from_isr_failed: tstamp:21207703294 queue:0x0bcb41f8
     kworker/0:2-8612  [000] ....   667.010007: rtos_queue_send_from_isr_failed: tstamp:21207703443 queue:0x0bcb8a60
     kworker/0:2-8612  [000] ....   667.010009: rtos_queue_send_from_isr_failed: tstamp:21207703595 queue:0x0bcba5e0
     kworker/0:2-8612  [000] ....   667.010010: rtos_queue_send_from_isr_failed: tstamp:21207703741 queue:0x0bcbb3a0
     kworker/0:2-8612  [000] ....   667.010011: rtos_queue_send_from_isr_failed: tstamp:21207703888 queue:0x0bcbc160
     kworker/0:2-8612  [000] ....   667.010014: rtcpu_nvcsi_intr: tstamp:21207710159 class:GLOBAL type:PHY_INTR0 phy:3 cil:0 st:0 vc:0 status:0x00000010
     kworker/0:2-8612  [000] ....   667.010016: rtcpu_nvcsi_intr: tstamp:21207710159 class:CORRECTABLE_ERR type:PHY_INTR phy:3 cil:0 st:0 vc:0 status:0x00000110
     kworker/0:2-8612  [000] ....   667.010017: rtcpu_nvcsi_intr: tstamp:21207710741 class:GLOBAL type:PHY_INTR0 phy:3 cil:0 st:0 vc:0 status:0x00000100
     kworker/0:2-8612  [000] ....   667.010020: rtcpu_string: tstamp:21207711566 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:2-8612  [000] ....   667.010065: rtcpu_string: tstamp:21207711751 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/0:2-8612  [000] ....   667.010084: rtcpu_string: tstamp:21207712005 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/0:2-8612  [000] ....   667.010100: rtcpu_string: tstamp:21207712270 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/0:2-8612  [000] ....   667.010116: rtcpu_string: tstamp:21207712503 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/0:2-8612  [000] ....   667.010131: rtcpu_string: tstamp:21207712734 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/0:2-8612  [000] ....   667.010147: rtcpu_string: tstamp:21207712965 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/0:2-8612  [000] ....   667.010163: rtos_queue_send_from_isr_failed: tstamp:21207811690 queue:0x0bcb41f8
     kworker/0:2-8612  [000] ....   667.010164: rtos_queue_send_from_isr_failed: tstamp:21207811841 queue:0x0bcb8a60
     kworker/0:2-8612  [000] ....   667.010165: rtos_queue_send_from_isr_failed: tstamp:21207811996 queue:0x0bcba5e0
     kworker/0:2-8612  [000] ....   667.010167: rtos_queue_send_from_isr_failed: tstamp:21207812141 queue:0x0bcbb3a0
     kworker/0:2-8612  [000] ....   667.010168: rtos_queue_send_from_isr_failed: tstamp:21207812288 queue:0x0bcbc160
     kworker/0:2-8612  [000] ....   667.010170: rtcpu_vinotify_event: tstamp:21207817853 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:21520099328 data:0x10000000
     kworker/0:2-8612  [000] ....   667.010172: rtos_queue_send_from_isr_failed: tstamp:21208012049 queue:0x0bcb41f8
     kworker/0:2-8612  [000] ....   667.010173: rtos_queue_send_from_isr_failed: tstamp:21208012197 queue:0x0bcb8a60
     kworker/0:2-8612  [000] ....   667.010174: rtos_queue_send_from_isr_failed: tstamp:21208012348 queue:0x0bcba5e0
     kworker/0:2-8612  [000] ....   667.010176: rtos_queue_send_from_isr_failed: tstamp:21208012495 queue:0x0bcbb3a0
     kworker/0:2-8612  [000] ....   667.010177: rtos_queue_send_from_isr_failed: tstamp:21208012643 queue:0x0bcbc160
     kworker/0:2-8612  [000] ....   667.010178: rtcpu_vinotify_event: tstamp:21208342816 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:21520217728 data:0x31000001
     kworker/0:2-8612  [000] ....   667.010180: rtcpu_vinotify_event: tstamp:21208342958 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:21520226304 data:0x07020001
     kworker/0:2-8612  [000] ....   667.010181: rtcpu_vinotify_event: tstamp:21208343119 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:21526522720 data:0x10000000
     kworker/0:2-8612  [000] ....   667.010182: rtcpu_vinotify_event: tstamp:21208343255 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:21526552640 data:0x31000002
     kworker/0:2-8612  [000] ....   667.121975: rtos_queue_peek_from_isr_failed: tstamp:21211701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   667.289976: rtos_queue_peek_from_isr_failed: tstamp:21216701794 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   667.457968: rtos_queue_peek_from_isr_failed: tstamp:21221701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   667.625972: rtos_queue_peek_from_isr_failed: tstamp:21226701790 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   667.737968: rtos_queue_peek_from_isr_failed: tstamp:21231701790 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   667.905972: rtos_queue_peek_from_isr_failed: tstamp:21236701794 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   668.073971: rtos_queue_peek_from_isr_failed: tstamp:21241701790 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   668.241973: rtos_queue_peek_from_isr_failed: tstamp:21246701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   668.409971: rtos_queue_peek_from_isr_failed: tstamp:21251701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   668.577974: rtos_queue_peek_from_isr_failed: tstamp:21256701790 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   668.745974: rtos_queue_peek_from_isr_failed: tstamp:21261701791 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   668.857973: rtos_queue_peek_from_isr_failed: tstamp:21266701794 queue:0x0bcbcf78
     kworker/0:2-8612  [000] ....   669.025975: rtos_queue_peek_from_isr_failed: tstamp:21271701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   669.193985: rtos_queue_peek_from_isr_failed: tstamp:21276702328 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   669.361971: rtos_queue_peek_from_isr_failed: tstamp:21281701791 queue:0x0bcbcf78
 vi-output, ov92-8675  [000] ....   669.528180: tegra_channel_capture_setup: vnc_id 0 W 1280 H 800 fmt 5
     kworker/0:4-5982  [000] ....   669.530030: rtos_queue_peek_from_isr_failed: tstamp:21286701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   669.530034: rtos_queue_send_from_isr_failed: tstamp:21287655439 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   669.530036: rtos_queue_send_from_isr_failed: tstamp:21287655589 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   669.530037: rtos_queue_send_from_isr_failed: tstamp:21287655741 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   669.530038: rtos_queue_send_from_isr_failed: tstamp:21287655886 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   669.530039: rtos_queue_send_from_isr_failed: tstamp:21287656034 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   669.530040: rtos_queue_send_from_isr_failed: tstamp:21287706472 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   669.530042: rtos_queue_send_from_isr_failed: tstamp:21287706621 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   669.530043: rtos_queue_send_from_isr_failed: tstamp:21287706773 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   669.530044: rtos_queue_send_from_isr_failed: tstamp:21287706918 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   669.530045: rtos_queue_send_from_isr_failed: tstamp:21287707064 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   669.530048: rtcpu_string: tstamp:21287713731 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:4-5982  [000] ....   669.530076: rtcpu_string: tstamp:21287713920 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/0:4-5982  [000] ....   669.530093: rtcpu_string: tstamp:21287714177 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/0:4-5982  [000] ....   669.530160: rtcpu_string: tstamp:21287714440 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/0:4-5982  [000] ....   669.530174: rtcpu_string: tstamp:21287714672 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/0:4-5982  [000] ....   669.530184: rtcpu_string: tstamp:21287714905 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/0:4-5982  [000] ....   669.530194: rtcpu_string: tstamp:21287715138 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/0:4-5982  [000] ....   669.585986: rtos_queue_send_from_isr_failed: tstamp:21287788365 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   669.585992: rtos_queue_send_from_isr_failed: tstamp:21287788517 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   669.585993: rtos_queue_send_from_isr_failed: tstamp:21287788668 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   669.585994: rtos_queue_send_from_isr_failed: tstamp:21287788815 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   669.585996: rtos_queue_send_from_isr_failed: tstamp:21287788963 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   669.585997: rtos_queue_send_from_isr_failed: tstamp:21287980495 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   669.586027: rtos_queue_send_from_isr_failed: tstamp:21287980645 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   669.586028: rtos_queue_send_from_isr_failed: tstamp:21287980795 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   669.586029: rtos_queue_send_from_isr_failed: tstamp:21287980942 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   669.586030: rtos_queue_send_from_isr_failed: tstamp:21287981087 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   669.586033: rtcpu_vinotify_event: tstamp:21288096196 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:19784393888 data:0x10000000
     kworker/0:4-5982  [000] ....   669.586035: rtcpu_vinotify_event: tstamp:21288096340 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:19784423264 data:0x31000001
     kworker/0:4-5982  [000] ....   669.586036: rtcpu_vinotify_event: tstamp:21288096503 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:19784431872 data:0x07020001
     kworker/0:4-5982  [000] ....   669.586037: rtcpu_vinotify_event: tstamp:21288096638 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:19790538016 data:0x10000000
     kworker/0:4-5982  [000] ....   669.586038: rtcpu_vinotify_event: tstamp:21288096799 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:19790577376 data:0x31000002
     kworker/0:4-5982  [000] ....   669.697982: rtos_queue_peek_from_isr_failed: tstamp:21291701794 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   669.865966: rtos_queue_peek_from_isr_failed: tstamp:21296701795 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   669.977978: rtos_queue_peek_from_isr_failed: tstamp:21301701794 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   670.145964: rtos_queue_peek_from_isr_failed: tstamp:21306701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   670.313965: rtos_queue_peek_from_isr_failed: tstamp:21311701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   670.481963: rtos_queue_peek_from_isr_failed: tstamp:21316701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   670.649973: rtos_queue_peek_from_isr_failed: tstamp:21321701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   670.817968: rtos_queue_peek_from_isr_failed: tstamp:21326701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   670.985970: rtos_queue_peek_from_isr_failed: tstamp:21331701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   671.097970: rtos_queue_peek_from_isr_failed: tstamp:21336701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   671.265968: rtos_queue_peek_from_isr_failed: tstamp:21341701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   671.433969: rtos_queue_peek_from_isr_failed: tstamp:21346701795 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   671.601973: rtos_queue_peek_from_isr_failed: tstamp:21351701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   671.769972: rtos_queue_peek_from_isr_failed: tstamp:21356701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   671.937975: rtos_queue_peek_from_isr_failed: tstamp:21361701793 queue:0x0bcbcf78
 vi-output, ov92-8675  [000] ....   672.088134: tegra_channel_capture_setup: vnc_id 0 W 1280 H 800 fmt 5
     kworker/0:4-5982  [000] ....   672.105983: rtos_queue_peek_from_isr_failed: tstamp:21366701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   672.105989: rtos_queue_send_from_isr_failed: tstamp:21367655928 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   672.105990: rtos_queue_send_from_isr_failed: tstamp:21367656079 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   672.105992: rtos_queue_send_from_isr_failed: tstamp:21367656232 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   672.105993: rtos_queue_send_from_isr_failed: tstamp:21367656380 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   672.105994: rtos_queue_send_from_isr_failed: tstamp:21367656527 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   672.105995: rtos_queue_send_from_isr_failed: tstamp:21367705103 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   672.106002: rtos_queue_send_from_isr_failed: tstamp:21367705253 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   672.106003: rtos_queue_send_from_isr_failed: tstamp:21367705405 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   672.106004: rtos_queue_send_from_isr_failed: tstamp:21367705552 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   672.106005: rtos_queue_send_from_isr_failed: tstamp:21367705698 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   672.106007: rtcpu_nvcsi_intr: tstamp:21367711952 class:GLOBAL type:PHY_INTR0 phy:3 cil:0 st:0 vc:0 status:0x00000110
     kworker/0:4-5982  [000] ....   672.106009: rtcpu_nvcsi_intr: tstamp:21367711952 class:CORRECTABLE_ERR type:PHY_INTR phy:3 cil:0 st:0 vc:0 status:0x00000110
     kworker/0:4-5982  [000] ....   672.106011: rtcpu_string: tstamp:21367712967 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/0:4-5982  [000] ....   672.106045: rtcpu_string: tstamp:21367713149 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/0:4-5982  [000] ....   672.106055: rtcpu_string: tstamp:21367713404 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/0:4-5982  [000] ....   672.106064: rtcpu_string: tstamp:21367713668 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/0:4-5982  [000] ....   672.106073: rtcpu_string: tstamp:21367713900 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/0:4-5982  [000] ....   672.106082: rtcpu_string: tstamp:21367714132 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/0:4-5982  [000] ....   672.106090: rtcpu_string: tstamp:21367714364 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/0:4-5982  [000] ....   672.106098: rtos_queue_send_from_isr_failed: tstamp:21367782832 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   672.106100: rtos_queue_send_from_isr_failed: tstamp:21367782984 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   672.106101: rtos_queue_send_from_isr_failed: tstamp:21367783137 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   672.106102: rtos_queue_send_from_isr_failed: tstamp:21367783284 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   672.106103: rtos_queue_send_from_isr_failed: tstamp:21367783430 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   672.106106: rtcpu_vinotify_event: tstamp:21367849693 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:22344207232 data:0x10000000
     kworker/0:4-5982  [000] ....   672.106107: rtcpu_vinotify_event: tstamp:21367849867 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:22344236608 data:0x31000001
     kworker/0:4-5982  [000] ....   672.106108: rtcpu_vinotify_event: tstamp:21367850006 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:22344245184 data:0x07020001
     kworker/0:4-5982  [000] ....   672.106109: rtos_queue_send_from_isr_failed: tstamp:21367982398 queue:0x0bcb41f8
     kworker/0:4-5982  [000] ....   672.106111: rtos_queue_send_from_isr_failed: tstamp:21367982546 queue:0x0bcb8a60
     kworker/0:4-5982  [000] ....   672.106112: rtos_queue_send_from_isr_failed: tstamp:21367982696 queue:0x0bcba5e0
     kworker/0:4-5982  [000] ....   672.106113: rtos_queue_send_from_isr_failed: tstamp:21367982844 queue:0x0bcbb3a0
     kworker/0:4-5982  [000] ....   672.106114: rtos_queue_send_from_isr_failed: tstamp:21367982989 queue:0x0bcbc160
     kworker/0:4-5982  [000] ....   672.161981: rtcpu_vinotify_event: tstamp:21368392297 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:22350607840 data:0x10000000
     kworker/0:4-5982  [000] ....   672.161987: rtcpu_vinotify_event: tstamp:21368392456 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:22350637568 data:0x31000002
     kworker/0:4-5982  [000] ....   672.218000: rtos_queue_peek_from_isr_failed: tstamp:21371701794 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   672.385977: rtos_queue_peek_from_isr_failed: tstamp:21376701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   672.554017: rtos_queue_peek_from_isr_failed: tstamp:21381701790 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   672.721975: rtos_queue_peek_from_isr_failed: tstamp:21386701793 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   672.889973: rtos_queue_peek_from_isr_failed: tstamp:21391701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   673.057973: rtos_queue_peek_from_isr_failed: tstamp:21396701791 queue:0x0bcbcf78
     kworker/0:4-5982  [000] ....   673.225985: rtos_queue_peek_from_isr_failed: tstamp:21401701794 queue:0x0bcbcf78

device tree


				vi_port5: port@5 {
					status="okay";
					reg = <5>;
					ov9281_vi_in5: endpoint {
						status="okay";
						port-index = <6>;
						bus-width = <2>;
						remote-endpoint = <&ov9281_csi_out5>;
					};
				};	
			
.
.
.
			

          csi_chan5: channel@5 {
				reg = <5>;
				status="okay";
				ports {
					#address-cells = <1>;
					#size-cells = <0>;
					status="okay";
					csi_chan5_port0: port@0 {
						status="okay";
						reg = <0>;
						ov9281_csi_in5: endpoint@10 {
							status="okay";
							port-index = <6>;
							bus-width = <2>;
							remote-endpoint = <&ov9281_out5>;
						};
					};
					csi_chan5_port1: port@1 {
						status="okay";
						reg = <1>;
						ov9281_csi_out5: endpoint@11 {
							status="okay";
							remote-endpoint = <&ov9281_vi_in5>;
						};
					};
				};
			};

.
.
.


       i2c_5: i2c@5 {			
				ov9281_cam5: ov9281_g@60 {
					status="okay";
					compatible = "nvidia,ov9281";
					/* I2C device address */ 
					reg = <0x60>;
					/* V4L2 device node location */
					devnode = "video5";

					/* Physical dimensions of sensor */
                	physical_w = "3.680";
                	physical_h = "2.760";

					sensor_model = "ov9281";
					use_sensor_mode_id = "true";
					fsync = "none"; /*none/slave/master/slave-trigger*/

					mode0 { 						
						mclk_khz = "26000";						
						num_lanes = "2";
						tegra_sinterface = "serial_g";
						phy_mode = "DPHY";
						discontinuous_clk = "no";
						dpcm_enable = "false";
						cil_settletime = "0";

						active_w = "1280";
						active_h = "800";	
						pixel_t = "gray";

						readout_orientation = "90";
						line_length = "1408";											
						inherent_gain = "1";		
						mclk_multiplier = "7.69";						
						pix_clk_hz = "200000000";

						gain_factor = "1";
						framerate_factor = "1";
						exposure_factor = "1";
						min_gain_val = "1"; 
						max_gain_val = "8191"; 
						step_gain_val = "1";
						default_gain = "16"; /* 1.0x real gain */
						min_framerate = "1"; /* ? fps */
						max_framerate = "65535"; 
						step_framerate = "1";
						default_framerate = "64625"; /* 120.0 fps */
						min_exp_time = "1"; 
						max_exp_time = "16777215"; 
						step_exp_time = "1";
						default_exp_time = "10896";

						embedded_metadata_height = "0";
					};					
					ports {
						#address-cells = <1>;
						#size-cells = <0>;
						status="okay";

						port@0 {
							reg = <0>;
							status="okay";
							ov9281_out5: endpoint {
								status="okay";
								port-index = <6>;
								bus-width = <2>;
								remote-endpoint = <&ov9281_csi_in5>;
							};
						};
					};
				};

.
.
.

        num_csi_lanes = <12>;
		max_lane_speed = <800000>;
		min_bits_per_pixel = <8>;
		vi_peak_byte_per_pixel = <2>;
		vi_bw_margin_pct = <25>;
		max_pixel_rate = <200000>;
		isp_peak_byte_per_pixel = <5>;
		isp_bw_margin_pct = <25>;

.
.
.

          cam_module5: module5 {
				badge = "jakku_topleft_AVOV9281";
				position = "topleft";
				orientation = "1";
				status="okay";
				cam_module5_drivernode0: drivernode0 {
					status="okay";
					pcl_id = "v4l2_sensor";
					devname = "ov9281 35-0060";
					proc-device-tree = "/proc/device-tree/i2c@3180000/tca9548@70/i2c@5/ov9281_g@60";
				};
			};

Thanks for any help,

Regards
Tom.

Have a try boost below clocks to try.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate

Hi, here’s the output from your suggested commands:

cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
460800000
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
576000000
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
314000000

Camera behavior is the same as before.

Modify below port-index to 5 in the VI and sensor scope but don’t in NVCSI scope.

				vi_port5: port@5 {
					status="okay";
					reg = <5>;
					ov9281_vi_in5: endpoint {
						status="okay";
						port-index = <6>;
						bus-width = <2>;
						remote-endpoint = <&ov9281_csi_out5>;
					};
				};	
			
..
..
..
						port@0 {
							reg = <0>;
							status="okay";
							ov9281_out5: endpoint {
								status="okay";
								port-index = <6>;
								bus-width = <2>;
								remote-endpoint = <&ov9281_csi_in5>;
							};

Hi, ShaneCCC,

Thank you so much for this tip, it helps and now I can access video stream on CSI G too.

But I have checked twice, and in our working device tree for imx219 are all ports (VI/ NCCSI / sensor) set to 6. Why this configuration works for one camera and not for another ? One difference which I can see is that I’m using two different gstreamer plugins nvarguscamerascr/v4l2src. Could you please refer me to some material which will make it clear for me to the future ?

Regards
Tom

Maybe this can explain it. Due to EF and GH can’t use 2x2 lanes configure so it count as CSI A/B/C/D/EF/GH map to CSI 0/1/2/3/4,5/6 and 5 won’t used. for the VI and sensor it count from 0 - 5 for 6 cameras.