4x 1080p60 V4L2src Capture+Encode gStreamer Performance Issues

I have a custom carrier board for the X2 module we developed which provides 4x 1080p60 YUV 4:2:2 (UYVY) input feeds over MIPI-CSI to the X2 module. Our application requires gStreamer 1.11.0 to be used to interface with the encoding and nvidconv cores. We made a custom compile of gStreamer 1.11.0 using the ridgerun wiki:
http://developer.ridgerun.com/wiki/index.php?title=Compile_gstreamer_on_Jetson_TX1_and_TX2

We took the ov5693 driver and simply edited it remove all register reads/writes to the physical ov5693 part and added UYVY support. So we essentially have a dummy yuv 4:2:2 v4l2src distributing 1080p60 frames. I am not using Jetpack. I am using Linux for Tegra 28.2.1 with no GUI.

A single stream using this gst-launch pipeline works fine:

~/gst_1.11.0/out/bin/gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! omxh264enc bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! mpegtsmux ! udpsink host=224.0.0.2 port=5068

There are no visual anomolies and the input holds to 60fps

Running 2 inputs using the following pipeline results in 60fps for both channels at the beginning, but after 20 seconds both drop to 40fps:

~/gst_1.11.0/out/bin/gst-launch-1.0 \
v4l2src device=/dev/video1 io-mode=2 do-timestamp=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=4 interpolation-method=5  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h264parse ! queue ! mpegtsmux ! queue ! udpsink host=224.0.0.2 port=5068 \
v4l2src device=/dev/video2 io-mode=2 do-timestamp=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=4 interpolation-method=5  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h264parse ! queue ! mpegtsmux ! queue ! udpsink host=224.0.0.2 port=6068

Adding a 3rd results in 20fps.

yavta & v4l2-ctrl both report that our input sensors are running at 60fps, even when i open 3 instances.

I added more output-buffers to the nvidconv element and it actually then started streaming at 60fps, but the video had a bunch of corruption issues. There was a bunch of tearing and occasionally the video would actually skip back a few frames. It is almost as if the buffer of video frames is a ring buffer falling behind, but i dont understand the under workings.

I changed my pipeline to use videoconvert instead of nvidconv, but even with one stream it sits at 30fps. I do not know why as using a videotestsrc at 60 fps, it streams at 60fps.

Lastly i setup a videotestsrc 1920x1080@60 UYVY with nvvidconv and ran 4 of them. As long as i kept the default pattern on the test src, everything streamed perfect. Here is that pipeline:

~/gst_1.11.0/out/bin/gst-launch-1.0 \
videotestsrc is-live=true do-timestamp=true pattern=0 ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=60   ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc  bit-packetization=true bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h264parse ! mpegtsmux ! udpsink host=224.0.0.2 port=5068 \
videotestsrc is-live=true do-timestamp=true pattern=0 ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=60   ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc  bit-packetization=true bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' !  queue ! h264parse !  mpegtsmux ! udpsink host=224.0.0.2 port=6068 \
videotestsrc is-live=true do-timestamp=true pattern=0 ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=60  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc  bit-packetization=true bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h264parse !  mpegtsmux !  udpsink host=224.0.0.2 port=7068 \
videotestsrc is-live=true do-timestamp=true pattern=0 ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=60  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc  bit-packetization=true bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h264parse !  mpegtsmux !  udpsink host=224.0.0.2 port=8068

If i change the pattern i get issues with performance but i believe that is due to the CPU generating these images and is falling behind.

Any ideas? Since videotestsrc is working, that seems to imply the pipeline should work. Is there some issue with my setup and v4l2src interfacing w/ everything else? I believe its using io-mode mmap.

Thanks in advance!

All

I just noticed that these visual anomalies do still occur on a single input of my v4l2src feed, they are just less prevalent. With 1 feed it occurs only near the top of the video.

Here are 2 videos to demonstrate the video anomalies i refer to when adding additional output-buffers to nvvidconv. As mentioned, upping the buffer count allows the 60fps to travel through, but i get these stutters.

Single stream
https://vimeo.com/300834186

v4l2src device=/dev/video1 io-mode=2 do-timestamp=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! nvvidconv output-buffers=4 interpolation-method=5  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh265enc bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h265parse ! queue ! mpegtsmux ! queue ! udpsink host=224.0.0.2 port=5068

4x stream
https://vimeo.com/300834176

v4l2src device=/dev/video1 io-mode=2 do-timestamp=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=8 interpolation-method=5  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h264parse ! queue ! mpegtsmux ! queue ! udpsink host=224.0.0.2 port=5068 \
v4l2src device=/dev/video2 io-mode=2 do-timestamp=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=8 interpolation-method=5  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h264parse ! queue ! mpegtsmux ! queue ! udpsink host=224.0.0.2 port=6068 \
v4l2src device=/dev/video3 io-mode=2 do-timestamp=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1" ! queue ! nvvidconv output-buffers=8 interpolation-method=5  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' ! queue ! omxh264enc bitrate=5000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! queue ! h264parse ! queue ! mpegtsmux ! queue ! udpsink host=224.0.0.2 port=7068

The source is our camera feed from our custom UYVY MIPI feed.

Hi,
Please check the frame rate of v4l2 sources via fpsdisplaysink:
https://devtalk.nvidia.com/default/topic/1027772/jetson-tx2/two-csi-cameras-on-tx2/post/5227935/#5227935

So i think im fighting 2 issues here.

  1. Framerate dips below 60 at certain times.
  2. Tearing in video feed when under more load.

FPS ISSUE:
Thanks a ton for the tip on fpsdisplaysink. i didn’t know that was an available element. I don’t have a GUI so i set it to fakesink and piped the signals to gstdebug log.

I first just ran all 4 of my 1080p60 feeds direstly to fpsdisplaysink using the following pipeline:

out/bin/gst-launch-1.0 \
v4l2src device=/dev/video0 io-mode=2 do-timestamp=true typefind=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1"  !   fpsdisplaysink video-sink=fakesink sync=false async=true signal-fps-measurements=true \
v4l2src device=/dev/video1 io-mode=2 do-timestamp=true typefind=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1"  !   fpsdisplaysink video-sink=fakesink sync=false async=true signal-fps-measurements=true \
v4l2src device=/dev/video2 io-mode=2 do-timestamp=true typefind=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1"  !   fpsdisplaysink video-sink=fakesink sync=false async=true signal-fps-measurements=true \
v4l2src device=/dev/video3 io-mode=2 do-timestamp=true typefind=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1"  !   fpsdisplaysink video-sink=fakesink sync=false async=true signal-fps-measurements=true \
--gst-debug=fpsdisplaysink:6

Every single time i go 60fps on all 4 channels without any issues.
Tegrastats reports the following:

RAM 382/7855MB (lfb 1800x4MB) CPU [73%@1996,12%@345,13%@345,75%@1938,66%@1937,64%@1940] EMC_FREQ 6%@1866 GR3D_FREQ 0%@140 APE 150 MTS fg 0% bg 0% BCPU@46.5C MCPU@46.5C GPU@52.5C PLL@46.5C AO@44C Tboard@40C Tdiode@43.75C PMIC@100C thermal@45.5C VDD_IN 5178/5256 VDD_CPU 1293/1382 VDD_GPU 190/190 VDD_SOC 875/875 VDD_WIFI 57/57 VDD_DDR 1473/1469
RAM 381/7855MB (lfb 1800x4MB) CPU [66%@1882,15%@345,12%@345,68%@1881,68%@1881,69%@1881] EMC_FREQ 6%@1866 GR3D_FREQ 0%@140 APE 150 MTS fg 0% bg 0% BCPU@46.5C MCPU@46.5C GPU@52.5C PLL@46.5C AO@44C Tboard@40C Tdiode@43.5C PMIC@100C thermal@45.5C VDD_IN 5178/5250 VDD_CPU 1293/1375 VDD_GPU 190/190 VDD_SOC 875/875 VDD_WIFI 38/55 VDD_DDR 1473/1469

I then decided to put the nvvidconv in between and thats when i discovered something interesting.
The pipeline is as follows:

out/bin/gst-launch-1.0 \
v4l2src device=/dev/video0 io-mode=2 do-timestamp=true typefind=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1"  !  nvvidconv  output-buffers=4  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1'  !  fpsdisplaysink video-sink=fakesink sync=false async=true signal-fps-measurements=true \
v4l2src device=/dev/video1 io-mode=2 do-timestamp=true typefind=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1"  !  nvvidconv  output-buffers=4  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1'  !  fpsdisplaysink video-sink=fakesink sync=false async=true signal-fps-measurements=true \
v4l2src device=/dev/video2 io-mode=2 do-timestamp=true typefind=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1"  !  nvvidconv  output-buffers=4  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1'  !  fpsdisplaysink video-sink=fakesink sync=false async=true signal-fps-measurements=true \
v4l2src device=/dev/video3 io-mode=2 do-timestamp=true typefind=true ! "video/x-raw, width=1920, height=1080,  format=(string)UYVY, framerate=(fraction)60/1"  !  nvvidconv  output-buffers=4  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=60/1' !   fpsdisplaysink video-sink=fakesink sync=false async=true signal-fps-measurements=true \
--gst-debug=fpsdisplaysink:6

Usually i would get 60fps on all 4 channels. Every so often (every 10 iterations or so) i would get 30-40fps for some of the feeds. This would happen immediately upon execution and stay for the whole time. IE it either starts in this low-fps state or perfect fps and then stays there.

Checking tegrastats i saw the following differential between the 2 situations:

all 4 streams @ 60fps:

7/1402 VDD_WIFI 57/47 VDD_DDR 1550/1545
RAM 297/7855MB (lfb 1828x4MB) CPU [57%@1113,0%@345,3%@345,56%@1113,53%@1113,59%@1113] BCPU@48.5C MCPU@48.5C GPU@54.5C PLL@48.5C AO@46.5C Tboard@43C Tdiode@46C PMIC@100C thermal@47.7C VDD_IN 5369/5311 VDD_CPU 722/679 VDD_GPU 190/190 VDD_SOC 1407/1402 VDD_WIFI 57/48 VDD_DDR 1569/1547
RAM 297/7855MB (lfb 1828x4MB) CPU [50%@1113,0%@345,2%@345,53%@1113,48%@1113,60%@1113] BCPU@48.5C MCPU@48.5C GPU@54.5C PLL@48.5C AO@46.5C Tboard@43C Tdiode@46C PMIC@100C thermal@47.7C VDD_IN 5407/5321 VDD_CPU 722/684 VDD_GPU 190/190 VDD_SOC 1407/1403 VDD_WIFI 57/49 VDD_DDR 1588/1551

Low fps situation:

RAM 298/7855MB (lfb 1825x4MB) CPU [38%@499,2%@2035,<b>95%@2035</b>,44%@499,38%@499,54%@499] BCPU@49.5C MCPU@49.5C GPU@55.5C PLL@49.5C AO@47.5C Tboard@44C Tdiode@47.25C PMIC@100C thermal@49C VDD_IN 6201/6198 VDD_CPU 1064/1052 VDD_GPU 190/190 VDD_SOC 1520/1520 VDD_WIFI 57/55 VDD_DDR 1913/1914
RAM 297/7855MB (lfb 1825x4MB) CPU [46%@499,1%@2035,<b>95%@2035</b>,34%@499,49%@499,53%@499] BCPU@49.5C MCPU@49.5C GPU@55.5C PLL@49.5C AO@47.5C Tboard@44C Tdiode@47.25C PMIC@100C thermal@49C VDD_IN 6201/6198 VDD_CPU 1026/1050 VDD_GPU 190/190 VDD_SOC 1520/1520 VDD_WIFI 57/55 VDD_DDR 1913/1914
RAM 297/7855MB (lfb 1825x4MB) CPU [39%@499,1%@2035,<b>96%@2035</b>,53%@499,47%@499,40%@499] BCPU@50C MCPU@50C GPU@56C PLL@50C AO@47.5C Tboard@44C Tdiode@47.25C PMIC@100C thermal@49C VDD_IN 6220/6199 VDD_CPU 1026/1048 VDD_GPU 190/190 VDD_SOC 1520/1520 VDD_WIFI 57/55 VDD_DDR 1913/1914

Note sometimes this high cpu core usage is on core 2 and sometimes on core 3.

I turned off core 2 by using the following setting:

echo 0 > /sys/devices/system/cpu/cpu1/online

And i haven’t had it enter this low fps state again yet. I will play around with some nvpmodel settings to see if i can force those cores to be less utilized or something.

My full streaming pipeline of 4x 1080p60 encodes also keeps up.

So i haven’t done enough testing but it seems like i have a bandaid for now for the low fps issue, but i still have the tearing issue. I need to gather additional debug information. Should i start a separate thread for this or not?

All,

I’ve narrowed down actually what this tearing issue is, but not what is causing it.

It seems the V4L2 driver is being told the frame is ready to read out before the DMA from the mipi core is finished.

IE our MIPI input device hasn’t finished filling the buffer up before V4L2 reads it out.
This results in my first 4 frames (i have 4 V4L2 buffers set up) looking like the following:

since the frame isn’t filled yet and most of it is green (0x00 in YUV colorspace). Though V4L2 finishes copying the data, our MIPI input is still filling the frame out. This results in the circular buffer then wrapping back around to the first buffer (which now has full data). It then repeats with only the partial top being filled in followed by the remenants of the last filled frame.

Meaning most of what I’m viewing is frame data from 4 frames ago.

I see the MIPI protocol has Frame Packet Start & Frame Packet End codes which i assume signal to the V4L2 driver when a frame is ready. Where could that go wrong? Is there a way to askew which buffer is the latest? Might be something on our MIPI input side. we are investigating.

hello x1tester62,

could you please apply kernel patch from Topic 1038067, and try to reproduce the issue again.
thanks

Hey Jerry,

Looks like that may have fixed the tearing issue. Thanks so much!

Can you give any further detail as to what the issue was and why this fixes it?

Jerry,

It fixed the tearing issue, but i am now getting crashes i wasn’t getting before.

After a short period of time (usually under 5 min) i will get the following message and my gstreamer pipeline stops:

[ 2645.195170] tegra-vi4 15700000.vi: Status:  7 channel:00 frame:0000
[ 2645.201459] tegra-vi4 15700000.vi:          timestamp sof 2655822864992 eof 2655838863200 data 0x00000001
[ 2645.211034] tegra-vi4 15700000.vi:          capture_id 171450 stream  0 vchan  0
[ 2646.175698] tegra-vi4 15700000.vi: ATOMP_FE syncpt timeout! err = -11
[ 2646.175709] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 2647.187711] tegra-vi4 15700000.vi: ATOMP_FE syncpt timeout!

Here is the kernel tracing log. Ive truncated it some since i assume the bottom would be the part of relevance. This is running 4x 1080p60 gst-launch capture+encode+stream. Ive posted that pipeline earlier. Same issue happens just going directly to a fakesink with 4x inputs.

I am now testing if it happens with only single input or not. EDIT: it does happen with just a single input. As mentioned before this didn’t happen before the new patch which fixed the tearing.

kworker/1:3-360   [001] ...1   617.126346: rtos_queue_peek_from_isr_failed: tstamp:19616729123 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   617.126350: rtcpu_vinotify_handle_msg: tstamp:19616920146 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2437050524 data:0x28000000
     kworker/1:3-360   [001] ...1   617.126351: rtcpu_vinotify_handle_msg: tstamp:19617064877 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2437194211 data:0x04370002
     kworker/1:3-360   [001] ...1   617.126352: rtcpu_vinotify_handle_msg: tstamp:19617064986 tag:CHANSEL_PXL_EOF channel:0x03 frame:0 vi_tstamp:2437194212 data:0x04370002
     kworker/1:3-360   [001] ...1   617.126353: rtcpu_vinotify_handle_msg: tstamp:19617065086 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2437194212 data:0x04370002
     kworker/1:3-360   [001] ...1   617.126354: rtcpu_vinotify_handle_msg: tstamp:19617065192 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2437194212 data:0x04370002
     kworker/1:3-360   [001] ...1   617.126355: rtcpu_vinotify_handle_msg: tstamp:19617065291 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2437194235 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126356: rtcpu_vinotify_handle_msg: tstamp:19617065415 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:2437194236 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126356: rtcpu_vinotify_handle_msg: tstamp:19617065535 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2437194237 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126357: rtcpu_vinotify_handle_msg: tstamp:19617065650 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2437194237 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126358: rtcpu_vinotify_handle_msg: tstamp:19617085349 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2437215126 data:0x00000001
     kworker/1:3-360   [001] ...1   617.126359: rtcpu_vinotify_handle_msg: tstamp:19617085498 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2437215126 data:0x00000001
     kworker/1:3-360   [001] ...1   617.126360: rtcpu_vinotify_handle_msg: tstamp:19617085653 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2437215127 data:0x00000001
     kworker/1:3-360   [001] ...1   617.126360: rtcpu_vinotify_handle_msg: tstamp:19617085813 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2437215133 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126362: rtcpu_vinotify_handle_msg: tstamp:19617085934 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2437215133 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126362: rtcpu_vinotify_handle_msg: tstamp:19617086040 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2437215133 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126363: rtcpu_vinotify_handle_msg: tstamp:19617088685 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2437219089 data:0x08000000
     kworker/1:3-360   [001] ...1   617.126364: rtcpu_vinotify_handle_msg: tstamp:19617089209 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2437219594 data:0x18000000
     kworker/1:3-360   [001] ...1   617.126365: rtcpu_vinotify_handle_msg: tstamp:19617585422 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2437715049 data:0x04370002
     kworker/1:3-360   [001] ...1   617.126366: rtcpu_vinotify_handle_msg: tstamp:19617585531 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2437715049 data:0x04370002
     kworker/1:3-360   [001] ...1   617.126367: rtcpu_vinotify_handle_msg: tstamp:19617585637 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2437715049 data:0x04370002
     kworker/1:3-360   [001] ...1   617.126370: rtcpu_vinotify_handle_msg: tstamp:19617585741 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2437715072 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126373: rtcpu_vinotify_handle_msg: tstamp:19617585876 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2437715073 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126374: rtcpu_vinotify_handle_msg: tstamp:19617586002 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2437715074 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126375: rtcpu_vinotify_handle_msg: tstamp:19617606184 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2437735963 data:0x00000001
     kworker/1:3-360   [001] ...1   617.126376: rtcpu_vinotify_handle_msg: tstamp:19617606327 tag:CHANSEL_PXL_SOF channel:0x03 frame:0 vi_tstamp:2437735963 data:0x00000001
     kworker/1:3-360   [001] ...1   617.126377: rtcpu_vinotify_handle_msg: tstamp:19617606498 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2437735964 data:0x00000001
     kworker/1:3-360   [001] ...1   617.126378: rtcpu_vinotify_handle_msg: tstamp:19617606638 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2437735971 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126379: rtcpu_vinotify_handle_msg: tstamp:19617606738 tag:ATOMP_FS channel:0x03 frame:0 vi_tstamp:2437735971 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126380: rtcpu_vinotify_handle_msg: tstamp:19617606838 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2437735972 data:0x00000000
     kworker/1:3-360   [001] ...1   617.126381: rtcpu_vinotify_handle_msg: tstamp:19617617866 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2437748244 data:0x08000000
     kworker/1:3-360   [001] ...1   617.126381: rtcpu_vinotify_handle_msg: tstamp:19617622546 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2437752947 data:0x18000000
     kworker/1:3-360   [001] ...1   617.126382: rtcpu_vinotify_handle_msg: tstamp:19617865696 tag:CHANSEL_LOAD_FRAMED channel:0x08 frame:0 vi_tstamp:2437996088 data:0x38000000
     kworker/1:3-360   [001] ...1   617.178285: rtcpu_vinotify_handle_msg: tstamp:19618106273 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2438235887 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178288: rtcpu_vinotify_handle_msg: tstamp:19618106380 tag:CHANSEL_PXL_EOF channel:0x03 frame:0 vi_tstamp:2438235887 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178289: rtcpu_vinotify_handle_msg: tstamp:19618106480 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2438235888 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178290: rtcpu_vinotify_handle_msg: tstamp:19618106579 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2438235910 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178290: rtcpu_vinotify_handle_msg: tstamp:19618106703 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:2438235910 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178291: rtcpu_vinotify_handle_msg: tstamp:19618106821 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2438235911 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178292: rtcpu_vinotify_handle_msg: tstamp:19618127267 tag:CHANSEL_PXL_SOF channel:0x03 frame:0 vi_tstamp:2438256801 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178293: rtcpu_vinotify_handle_msg: tstamp:19618127410 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2438256801 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178294: rtcpu_vinotify_handle_msg: tstamp:19618127568 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2438256801 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178295: rtcpu_vinotify_handle_msg: tstamp:19618127708 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2438256801 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178296: rtcpu_vinotify_handle_msg: tstamp:19618127849 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2438256809 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178296: rtcpu_vinotify_handle_msg: tstamp:19618127949 tag:ATOMP_FS channel:0x03 frame:0 vi_tstamp:2438256809 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178297: rtcpu_vinotify_handle_msg: tstamp:19618128049 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2438256809 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178298: rtcpu_vinotify_handle_msg: tstamp:19618128146 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2438256810 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178299: rtcpu_vinotify_handle_msg: tstamp:19618130121 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2438260530 data:0x08000000
     kworker/1:3-360   [001] ...1   617.178300: rtcpu_vinotify_handle_msg: tstamp:19618130729 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2438261128 data:0x18000000
     kworker/1:3-360   [001] ...1   617.178301: rtcpu_vinotify_handle_msg: tstamp:19618431005 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2438561391 data:0x28000000
     kworker/1:3-360   [001] ...1   617.178302: rtcpu_vinotify_handle_msg: tstamp:19618627419 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2438756724 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178302: rtcpu_vinotify_handle_msg: tstamp:19618627536 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2438756725 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178303: rtcpu_vinotify_handle_msg: tstamp:19618627643 tag:CHANSEL_PXL_EOF channel:0x03 frame:0 vi_tstamp:2438756725 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178304: rtcpu_vinotify_handle_msg: tstamp:19618627763 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2438756725 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178305: rtcpu_vinotify_handle_msg: tstamp:19618627861 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2438756748 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178306: rtcpu_vinotify_handle_msg: tstamp:19618628009 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:2438756748 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178307: rtcpu_vinotify_handle_msg: tstamp:19618628157 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2438756749 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178307: rtcpu_vinotify_handle_msg: tstamp:19618628283 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2438756750 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178308: rtcpu_vinotify_handle_msg: tstamp:19618647869 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2438777639 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178309: rtcpu_vinotify_handle_msg: tstamp:19618648026 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2438777639 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178310: rtcpu_vinotify_handle_msg: tstamp:19618648189 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2438777639 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178311: rtcpu_vinotify_handle_msg: tstamp:19618648338 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2438777646 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178311: rtcpu_vinotify_handle_msg: tstamp:19618648440 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2438777646 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178312: rtcpu_vinotify_handle_msg: tstamp:19618648546 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2438777646 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178313: rtcpu_vinotify_handle_msg: tstamp:19618669153 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2438799559 data:0x08000000
     kworker/1:3-360   [001] ...1   617.178314: rtcpu_vinotify_handle_msg: tstamp:19618669673 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2438800095 data:0x18000000
     kworker/1:3-360   [001] ...1   617.178315: rtcpu_vinotify_handle_msg: tstamp:19619147935 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2439277562 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178316: rtcpu_vinotify_handle_msg: tstamp:19619148044 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2439277563 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178316: rtcpu_vinotify_handle_msg: tstamp:19619148147 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2439277563 data:0x04370002
     kworker/1:3-360   [001] ...1   617.178317: rtcpu_vinotify_handle_msg: tstamp:19619148249 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2439277585 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178318: rtcpu_vinotify_handle_msg: tstamp:19619148389 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2439277587 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178319: rtcpu_vinotify_handle_msg: tstamp:19619148510 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2439277587 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178320: rtcpu_vinotify_handle_msg: tstamp:19619168715 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2439298476 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178320: rtcpu_vinotify_handle_msg: tstamp:19619168862 tag:CHANSEL_PXL_SOF channel:0x03 frame:0 vi_tstamp:2439298477 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178321: rtcpu_vinotify_handle_msg: tstamp:19619169010 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2439298477 data:0x00000001
     kworker/1:3-360   [001] ...1   617.178322: rtcpu_vinotify_handle_msg: tstamp:19619169152 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2439298484 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178323: rtcpu_vinotify_handle_msg: tstamp:19619169251 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2439298484 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178324: rtcpu_vinotify_handle_msg: tstamp:19619169351 tag:ATOMP_FS channel:0x03 frame:0 vi_tstamp:2439298484 data:0x00000000
     kworker/1:3-360   [001] ...1   617.178325: rtcpu_vinotify_handle_msg: tstamp:19619170929 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2439301334 data:0x18000000
     kworker/1:3-360   [001] ...1   617.178326: rtcpu_vinotify_handle_msg: tstamp:19619172589 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2439302999 data:0x08000000
     kworker/1:3-360   [001] ...1   617.178326: rtcpu_vinotify_handle_msg: tstamp:19619396703 tag:CHANSEL_LOAD_FRAMED channel:0x08 frame:0 vi_tstamp:2439527104 data:0x38000000
     kworker/1:3-360   [001] ...1   617.230334: rtcpu_vinotify_handle_msg: tstamp:19619668787 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2439798400 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230337: rtcpu_vinotify_handle_msg: tstamp:19619668894 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2439798400 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230338: rtcpu_vinotify_handle_msg: tstamp:19619668992 tag:CHANSEL_SHORT_FRAME channel:0x08 frame:0 vi_tstamp:2439798421 data:0x00000008
     kworker/1:3-360   [001] ...1   617.230339: rtcpu_vinotify_handle_msg: tstamp:19619669187 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2439798423 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230340: rtcpu_vinotify_handle_msg: tstamp:19619669317 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2439798424 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230341: rtcpu_vinotify_handle_msg: tstamp:19619669435 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:2439798425 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230342: rtcpu_vinotify_handle_msg: tstamp:19619689530 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2439819314 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230343: rtcpu_vinotify_handle_msg: tstamp:19619689673 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2439819314 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230343: rtcpu_vinotify_handle_msg: tstamp:19619689825 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2439819314 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230344: rtcpu_vinotify_handle_msg: tstamp:19619689964 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2439819322 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230345: rtcpu_vinotify_handle_msg: tstamp:19619690063 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2439819322 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230346: rtcpu_vinotify_handle_msg: tstamp:19619690162 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2439819322 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230347: rtcpu_vinotify_handle_msg: tstamp:19619692663 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2439822857 data:0x08000000
     kworker/1:3-360   [001] ...1   617.230348: rtcpu_vinotify_handle_msg: tstamp:19619692758 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2439823139 data:0x18000000
     kworker/1:3-360   [001] ...1   617.230348: rtcpu_vinotify_handle_msg: tstamp:19619905276 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2440035683 data:0x28000000
     kworker/1:3-360   [001] ...1   617.230349: rtcpu_vinotify_handle_msg: tstamp:19620189624 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2440319238 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230350: rtcpu_vinotify_handle_msg: tstamp:19620189734 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2440319238 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230351: rtcpu_vinotify_handle_msg: tstamp:19620189835 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2440319238 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230352: rtcpu_vinotify_handle_msg: tstamp:19620189944 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2440319260 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230353: rtcpu_vinotify_handle_msg: tstamp:19620190066 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2440319261 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230353: rtcpu_vinotify_handle_msg: tstamp:19620190187 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2440319262 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230354: rtcpu_vinotify_handle_msg: tstamp:19620210380 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2440340152 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230355: rtcpu_vinotify_handle_msg: tstamp:19620210525 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2440340152 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230356: rtcpu_vinotify_handle_msg: tstamp:19620210667 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2440340152 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230357: rtcpu_vinotify_handle_msg: tstamp:19620210808 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2440340159 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230357: rtcpu_vinotify_handle_msg: tstamp:19620210908 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2440340160 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230359: rtcpu_vinotify_handle_msg: tstamp:19620211006 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2440340160 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230359: rtcpu_vinotify_handle_msg: tstamp:19620213299 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2440343498 data:0x08000000
     kworker/1:3-360   [001] ...1   617.230360: rtcpu_vinotify_handle_msg: tstamp:19620213393 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2440343708 data:0x18000000
     kworker/1:3-360   [001] ...1   617.230361: rtcpu_vinotify_handle_msg: tstamp:19620670646 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2440801031 data:0x28000000
     kworker/1:3-360   [001] ...1   617.230362: rtcpu_vinotify_handle_msg: tstamp:19620710450 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2440840075 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230363: rtcpu_vinotify_handle_msg: tstamp:19620710554 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2440840076 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230364: rtcpu_vinotify_handle_msg: tstamp:19620710654 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2440840076 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230364: rtcpu_vinotify_handle_msg: tstamp:19620710751 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2440840098 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230365: rtcpu_vinotify_handle_msg: tstamp:19620710874 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2440840099 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230366: rtcpu_vinotify_handle_msg: tstamp:19620711000 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2440840099 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230367: rtcpu_vinotify_handle_msg: tstamp:19620731218 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2440860989 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230368: rtcpu_vinotify_handle_msg: tstamp:19620731363 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2440860990 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230369: rtcpu_vinotify_handle_msg: tstamp:19620731507 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2440860990 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230369: rtcpu_vinotify_handle_msg: tstamp:19620731648 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2440860997 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230370: rtcpu_vinotify_handle_msg: tstamp:19620731747 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2440860997 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230371: rtcpu_vinotify_handle_msg: tstamp:19620731847 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2440860997 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230372: rtcpu_vinotify_handle_msg: tstamp:19620733434 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2440863838 data:0x08000000
     kworker/1:3-360   [001] ...1   617.230373: rtcpu_vinotify_handle_msg: tstamp:19620735401 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2440865807 data:0x18000000
     kworker/1:3-360   [001] ...1   617.230374: rtcpu_vinotify_handle_msg: tstamp:19621231276 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2441360913 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230375: rtcpu_vinotify_handle_msg: tstamp:19621231394 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2441360913 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230375: rtcpu_vinotify_handle_msg: tstamp:19621231495 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2441360913 data:0x04370002
     kworker/1:3-360   [001] ...1   617.230376: rtcpu_vinotify_handle_msg: tstamp:19621231594 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2441360936 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230377: rtcpu_vinotify_handle_msg: tstamp:19621231713 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2441360937 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230378: rtcpu_vinotify_handle_msg: tstamp:19621231831 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2441360937 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230379: rtcpu_vinotify_handle_msg: tstamp:19621251796 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2441381827 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230380: rtcpu_vinotify_handle_msg: tstamp:19621251939 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2441381827 data:0x00000001
     kworker/1:3-360   [001] ...1   617.230381: rtcpu_vinotify_handle_msg: tstamp:19621252087 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2441381834 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230387: rtcpu_vinotify_handle_msg: tstamp:19621252185 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2441381835 data:0x00000000
     kworker/1:3-360   [001] ...1   617.230392: rtcpu_vinotify_handle_msg: tstamp:19621254199 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2441384604 data:0x08000000
     kworker/1:3-360   [001] ...1   617.230393: rtcpu_vinotify_handle_msg: tstamp:19621254506 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2441385005 data:0x18000000
     kworker/1:3-360   [001] ...1   617.282427: rtos_queue_peek_from_isr_failed: tstamp:19621729621 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   617.282444: rtcpu_vinotify_handle_msg: tstamp:19621751817 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2441881750 data:0x04370002
     kworker/1:3-360   [001] ...1   617.282450: rtcpu_vinotify_handle_msg: tstamp:19621751918 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2441881751 data:0x04370002
     kworker/1:3-360   [001] ...1   617.282454: rtcpu_vinotify_handle_msg: tstamp:19621752021 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2441881774 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282457: rtcpu_vinotify_handle_msg: tstamp:19621752149 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2441881775 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282461: rtcpu_vinotify_handle_msg: tstamp:19621772882 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2441902664 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282464: rtcpu_vinotify_handle_msg: tstamp:19621773036 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2441902665 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282467: rtcpu_vinotify_handle_msg: tstamp:19621773183 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2441902665 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282471: rtcpu_vinotify_handle_msg: tstamp:19621773324 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2441902672 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282474: rtcpu_vinotify_handle_msg: tstamp:19621773423 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2441902672 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282477: rtcpu_vinotify_handle_msg: tstamp:19621773522 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2441902672 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282481: rtcpu_vinotify_handle_msg: tstamp:19621775172 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2441905366 data:0x08000000
     kworker/1:3-360   [001] ...1   617.282484: rtcpu_vinotify_handle_msg: tstamp:19621775478 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2441905609 data:0x18000000
     kworker/1:3-360   [001] ...1   617.282487: rtcpu_vinotify_handle_msg: tstamp:19621775572 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2441905981 data:0x28000000
     kworker/1:3-360   [001] ...1   617.282491: rtcpu_vinotify_handle_msg: tstamp:19622272974 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2442402588 data:0x04370002
     kworker/1:3-360   [001] ...1   617.282494: rtcpu_vinotify_handle_msg: tstamp:19622273075 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2442402588 data:0x04370002
     kworker/1:3-360   [001] ...1   617.282497: rtcpu_vinotify_handle_msg: tstamp:19622273175 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2442402588 data:0x04370002
     kworker/1:3-360   [001] ...1   617.282501: rtcpu_vinotify_handle_msg: tstamp:19622273272 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2442402611 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282504: rtcpu_vinotify_handle_msg: tstamp:19622273389 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2442402612 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282507: rtcpu_vinotify_handle_msg: tstamp:19622273507 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2442402613 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282511: rtcpu_vinotify_handle_msg: tstamp:19622293720 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2442423502 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282514: rtcpu_vinotify_handle_msg: tstamp:19622293862 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2442423502 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282517: rtcpu_vinotify_handle_msg: tstamp:19622294005 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2442423502 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282521: rtcpu_vinotify_handle_msg: tstamp:19622294154 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2442423510 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282524: rtcpu_vinotify_handle_msg: tstamp:19622294254 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2442423510 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282528: rtcpu_vinotify_handle_msg: tstamp:19622294354 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2442423510 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282531: rtcpu_vinotify_handle_msg: tstamp:19622295668 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2442425867 data:0x18000000
     kworker/1:3-360   [001] ...1   617.282534: rtcpu_vinotify_handle_msg: tstamp:19622295762 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2442426193 data:0x28000000
     kworker/1:3-360   [001] ...1   617.282538: rtcpu_vinotify_handle_msg: tstamp:19622296611 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2442427017 data:0x08000000
     kworker/1:3-360   [001] ...1   617.282541: rtcpu_vinotify_handle_msg: tstamp:19622793809 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2442923426 data:0x04370002
     kworker/1:3-360   [001] ...1   617.282544: rtcpu_vinotify_handle_msg: tstamp:19622793912 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2442923426 data:0x04370002
     kworker/1:3-360   [001] ...1   617.282548: rtcpu_vinotify_handle_msg: tstamp:19622794012 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2442923427 data:0x04370002
     kworker/1:3-360   [001] ...1   617.282552: rtcpu_vinotify_handle_msg: tstamp:19622794111 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2442923448 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282555: rtcpu_vinotify_handle_msg: tstamp:19622794228 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2442923449 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282558: rtcpu_vinotify_handle_msg: tstamp:19622794345 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2442923450 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282561: rtcpu_vinotify_handle_msg: tstamp:19622814559 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2442944340 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282565: rtcpu_vinotify_handle_msg: tstamp:19622814700 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2442944340 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282568: rtcpu_vinotify_handle_msg: tstamp:19622814841 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2442944340 data:0x00000001
     kworker/1:3-360   [001] ...1   617.282571: rtcpu_vinotify_handle_msg: tstamp:19622814984 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2442944348 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282575: rtcpu_vinotify_handle_msg: tstamp:19622815084 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2442944348 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282578: rtcpu_vinotify_handle_msg: tstamp:19622815182 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2442944348 data:0x00000000
     kworker/1:3-360   [001] ...1   617.282582: rtcpu_vinotify_handle_msg: tstamp:19622816164 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:2442946569 data:0x18000000
     kworker/1:3-360   [001] ...1   617.282585: rtcpu_vinotify_handle_msg: tstamp:19622817301 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:2442947503 data:0x08000000
     kworker/1:3-360   [001] ...1   617.282588: rtcpu_vinotify_handle_msg: tstamp:19622817402 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2442947750 data:0x28000000
     kworker/1:3-360   [001] ...1   617.334358: rtcpu_vinotify_handle_msg: tstamp:19623314652 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2443444264 data:0x04370002
     kworker/1:3-360   [001] ...1   617.334362: rtcpu_vinotify_handle_msg: tstamp:19623314755 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2443444264 data:0x04370002
     kworker/1:3-360   [001] ...1   617.334366: rtcpu_vinotify_handle_msg: tstamp:19623314855 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2443444264 data:0x04370002
     kworker/1:3-360   [001] ...1   617.334369: rtcpu_vinotify_handle_msg: tstamp:19623314952 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2443444287 data:0x00000000
     kworker/1:3-360   [001] ...1   617.334373: rtcpu_vinotify_handle_msg: tstamp:19623315068 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2443444287 data:0x00000000
     kworker/1:3-360   [001] ...1   617.334376: rtcpu_vinotify_handle_msg: tstamp:19623315194 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2443444288 data:0x00000000
     kworker/1:3-360   [001] ...1   617.334379: rtcpu_vinotify_handle_msg: tstamp:19623335394 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:2443465177 data:0x00000001
     kworker/1:3-360   [001] ...1   617.334382: rtcpu_vinotify_handle_msg: tstamp:19623335537 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:2443465178 data:0x00000001
     kworker/1:3-360   [001] ...1   617.334386: rtcpu_vinotify_handle_msg: tstamp:19623335678 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2443465178 data:0x00000001
     kworker/1:3-360   [001] ...1   617.334389: rtcpu_vinotify_handle_msg: tstamp:19623335819 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:2443465185 data:0x00000000
     kworker/1:3-360   [001] ...1   617.334393: rtcpu_vinotify_handle_msg: tstamp:19623335919 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:2443465185 data:0x00000000
     kworker/1:3-360   [001] ...1   617.334396: rtcpu_vinotify_handle_msg: tstamp:19623336019 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2443465185 data:0x00000000
     kworker/1:3-360   [001] ...1   617.334400: rtcpu_vinotify_handle_msg: tstamp:19623835451 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:2443965101 data:0x04370002
     kworker/1:3-360   [001] ...1   617.334403: rtcpu_vinotify_handle_msg: tstamp:19623835552 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:2443965101 data:0x04370002
     kworker/1:3-360   [001] ...1   617.334406: rtcpu_vinotify_handle_msg: tstamp:19623835652 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2443965102 data:0x04370002
     kworker/1:3-360   [001] ...1   617.334410: rtcpu_vinotify_handle_msg: tstamp:19623835751 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:2443965124 data:0x00000000
     kworker/1:3-360   [001] ...1   617.334413: rtcpu_vinotify_handle_msg: tstamp:19623835871 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:2443965125 data:0x00000000
     kworker/1:3-360   [001] ...1   617.334416: rtcpu_vinotify_handle_msg: tstamp:19623835991 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2443965126 data:0x00000000
     kworker/1:3-360   [001] ...1   617.438349: rtos_queue_peek_from_isr_failed: tstamp:19626730136 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   617.594489: rtos_queue_peek_from_isr_failed: tstamp:19631730643 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   617.750367: rtos_queue_peek_from_isr_failed: tstamp:19636731147 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   617.906351: rtos_queue_peek_from_isr_failed: tstamp:19641731658 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   618.062375: rtos_queue_peek_from_isr_failed: tstamp:19646732165 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   618.218382: rtos_queue_peek_from_isr_failed: tstamp:19651732674 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   618.374350: rtos_queue_peek_from_isr_failed: tstamp:19656733184 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   618.530351: rtos_queue_peek_from_isr_failed: tstamp:19661733691 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   618.686374: rtos_queue_peek_from_isr_failed: tstamp:19666734205 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   618.894366: rtos_queue_peek_from_isr_failed: tstamp:19671734710 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   619.050367: rtos_queue_peek_from_isr_failed: tstamp:19676735195 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   619.206433: rtos_queue_peek_from_isr_failed: tstamp:19681735727 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   619.362337: rtos_queue_peek_from_isr_failed: tstamp:19686736205 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   619.518480: rtos_queue_peek_from_isr_failed: tstamp:19691736714 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   619.674378: rtos_queue_peek_from_isr_failed: tstamp:19696737357 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   619.830508: rtos_queue_peek_from_isr_failed: tstamp:19701737775 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   619.986387: rtos_queue_peek_from_isr_failed: tstamp:19706738280 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   620.142368: rtos_queue_peek_from_isr_failed: tstamp:19711738788 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   620.298380: rtos_queue_peek_from_isr_failed: tstamp:19716739295 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   620.454366: rtos_queue_peek_from_isr_failed: tstamp:19721739804 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   620.610374: rtos_queue_peek_from_isr_failed: tstamp:19726740358 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   620.766375: rtos_queue_peek_from_isr_failed: tstamp:19731740866 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   620.974378: rtos_queue_peek_from_isr_failed: tstamp:19736741371 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   621.130382: rtos_queue_peek_from_isr_failed: tstamp:19741741879 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   621.286378: rtos_queue_peek_from_isr_failed: tstamp:19746742387 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   621.442377: rtos_queue_peek_from_isr_failed: tstamp:19751742893 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   621.598398: rtos_queue_peek_from_isr_failed: tstamp:19756743400 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   621.754382: rtos_queue_peek_from_isr_failed: tstamp:19761743906 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   621.910386: rtos_queue_peek_from_isr_failed: tstamp:19766744413 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   622.066401: rtos_queue_peek_from_isr_failed: tstamp:19771744922 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   622.222386: rtos_queue_peek_from_isr_failed: tstamp:19776745427 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   622.378386: rtos_queue_peek_from_isr_failed: tstamp:19781745923 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   622.534386: rtos_queue_peek_from_isr_failed: tstamp:19786746440 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   622.690389: rtos_queue_peek_from_isr_failed: tstamp:19791746949 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   622.846383: rtos_queue_peek_from_isr_failed: tstamp:19796747459 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   623.054388: rtos_queue_peek_from_isr_failed: tstamp:19801747963 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   623.210394: rtos_queue_peek_from_isr_failed: tstamp:19806748470 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   623.366393: rtos_queue_peek_from_isr_failed: tstamp:19811748975 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   623.522394: rtos_queue_peek_from_isr_failed: tstamp:19816749484 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   623.678401: rtos_queue_peek_from_isr_failed: tstamp:19821749988 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   623.834395: rtos_queue_peek_from_isr_failed: tstamp:19826750497 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   623.990397: rtos_queue_peek_from_isr_failed: tstamp:19831751002 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   624.146602: rtos_queue_peek_from_isr_failed: tstamp:19836751510 queue:0x0b4a3c58
     kworker/1:3-360   [001] ...1   624.198409: rtos_queue_peek_from_isr_failed: tstamp:19838477658 queue:0x0b4a3c58
root@tegra-ubuntu:/home/nvidia#

hello x1tester62,

since you’re working with multiple camera use-case.
could you please refer to Topic 1037809 to add error handling mechanism for TX2.
thanks

I ran for a little bit with that kernel mod and it hasn’t crashed.
i will update if i run into any problems with it.

Thanks!

It ran for a bit and then crashed again with the following dmesg log:

[11961.623334] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[11961.630031] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[11961.630128] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x00000134
[11961.638601] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x00000134
[12541.550339] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[12541.557364] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[12541.557468] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000004
[12541.565512] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000004
[12541.576125] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x02000134
[12541.584474] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x02000134
[12917.122886] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[12917.129327] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[12917.129450] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000004
[12917.137901] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000004
[12917.146315] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x02000174
[12917.154437] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x02000174
[12981.290325] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[12981.296701] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[12981.296799] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000004
[12981.304703] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000004
[12981.312913] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x02000174
[12981.320862] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x02000174
[14116.597208] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[14116.603624] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[14116.603713] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000008
[14116.611597] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000008
[14116.619817] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x02000174
[14116.627780] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x02000174
[14129.167273] tegra-vi4 15700000.vi: Status:  7 channel:02 frame:0000
[14129.173645] tegra-vi4 15700000.vi:          timestamp sof 14139790522784 eof 14139806520768 data 0x00000004
[14129.183513] tegra-vi4 15700000.vi:          capture_id 731178 stream  0 vchan  0
[14129.419947] ov5693 33-0037: ov5693_s_stream++
[14129.419972] ov5693 33-0037: ov5693_s_stream--
[14129.420031] nvcsi 150c0000.nvcsi: csi4_cil_check_status (3) CIL_INTR_STATUS 0x00000134
[14129.428045] nvcsi 150c0000.nvcsi: csi4_cil_check_status (3) CIL_ERR_INTR_STATUS 0x00000134
[14129.443477] ov5693 32-0037: ov5693_s_stream++
[14129.443491] ov5693 32-0037: ov5693_s_stream--
[14129.443518] nvcsi 150c0000.nvcsi: csi4_cil_check_status (2) CIL_INTR_STATUS 0x00000154
[14129.451553] nvcsi 150c0000.nvcsi: csi4_cil_check_status (2) CIL_ERR_INTR_STATUS 0x00000154
[14129.474961] ov5693 31-0036: ov5693_s_stream++
[14129.474975] ov5693 31-0036: ov5693_s_stream--
[14129.475002] nvcsi 150c0000.nvcsi: csi4_cil_check_status (1) CIL_INTR_STATUS 0x00000134
[14129.483034] nvcsi 150c0000.nvcsi: csi4_cil_check_status (1) CIL_ERR_INTR_STATUS 0x00000134
[14130.165095] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[14130.171638] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[14130.171807] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000008
[14130.179947] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000008
[14130.188498] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x02000174
[14130.196694] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x02000174
[14130.233987] ov5693 30-0036: ov5693_s_stream++
[14130.233994] ov5693 30-0036: ov5693_s_stream--
[14130.234014] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000008
[14130.241863] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000008
[14130.250067] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x02000174
[14130.258010] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x02000174

Note the ov5693 messages are from my modded ov5693 driver i’m using for our input source.

Not seeing any faults in the trace either.

kworker/0:0-14880 [000] ...1 428166.532437: rtcpu_vinotify_handle_msg: tstamp:13380536608840 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:1713480308 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.532438: rtcpu_vinotify_handle_msg: tstamp:13380536608947 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:1713480309 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.532439: rtcpu_vinotify_handle_msg: tstamp:13380536609049 tag:CHANSEL_PXL_EOF channel:0x03 frame:0 vi_tstamp:1713480309 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.532439: rtcpu_vinotify_handle_msg: tstamp:13380536609152 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:1713480309 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.532440: rtcpu_vinotify_handle_msg: tstamp:13380536609253 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:1713480332 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.532441: rtcpu_vinotify_handle_msg: tstamp:13380536609381 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:1713480333 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.532442: rtcpu_vinotify_handle_msg: tstamp:13380536609508 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:1713480334 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.532442: rtcpu_vinotify_handle_msg: tstamp:13380536609630 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:1713480335 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584429: rtcpu_vinotify_handle_msg: tstamp:13380536629325 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:1713501222 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584432: rtcpu_vinotify_handle_msg: tstamp:13380536629474 tag:CHANSEL_PXL_SOF channel:0x03 frame:0 vi_tstamp:1713501223 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584432: rtcpu_vinotify_handle_msg: tstamp:13380536629622 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:1713501223 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584433: rtcpu_vinotify_handle_msg: tstamp:13380536629769 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:1713501231 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584434: rtcpu_vinotify_handle_msg: tstamp:13380536629870 tag:ATOMP_FS channel:0x03 frame:0 vi_tstamp:1713501231 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584435: rtcpu_vinotify_handle_msg: tstamp:13380536629972 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:1713501231 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584436: rtcpu_vinotify_handle_msg: tstamp:13380536900571 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:1713773081 data:0x38000000
     kworker/0:0-14880 [000] ...1 428166.584436: rtcpu_vinotify_handle_msg: tstamp:13380536995294 tag:CHANSEL_LOAD_FRAMED channel:0x08 frame:0 vi_tstamp:1713867809 data:0x18000000
     kworker/0:0-14880 [000] ...1 428166.584437: rtcpu_vinotify_handle_msg: tstamp:13380537129376 tag:CHANSEL_PXL_EOF channel:0x03 frame:0 vi_tstamp:1714001143 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584438: rtcpu_vinotify_handle_msg: tstamp:13380537129488 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:1714001143 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584439: rtcpu_vinotify_handle_msg: tstamp:13380537129590 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:1714001143 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584440: rtcpu_vinotify_handle_msg: tstamp:13380537129691 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:1714001165 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584440: rtcpu_vinotify_handle_msg: tstamp:13380537129821 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:1714001166 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584441: rtcpu_vinotify_handle_msg: tstamp:13380537129944 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:1714001167 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584443: rtcpu_vinotify_handle_msg: tstamp:13380537150179 tag:CHANSEL_PXL_SOF channel:0x03 frame:0 vi_tstamp:1714022056 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584444: rtcpu_vinotify_handle_msg: tstamp:13380537150343 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:1714022056 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584445: rtcpu_vinotify_handle_msg: tstamp:13380537150516 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:1714022056 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584445: rtcpu_vinotify_handle_msg: tstamp:13380537150692 tag:ATOMP_FS channel:0x03 frame:0 vi_tstamp:1714022063 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584447: rtcpu_vinotify_handle_msg: tstamp:13380537150796 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:1714022064 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584447: rtcpu_vinotify_handle_msg: tstamp:13380537150905 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:1714022064 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584448: rtcpu_vinotify_handle_msg: tstamp:13380537313708 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:1714186219 data:0x28000000
     kworker/0:0-14880 [000] ...1 428166.584449: rtcpu_vinotify_handle_msg: tstamp:13380537519290 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:1714391801 data:0x38000000
     kworker/0:0-14880 [000] ...1 428166.584450: rtcpu_vinotify_handle_msg: tstamp:13380537628614 tag:CHANSEL_LOAD_FRAMED channel:0x08 frame:0 vi_tstamp:1714501130 data:0x18000000
     kworker/0:0-14880 [000] ...1 428166.584451: rtcpu_vinotify_handle_msg: tstamp:13380537650218 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:1714521976 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584452: rtcpu_vinotify_handle_msg: tstamp:13380537650323 tag:CHANSEL_PXL_EOF channel:0x03 frame:0 vi_tstamp:1714521976 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584453: rtcpu_vinotify_handle_msg: tstamp:13380537650426 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:1714521977 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584453: rtcpu_vinotify_handle_msg: tstamp:13380537650533 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:1714521999 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584454: rtcpu_vinotify_handle_msg: tstamp:13380537650665 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:1714522000 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584455: rtcpu_vinotify_handle_msg: tstamp:13380537650788 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:1714522001 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584456: rtcpu_vinotify_handle_msg: tstamp:13380537671239 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:1714542890 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584457: rtcpu_vinotify_handle_msg: tstamp:13380537671389 tag:CHANSEL_PXL_SOF channel:0x03 frame:0 vi_tstamp:1714542890 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584458: rtcpu_vinotify_handle_msg: tstamp:13380537671535 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:1714542890 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584459: rtcpu_vinotify_handle_msg: tstamp:13380537671683 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:1714542890 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584460: rtcpu_vinotify_handle_msg: tstamp:13380537671829 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:1714542897 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584461: rtcpu_vinotify_handle_msg: tstamp:13380537671932 tag:ATOMP_FS channel:0x03 frame:0 vi_tstamp:1714542897 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584461: rtcpu_vinotify_handle_msg: tstamp:13380537672034 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:1714542898 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584462: rtcpu_vinotify_handle_msg: tstamp:13380537672134 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:1714542898 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584463: rtcpu_vinotify_handle_msg: tstamp:13380537790688 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:1714663203 data:0x08000000
     kworker/0:0-14880 [000] ...1 428166.584464: rtcpu_vinotify_handle_msg: tstamp:13380537945200 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:1714817716 data:0x28000000
     kworker/0:0-14880 [000] ...1 428166.584465: rtcpu_vinotify_handle_msg: tstamp:13380538167579 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:1715040093 data:0x38000000
     kworker/0:0-14880 [000] ...1 428166.584466: rtcpu_vinotify_handle_msg: tstamp:13380538171362 tag:CHANSEL_PXL_EOF channel:0x03 frame:0 vi_tstamp:1715042808 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584467: rtcpu_vinotify_handle_msg: tstamp:13380538171468 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:1715042809 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584468: rtcpu_vinotify_handle_msg: tstamp:13380538171578 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:1715042809 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584469: rtcpu_vinotify_handle_msg: tstamp:13380538171682 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:1715042809 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.584470: rtcpu_vinotify_handle_msg: tstamp:13380538171793 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:1715042832 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584471: rtcpu_vinotify_handle_msg: tstamp:13380538171924 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:1715042833 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584472: rtcpu_vinotify_handle_msg: tstamp:13380538172062 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:1715042833 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584473: rtcpu_vinotify_handle_msg: tstamp:13380538172186 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:1715042834 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584474: rtcpu_vinotify_handle_msg: tstamp:13380538191827 tag:CHANSEL_PXL_SOF channel:0x03 frame:0 vi_tstamp:1715063722 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584475: rtcpu_vinotify_handle_msg: tstamp:13380538191989 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:1715063723 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584476: rtcpu_vinotify_handle_msg: tstamp:13380538192137 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:1715063723 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.584476: rtcpu_vinotify_handle_msg: tstamp:13380538192281 tag:ATOMP_FS channel:0x03 frame:0 vi_tstamp:1715063730 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584477: rtcpu_vinotify_handle_msg: tstamp:13380538192383 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:1715063730 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.584478: rtcpu_vinotify_handle_msg: tstamp:13380538192496 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:1715063730 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636429: rtcpu_vinotify_handle_msg: tstamp:13380538424590 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:1715297104 data:0x08000000
     kworker/0:0-14880 [000] ...1 428166.636434: rtos_queue_peek_from_isr_failed: tstamp:13380538491649 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428166.636435: rtcpu_vinotify_handle_msg: tstamp:13380538579040 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:1715451567 data:0x28000000
     kworker/0:0-14880 [000] ...1 428166.636436: rtcpu_vinotify_handle_msg: tstamp:13380538691873 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:1715563642 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.636436: rtcpu_vinotify_handle_msg: tstamp:13380538691976 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:1715563642 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.636437: rtcpu_vinotify_handle_msg: tstamp:13380538692079 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:1715563663 data:0x00000008
     kworker/0:0-14880 [000] ...1 428166.636438: rtcpu_vinotify_handle_msg: tstamp:13380538692256 tag:ATOMP_FE channel:0x03 frame:0 vi_tstamp:1715563666 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636439: rtcpu_vinotify_handle_msg: tstamp:13380538692494 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:1715563667 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636440: rtcpu_vinotify_handle_msg: tstamp:13380538692626 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:1715563667 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636441: rtcpu_vinotify_handle_msg: tstamp:13380538712657 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:1715584556 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.636441: rtcpu_vinotify_handle_msg: tstamp:13380538712817 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:1715584556 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.636442: rtcpu_vinotify_handle_msg: tstamp:13380538712962 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:1715584556 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.636443: rtcpu_vinotify_handle_msg: tstamp:13380538713108 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:1715584563 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636444: rtcpu_vinotify_handle_msg: tstamp:13380538713212 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:1715584563 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636445: rtcpu_vinotify_handle_msg: tstamp:13380538713313 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:1715584564 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636446: rtcpu_vinotify_handle_msg: tstamp:13380538899551 tag:CHANSEL_LOAD_FRAMED channel:0x08 frame:0 vi_tstamp:1715772066 data:0x18000000
     kworker/0:0-14880 [000] ...1 428166.636447: rtcpu_vinotify_handle_msg: tstamp:13380539212718 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:1716084475 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.636448: rtcpu_vinotify_handle_msg: tstamp:13380539212830 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:1716084475 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.636449: rtcpu_vinotify_handle_msg: tstamp:13380539212933 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:1716084476 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.636450: rtcpu_vinotify_handle_msg: tstamp:13380539213035 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:1716084498 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636451: rtcpu_vinotify_handle_msg: tstamp:13380539213163 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:1716084499 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636452: rtcpu_vinotify_handle_msg: tstamp:13380539213286 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:1716084500 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636453: rtcpu_vinotify_handle_msg: tstamp:13380539232990 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:1716105389 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.636454: rtcpu_vinotify_handle_msg: tstamp:13380539233163 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:1716105397 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636455: rtcpu_vinotify_handle_msg: tstamp:13380539729283 tag:CHANSEL_LOAD_FRAMED channel:0x08 frame:0 vi_tstamp:1716601800 data:0x18000000
     kworker/0:0-14880 [000] ...1 428166.636455: rtcpu_vinotify_handle_msg: tstamp:13380539732939 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:1716605309 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.636456: rtcpu_vinotify_handle_msg: tstamp:13380539733045 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:1716605332 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636457: rtcpu_vinotify_handle_msg: tstamp:13380539754329 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:1716626223 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.636458: rtcpu_vinotify_handle_msg: tstamp:13380539754483 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:1716626223 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.636459: rtcpu_vinotify_handle_msg: tstamp:13380539754653 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:1716626223 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.636459: rtcpu_vinotify_handle_msg: tstamp:13380539754802 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:1716626230 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636460: rtcpu_vinotify_handle_msg: tstamp:13380539754910 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:1716626231 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636461: rtcpu_vinotify_handle_msg: tstamp:13380539755021 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:1716626231 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.636462: rtcpu_vinotify_handle_msg: tstamp:13380539847659 tag:CHANSEL_LOAD_FRAMED channel:0x02 frame:0 vi_tstamp:1716720188 data:0x28000000
     kworker/0:0-14880 [000] ...1 428166.688415: rtcpu_vinotify_handle_msg: tstamp:13380539892923 tag:CHANSEL_LOAD_FRAMED channel:0x08 frame:0 vi_tstamp:1716765248 data:0x18000000
     kworker/0:0-14880 [000] ...1 428166.688417: rtcpu_vinotify_handle_msg: tstamp:13380539893023 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:1716765369 data:0x08000000
     kworker/0:0-14880 [000] ...1 428166.688419: rtcpu_vinotify_handle_msg: tstamp:13380540254406 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:1717126142 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.688419: rtcpu_vinotify_handle_msg: tstamp:13380540254518 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:1717126142 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.688420: rtcpu_vinotify_handle_msg: tstamp:13380540254622 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:1717126143 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.688421: rtcpu_vinotify_handle_msg: tstamp:13380540254723 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:1717126166 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.688422: rtcpu_vinotify_handle_msg: tstamp:13380540254853 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:1717126166 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.688423: rtcpu_vinotify_handle_msg: tstamp:13380540254976 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:1717126166 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.688424: rtcpu_vinotify_handle_msg: tstamp:13380540275172 tag:CHANSEL_PXL_SOF channel:0x01 frame:0 vi_tstamp:1717147056 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.688425: rtcpu_vinotify_handle_msg: tstamp:13380540275322 tag:CHANSEL_PXL_SOF channel:0x02 frame:0 vi_tstamp:1717147057 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.688426: rtcpu_vinotify_handle_msg: tstamp:13380540275471 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:1717147057 data:0x00000001
     kworker/0:0-14880 [000] ...1 428166.688426: rtcpu_vinotify_handle_msg: tstamp:13380540275618 tag:ATOMP_FS channel:0x01 frame:0 vi_tstamp:1717147063 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.688427: rtcpu_vinotify_handle_msg: tstamp:13380540275724 tag:ATOMP_FS channel:0x02 frame:0 vi_tstamp:1717147064 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.688429: rtcpu_vinotify_handle_msg: tstamp:13380540275828 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:1717147064 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.688430: rtcpu_vinotify_handle_msg: tstamp:13380540775206 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:1717646975 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.688430: rtcpu_vinotify_handle_msg: tstamp:13380540775334 tag:CHANSEL_PXL_EOF channel:0x01 frame:0 vi_tstamp:1717646975 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.688431: rtcpu_vinotify_handle_msg: tstamp:13380540775439 tag:CHANSEL_PXL_EOF channel:0x02 frame:0 vi_tstamp:1717646976 data:0x04370002
     kworker/0:0-14880 [000] ...1 428166.688432: rtcpu_vinotify_handle_msg: tstamp:13380540775542 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:1717646999 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.688433: rtcpu_vinotify_handle_msg: tstamp:13380540775670 tag:ATOMP_FE channel:0x01 frame:0 vi_tstamp:1717647000 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.688433: rtcpu_vinotify_handle_msg: tstamp:13380540775793 tag:ATOMP_FE channel:0x02 frame:0 vi_tstamp:1717647000 data:0x00000000
     kworker/0:0-14880 [000] ...1 428166.792422: rtos_queue_peek_from_isr_failed: tstamp:13380543492171 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428166.948424: rtos_queue_peek_from_isr_failed: tstamp:13380548492681 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428167.104436: rtos_queue_peek_from_isr_failed: tstamp:13380553493173 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428167.260404: rtos_queue_peek_from_isr_failed: tstamp:13380558493677 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428167.416400: rtos_queue_peek_from_isr_failed: tstamp:13380563494219 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428167.572400: rtos_queue_peek_from_isr_failed: tstamp:13380568494693 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428167.624396: rtos_queue_peek_from_isr_failed: tstamp:13380570699073 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428167.680389: rtos_queue_peek_from_isr_failed: tstamp:13380571228497 queue:0x0b4a3c58
     kworker/0:0-14880 [000] ...1 428167.680391: rtcpu_start: tstamp:13380571229641
     kworker/0:0-14880 [000] ...1 428167.680393: rtos_queue_send_from_isr_failed: tstamp:13380571239350 queue:0x0b4a2f38
     kworker/0:0-14880 [000] ...1 428167.680394: rtcpu_vinotify_handle_msg: tstamp:13380571239652 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:1748111032 data:0x00000001
     kworker/0:0-14880 [000] ...1 428167.680395: rtos_queue_peek_from_isr_failed: tstamp:13380571766561 queue:0x0b4a3c58
root@tegra-ubuntu:/home/nvidia#

When running a single input, the error recovery triggers and then does recover.

[433612.831754] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[433612.838216] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[433612.838309] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x00000176
[433612.846338] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x00000176
[433649.999657] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[433650.006350] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[433650.006437] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000008
[433650.014799] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000008
[433650.023399] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x00000176
[433650.031719] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x00000176

i visually see this as a stop/start in my video stream.

Any idea why this would be happening on the Nvidia side of the Mipi transaction? I can investigate on my video source end if there are any overruns or anything.

Based on INTR_STATUS I am not receiving an End of Transmission when i should be.

The CIL_STATUS flags certain bits also, one of which is:
cil_data_lane_rxfifo_full

I am using 1080p60 video @ 4:2:2 YUV over i believe 2 lanes of MIPI per each of the 4 channels.
I know this can fit over 2 lanes, though we are close to the edge.

Issue is happening with 720p60 video as well though which is well below the data rate requirements (though it happens way less often than @ 1080p60).
EDIT: Also just noticed the error on the 720p60 video is 0x4 instead of 0x8 for INTR_STATUS. so intr_stat_pd_crc_err_vc0 instead of the timeout.

Is there any information on the rxfifo and maybe how it can be increased? Is there a way to pull the size?

i need to run some additional tests but it looks like it may only be happening on the first input channel.

I enabled debug messages into the VI4/CSI4 driver and noticed that only index 0 was being called for restart.

I’m now running longer term tests on the other 3 channels to verify this is the case or not.

hello x1tester62,

thanks for testing the patches.

may I know what’s your use-case.
your commands in comment #1 looks several plug-in being called.
had you tried to simply the pipeline, did you still able to reproduce the failure by only launching 4 sensors for previewing the frames.
thanks