Can't take multiple images with an external ISP in the v4l2 media pipeline

Hello,

I want to connect an external ISP between the image sensor and the CSI port. So pipeline something like sensor → isp chip (an FPGA in my case) → csi → vi. Without the ISP, everything seems to work great, I can take multiple images. But with the ISP in the v4l2 media pipeline, I can take the first image, but then it stops taking images. The sensor driver is implemented using the camera_common library.

Upon digging some, I found that in the RTCPU device driver, the rx_channel stops incrementing its w_count, denoting that the channel stops receiving the frames, but tx_channel’s r_count continues to increment, denoting that tx_channel is sending frames. With no increments in rx_channel’s w_count, the worker driver thinks there is nothing to process, the capture request times out, and no image gets taken.

Is there a particular flag or something special in the ISP’s initialization that I must set that I am not setting? Or is it not even possible to add an external ISP like that?

How do you check the rx/tx_channel?
Do you check the trace log if more information.

Thanks

Thanks for your reply. By ‘trace log’ do you mean dmesg log? If so, yes, I’ve been using dmesg logs to try to figure out what’s going on.

I will share a few screenshots here.

The first time I ask for it to take an image using the following command:
gst-launch-1.0 v4l2src device=/dev/gmsl/cam5 num-buffers=1 ! jpegenc ! filesink location=/home/root/pics/cam5.jpg

This is the sequence of events:

Inside the tegra_capture_ivc_tx, the ivc queue gets written with new data. Upon getting this new data, the tx_channel’s w_count increments.

Then, there is another write to the ivc queue. In the below screenshot, please notice how the tx_channel’s r_count has also increased. I assume that tx_channel’s r_count keeps track of the number of frames the tx_channel has successfully transferred to the rx_channel.

Then, to read from the ivc queue, the function tegra_capture_ivc_worker gets called. This function checks to see if the channel is empty, if it is not, the function runs a while loop processing all the frames that are not yet processed in the channel. The channel’s r_count and w_count are checked to check if a channel is empty. If w_count is greater than the r_count, the channel is not empty, that is, it has some unprocessed frames. If the counters are the same, that denotes that there is nothing to process, so the channel is empty. This will become important shortly. Notice in the below screenshot that the rx_channel’s w_count has incremented, denoting that there are frames to process, and the channel is not empty.

Since there are 2 frames in the ivc queue, the worker function runs the while loop twice.

Then, there is another write, followed by another read.

But the second time, I ran the gst-launch command, the sequence of events went like this:

Write → Write → missing timing ERROR can’t take an image → attempt to read but rx_channel is empty.

Here is the screenshot of the error:

The rx_channel always reads empty because the rx_channel’s w_count stops incrementing. Here is the screenshot of dmesg | grep ivccapture

I am also finding that these counters vary depending on the v4l2 settings from the fpga driver.

  1. With only .enum_frame_size in pad_ops set, the counters go till 3, but after the first read, the last 2 reads have the counter values all set to zeros.
  2. With no .close in internal_ops, the first capture counters only go till 2.
  3. With .video and .pad set to NULL in subdev_ops, the first capture counters also only go till 2.

Hopefully, the images will work alright. Please let me know if I can clarify something further. I appreciate your time!

Get the trace log by below command.

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 3 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace

#Run the v4l2-ctl command
#below to get the trace log.
cat /sys/kernel/debug/tracing/trace

Thanks for clarifying and helping with getting the trace log setup. Unfortunately, it didn’t tell me a whole lot more than what I had learned by putting ‘endless’ print statements pretty much all over.

Let me share the trace log results here. Please let me know if you see something obviously off.

This is the snippet from the trace log when there is no FPGA in the pipeline.

gst-launch-1.0-32657 [003] ....   715.479790: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-32657 [001] ....   715.481125: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-32657 [001] ....   715.481128: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-32657 [001] ....   715.481130: csi_s_power: enable : 0x1
  gst-launch-1.0-32657 [001] ....   715.482549: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-32657 [001] ....   715.482555: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-32657 [001] ....   715.482556: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-32657 [001] ....   715.482557: csi_s_power: enable : 0x0
  gst-launch-1.0-32657 [001] ....   715.492887: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-32657 [001] ....   715.492896: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-32657 [001] ....   715.492897: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-32657 [001] ....   715.492899: csi_s_power: enable : 0x1
    v4l2src0:src-32659 [007] ....   715.497945: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
...
...
...
v4l2src0:src-32659 [006] ....   716.530868: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-32659 [001] ....   716.532878: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x1
    v4l2src0:src-32659 [001] ....   716.532883: csi_s_stream: enable : 0x1
    v4l2src0:src-32659 [001] ....   716.533056: tegra_channel_set_stream: ar0231 2-0009 : 0x1
...
...
...
 vi-output, ar02-32662 [003] ....   718.117777: tegra_channel_capture_frame: sof:731.362188800
 vi-output, ar02-32662 [003] ....   718.117780: tegra_channel_capture_frame: eof:731.392632704
...
...
...
vi-output, ar02-32662 [003] ....   720.108316: tegra_channel_capture_frame: sof:732.461769472
 vi-output, ar02-32662 [003] ....   720.108320: tegra_channel_capture_frame: eof:732.492213376
  gst-launch-1.0-32657 [005] ....   720.108366: tegra_channel_set_stream: enable : 0x0
  gst-launch-1.0-32657 [005] ....   720.108369: tegra_channel_set_stream: ar0231 2-0009 : 0x0
  gst-launch-1.0-32657 [005] ....   720.120002: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-32657 [005] ....   720.120007: csi_s_stream: enable : 0x0
...
...
...
<...>-34654 [003] ....   771.932023: tegra_channel_open: vi-output, ar0231 2-0009
           <...>-34654 [001] ....   771.933580: tegra_channel_set_power: ar0231 2-0009 : 0x1
           <...>-34654 [001] ....   771.933584: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
           <...>-34654 [001] ....   771.933586: csi_s_power: enable : 0x1
           <...>-34654 [001] ....   771.935102: tegra_channel_close: vi-output, ar0231 2-0009
           <...>-34654 [001] ....   771.935108: tegra_channel_set_power: ar0231 2-0009 : 0x0
           <...>-34654 [001] ....   771.935109: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
           <...>-34654 [001] ....   771.935110: csi_s_power: enable : 0x0
           <...>-34654 [001] ....   771.941822: tegra_channel_open: vi-output, ar0231 2-0009
           <...>-34654 [001] ....   771.941830: tegra_channel_set_power: ar0231 2-0009 : 0x1
           <...>-34654 [001] ....   771.941831: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
           <...>-34654 [001] ....   771.941833: csi_s_power: enable : 0x1
           <...>-34655 [007] ....   771.946588: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
...
...
...
<...>-34655 [006] ....   772.980058: tegra_channel_set_stream: enable : 0x1
           <...>-34655 [001] ....   772.981804: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x1
           <...>-34655 [001] ....   772.981809: csi_s_stream: enable : 0x1
           <...>-34655 [001] ....   772.981960: tegra_channel_set_stream: ar0231 2-0009 : 0x1
...
...
...
<...>-34704 [007] ....   775.408642: tegra_channel_capture_frame: sof:787.811061792
           <...>-34704 [007] ....   775.408645: tegra_channel_capture_frame: eof:787.841505728
...
...
...
<...>-34704 [007] ....   776.577761: tegra_channel_capture_frame: sof:788.943963136
           <...>-34704 [007] ....   776.577766: tegra_channel_capture_frame: eof:788.974407072
           <...>-34654 [003] ....   776.577813: tegra_channel_set_stream: enable : 0x0
           <...>-34654 [003] ....   776.577816: tegra_channel_set_stream: ar0231 2-0009 : 0x0
           <...>-34654 [001] ....   776.589077: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x0
           <...>-34654 [001] ....   776.589083: csi_s_stream: enable : 0x0
...
...
...
<...>-34654 [001] ....   778.664794: tegra_channel_close: vi-output, ar0231 2-0009
           <...>-34654 [001] ....   778.664802: tegra_channel_set_power: ar0231 2-0009 : 0x0
           <...>-34654 [001] ....   778.664804: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
           <...>-34654 [001] ....   778.664806: csi_s_power: enable : 0x0

This is the snippet from the trace log when there is an FPGA in the pipeline.

gst-launch-1.0-9181  [002] ....    69.157054: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-9181  [001] ....    69.158445: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-9181  [001] ....    69.158449: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-9181  [001] ....    69.158450: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-9181  [001] ....    69.158451: csi_s_power: enable : 0x1
  gst-launch-1.0-9181  [001] ....    69.159772: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-9181  [001] ....    69.159777: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-9181  [001] ....    69.159778: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-9181  [001] ....    69.159778: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-9181  [001] ....    69.159779: csi_s_power: enable : 0x0
  gst-launch-1.0-9181  [001] ....    69.169841: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-9181  [001] ....    69.169850: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-9181  [001] ....    69.169851: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-9181  [001] ....    69.169852: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-9181  [001] ....    69.169854: csi_s_power: enable : 0x1
    v4l2src0:src-9184  [004] ....    69.175188: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
...
...
...
v4l2src0:src-9184  [006] ....    70.208050: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-9184  [006] ....    70.210793: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x1
    v4l2src0:src-9184  [006] ....    70.210795: csi_s_stream: enable : 0x1
    v4l2src0:src-9184  [006] ....    70.210939: tegra_channel_set_stream: rvnirp461 2-000a : 0x1
    v4l2src0:src-9184  [006] ....    70.210970: tegra_channel_set_stream: ar0231 2-0009 : 0x1
...
...
...
vi-output, ar02-9210  [006] ....    72.645553: tegra_channel_capture_frame: sof:85.64560032
 vi-output, ar02-9210  [006] ....    72.645557: tegra_channel_capture_frame: eof:85.95003968
...
...
...
gst-launch-1.0-9181  [003] ....    73.769126: tegra_channel_set_stream: enable : 0x0
  gst-launch-1.0-9181  [003] ....    73.769131: tegra_channel_set_stream: ar0231 2-0009 : 0x0
  gst-launch-1.0-9181  [001] ....    73.780552: tegra_channel_set_stream: rvnirp461 2-000a : 0x0
...
...
...
gst-launch-1.0-9181  [001] ....    73.785822: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-9181  [001] ....    73.785825: csi_s_stream: enable : 0x0
...
...
...
gst-launch-1.0-9181  [001] ....    75.863116: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-9181  [001] ....    75.863123: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-9181  [001] ....    75.863124: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-9181  [001] ....    75.863125: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-9181  [001] ....    75.863128: csi_s_power: enable : 0x0
...
...
...
gst-launch-1.0-13332 [003] ....   183.542215: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-13332 [001] ....   183.543664: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-13332 [001] ....   183.543668: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-13332 [001] ....   183.543669: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-13332 [001] ....   183.543670: csi_s_power: enable : 0x1
  gst-launch-1.0-13332 [001] ....   183.545061: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-13332 [001] ....   183.545066: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-13332 [001] ....   183.545067: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-13332 [001] ....   183.545068: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-13332 [001] ....   183.545068: csi_s_power: enable : 0x0
  gst-launch-1.0-13332 [001] ....   183.551517: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-13332 [001] ....   183.551526: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-13332 [001] ....   183.551527: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-13332 [001] ....   183.551527: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-13332 [001] ....   183.551529: csi_s_power: enable : 0x1
    v4l2src0:src-13333 [007] ....   183.556745: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
...
...
...
v4l2src0:src-13333 [006] ....   184.589960: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-13333 [006] ....   184.592641: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x1
    v4l2src0:src-13333 [006] ....   184.592644: csi_s_stream: enable : 0x1
    v4l2src0:src-13333 [006] ....   184.592807: tegra_channel_set_stream: rvnirp461 2-000a : 0x1
    v4l2src0:src-13333 [006] ....   184.592824: tegra_channel_set_stream: ar0231 2-0009 : 0x1
...
...
...
vi-output, ar02-13336 [005] ....   194.294622: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
...
...
...
gst-launch-1.0-13332 [004] ....   195.369323: tegra_channel_set_stream: enable : 0x0
  gst-launch-1.0-13332 [004] ....   195.369328: tegra_channel_set_stream: ar0231 2-0009 : 0x0
  gst-launch-1.0-13332 [001] ....   195.380261: tegra_channel_set_stream: rvnirp461 2-000a : 0x0
  gst-launch-1.0-13332 [001] ....   195.386025: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-13332 [001] ....   195.386029: csi_s_stream: enable : 0x0
...
...
...
gst-launch-1.0-13332 [001] ....   197.469474: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-13332 [001] ....   197.469482: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-13332 [001] ....   197.469483: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-13332 [001] ....   197.469484: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-13332 [001] ....   197.469486: csi_s_power: enable : 0x0

*The “…” are additional worker statements. Most of them show when some isr is failing. Here is a screenshot, please let me know if these are important.

Some of the things I noticed were:

  1. Without the FPGA in the pipeline, every time there is a capture, the device shows “sof” and “eof” time stamps twice. But in the case of the FPGA, for the first capture, the “sof” and “eof” time stamp is shown once and then it isn’t shown in the second capture at all.

  2. In the case without the FPGA, for the second capture, under the task-PID column, the tasks are no longer shown, instead there is a <…>. Whereas, in the case with the FPGA in the pipeline, the tasks continue to show.

My questions are:

  1. Anything that stands out to you as obviously wrong? Or seems odd? I mean in the log messages that I just shared.

  2. I want to share the driver code snippets as well. This is how I am ‘initializing’ the common_camera parameters for the FPGA driver.

common_data->ops		= &rvnirp_common_ops;
	// common_data->ctrl_handler	= &rvnirp->ctrl_handler;
	common_data->dev		= &client->dev;
	common_data->colorfmt		= camera_common_find_datafmt(RVNIRP_DEFAULT_DATAFMT);
	common_data->power		= &rvnirp->power;
	// common_data->ctrls		= rvnirp->ctrls;
	common_data->priv		= (void *)rvnirp;
	// common_data->numctrls		= ARRAY_SIZE(ctrl_config_list);
	common_data->numfmts		= 1;
	common_data->def_mode		= 0;
	common_data->def_width		= RVNIRP_INPUT_WIDTH; // THIS SHOULD BE 1928, but that's what ar0231 has.
	common_data->def_height		= RVNIRP_INPUT_HEIGHT;
	common_data->def_clk_freq	= 24000000;
	common_data->fmt_width		= common_data->def_width;
	common_data->fmt_height		= common_data->def_height;

	rvnirp->client		= client;
	rvnirp->s_data			= common_data;
	rvnirp->subdev			= &common_data->subdev;
	rvnirp->subdev->dev		= &client->dev;


	retval = camera_common_initialize(common_data, "rvnirp");
	if (retval) {
		dev_err(&client->dev, "Failed to initialize rvnirp\n");
		return -ENOMEM;
	}

    v4l2_i2c_subdev_init(&common_data->subdev, client, &rvnirp_subdev_ops);
    /* Initialize subdev */
	rvnirp->subdev->internal_ops = &rvnirp_internal_ops;
	rvnirp->subdev->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
	rvnirp->subdev->entity.ops = &rvnirp_media_ops;
	rvnirp->subdev->entity.function = MEDIA_INTF_T_V4L_SUBDEV;

First, do I even need to include the common_camera library in the FPGA driver? When I remove the common_camera library from the FPGA driver, the trace logs don’t show it in the power enable or the other sections, so I assumed that adding common_camera is a step closer to success.

Second, is it even possible to add an external ISP to the pipeline like this? The output of media-ctl /dev/media0 -p is:

Since the media device seems to be happy with rvnirp FPGA I took that as another step closer to success. But, does that look right to you?

I tested the system by capturing multiple images while the FPGA was in the pipeline, and all the images turned out great. However, when I closed the pipeline and reopened it, the system stopped taking images. What could be causing this behavior? Have I missed something?

Again, I appreciate your time and help with this so much!

Have you boost the clocks to try.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

I ran all of the above commands to boost the clocks. The results of cat commands are these:

cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
-> 998400000

cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
-> 1190400000

cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
-> 400000000

cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
-> 2133000000

I tried taking images after this and got the same results. It can take the first image just fine, but after that it stops taking images. Here is the trace log.

# tracer: nop
#
# entries-in-buffer/entries-written: 742/742   #P:8
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |

  gst-launch-1.0-15160 [003] ....   234.446434: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-15160 [001] ....   234.447794: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-15160 [001] ....   234.447798: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-15160 [001] ....   234.447799: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-15160 [001] ....   234.447801: csi_s_power: enable : 0x1
  gst-launch-1.0-15160 [001] ....   234.449146: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-15160 [001] ....   234.449151: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-15160 [001] ....   234.449151: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-15160 [001] ....   234.449152: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-15160 [001] ....   234.449153: csi_s_power: enable : 0x0
  gst-launch-1.0-15160 [001] ....   234.459625: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-15160 [001] ....   234.459637: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-15160 [001] ....   234.459638: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-15160 [001] ....   234.459638: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-15160 [001] ....   234.459641: csi_s_power: enable : 0x1
    v4l2src0:src-15161 [003] ....   234.463951: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
    
    v4l2src0:src-15161 [003] ....   235.478996: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-15161 [003] ....   235.481319: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x1
    v4l2src0:src-15161 [003] ....   235.481322: csi_s_stream: enable : 0x1
    v4l2src0:src-15161 [003] ....   235.481465: tegra_channel_set_stream: rvnirp461 2-000a : 0x1
    v4l2src0:src-15161 [003] ....   235.481476: tegra_channel_set_stream: ar0231 2-0009 : 0x1

 vi-output, ar02-15192 [004] ....   238.016829: tegra_channel_capture_frame: sof:250.401433664
 vi-output, ar02-15192 [004] ....   238.016834: tegra_channel_capture_frame: eof:250.431877728
     
 vi-output, ar02-15192 [004] ....   239.343602: tegra_channel_capture_frame: sof:251.634300320
 vi-output, ar02-15192 [004] ....   239.349137: tegra_channel_capture_frame: eof:251.664744256
    
  gst-launch-1.0-15160 [005] ....   239.470438: tegra_channel_set_stream: enable : 0x0
  gst-launch-1.0-15160 [005] ....   239.470442: tegra_channel_set_stream: ar0231 2-0009 : 0x0
  gst-launch-1.0-15160 [001] ....   239.481016: tegra_channel_set_stream: rvnirp461 2-000a : 0x0
  gst-launch-1.0-15160 [001] ....   239.486446: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-15160 [001] ....   239.486448: csi_s_stream: enable : 0x0
    
  gst-launch-1.0-15160 [001] ....   241.583087: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-15160 [001] ....   241.583093: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-15160 [001] ....   241.583095: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-15160 [001] ....   241.583096: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-15160 [001] ....   241.583098: csi_s_power: enable : 0x0
   
  gst-launch-1.0-16176 [006] ....   260.569749: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-16176 [001] ....   260.571192: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-16176 [001] ....   260.571197: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-16176 [001] ....   260.571198: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-16176 [001] ....   260.571200: csi_s_power: enable : 0x1
  gst-launch-1.0-16176 [001] ....   260.572623: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-16176 [001] ....   260.572630: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-16176 [001] ....   260.572631: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-16176 [001] ....   260.572632: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-16176 [001] ....   260.572633: csi_s_power: enable : 0x0
  gst-launch-1.0-16176 [001] ....   260.579251: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-16176 [001] ....   260.579260: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-16176 [001] ....   260.579261: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-16176 [001] ....   260.579262: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-16176 [001] ....   260.579264: csi_s_power: enable : 0x1
    v4l2src0:src-16177 [007] ....   260.583918: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
   
    v4l2src0:src-16177 [002] ....   261.599565: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-16177 [001] ....   261.601056: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x1
    v4l2src0:src-16177 [001] ....   261.601060: csi_s_stream: enable : 0x1
    v4l2src0:src-16177 [001] ....   261.601217: tegra_channel_set_stream: rvnirp461 2-000a : 0x1
    v4l2src0:src-16177 [001] ....   261.601238: tegra_channel_set_stream: ar0231 2-0009 : 0x1
    
 vi-output, ar02-16208 [001] ....   271.316258: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
     
  gst-launch-1.0-16176 [003] ....   272.512422: tegra_channel_set_stream: enable : 0x0
  gst-launch-1.0-16176 [003] ....   272.512426: tegra_channel_set_stream: ar0231 2-0009 : 0x0
  gst-launch-1.0-16176 [001] ....   272.522931: tegra_channel_set_stream: rvnirp461 2-000a : 0x0
  gst-launch-1.0-16176 [001] ....   272.528692: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-16176 [001] ....   272.528696: csi_s_stream: enable : 0x0
   
  gst-launch-1.0-16176 [001] ....   274.611251: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-16176 [001] ....   274.611257: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-16176 [001] ....   274.611259: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-16176 [001] ....   274.611260: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-16176 [001] ....   274.611262: csi_s_power: enable : 0x0

I tried taking multiple images after opening the pipeline for the first time and it was able to take multiple images just fine. However, after closing the pipeline and opening it again, it stopped taking images. I think I am not closing the pipeline correctly the first time. Let me know if you know of ways to check on that more thoroughly. Thanks again for your time and help!

Here is the “non refined” version of trace logs:

# tracer: nop
#
# entries-in-buffer/entries-written: 742/742   #P:8
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/1:3-2111  [001] ....    50.891799: rtos_queue_peek_from_isr_failed: tstamp:2004884538 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    51.063799: rtos_queue_peek_from_isr_failed: tstamp:2009884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    51.239781: rtos_queue_peek_from_isr_failed: tstamp:2014884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    51.407782: rtos_queue_peek_from_isr_failed: tstamp:2019884527 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    51.575777: rtos_queue_peek_from_isr_failed: tstamp:2024884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    51.691801: rtos_queue_peek_from_isr_failed: tstamp:2029884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    51.859815: rtos_queue_peek_from_isr_failed: tstamp:2034884527 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    52.031784: rtos_queue_peek_from_isr_failed: tstamp:2039884531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    52.203764: rtos_queue_peek_from_isr_failed: tstamp:2044884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    52.371785: rtos_queue_peek_from_isr_failed: tstamp:2049884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    52.539765: rtos_queue_peek_from_isr_failed: tstamp:2054884530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    52.651797: rtos_queue_peek_from_isr_failed: tstamp:2059884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    52.827760: rtos_queue_peek_from_isr_failed: tstamp:2064884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    52.999748: rtos_queue_peek_from_isr_failed: tstamp:2069884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    53.171767: rtos_queue_peek_from_isr_failed: tstamp:2074884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    53.339776: rtos_queue_peek_from_isr_failed: tstamp:2079884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    53.451763: rtos_queue_peek_from_isr_failed: tstamp:2084884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    53.627781: rtos_queue_peek_from_isr_failed: tstamp:2089884532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    53.795761: rtos_queue_peek_from_isr_failed: tstamp:2094884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    53.971759: rtos_queue_peek_from_isr_failed: tstamp:2099884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    54.143756: rtos_queue_peek_from_isr_failed: tstamp:2104884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    54.255756: rtos_queue_peek_from_isr_failed: tstamp:2109884527 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    54.431751: rtos_queue_peek_from_isr_failed: tstamp:2114884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    54.599761: rtos_queue_peek_from_isr_failed: tstamp:2119884531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    54.771750: rtos_queue_peek_from_isr_failed: tstamp:2124884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    54.939814: rtos_queue_peek_from_isr_failed: tstamp:2129884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    55.051758: rtos_queue_peek_from_isr_failed: tstamp:2134884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    55.219756: rtos_queue_peek_from_isr_failed: tstamp:2139884530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    55.395773: rtos_queue_peek_from_isr_failed: tstamp:2144884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    55.567759: rtos_queue_peek_from_isr_failed: tstamp:2149884528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....    55.679764: rtos_queue_peek_from_isr_failed: tstamp:2153941882 queue:0x0bcbb8b8
  gst-launch-1.0-15160 [003] ....   234.446434: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-15160 [001] ....   234.447794: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-15160 [001] ....   234.447798: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-15160 [001] ....   234.447799: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-15160 [001] ....   234.447801: csi_s_power: enable : 0x1
  gst-launch-1.0-15160 [001] ....   234.449146: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-15160 [001] ....   234.449151: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-15160 [001] ....   234.449151: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-15160 [001] ....   234.449152: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-15160 [001] ....   234.449153: csi_s_power: enable : 0x0
  gst-launch-1.0-15160 [001] ....   234.459625: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-15160 [001] ....   234.459637: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-15160 [001] ....   234.459638: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-15160 [001] ....   234.459638: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-15160 [001] ....   234.459641: csi_s_power: enable : 0x1
    v4l2src0:src-15161 [003] ....   234.463951: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
     kworker/1:3-2111  [001] ....   234.495786: rtos_queue_peek_from_isr_failed: tstamp:7741699171 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   234.495790: rtcpu_start: tstamp:7741700354
     kworker/1:3-2111  [001] ....   234.551813: rtos_queue_send_from_isr_failed: tstamp:7744294573 queue:0x0bcb2b38
     kworker/1:3-2111  [001] ....   234.551816: rtos_queue_send_from_isr_failed: tstamp:7744294711 queue:0x0bcb73a0
     kworker/1:3-2111  [001] ....   234.551817: rtos_queue_send_from_isr_failed: tstamp:7744294847 queue:0x0bcb8f20
     kworker/1:3-2111  [001] ....   234.551818: rtos_queue_send_from_isr_failed: tstamp:7744294981 queue:0x0bcb9ce0
     kworker/1:3-2111  [001] ....   234.551819: rtos_queue_send_from_isr_failed: tstamp:7744295117 queue:0x0bcbaaa0
     kworker/1:3-2111  [001] ....   234.551820: rtcpu_string: tstamp:7744295527 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/1:3-2111  [001] ....   234.551839: rtcpu_string: tstamp:7744295699 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/1:3-2111  [001] ....   234.551869: rtcpu_string: tstamp:7744295942 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/1:3-2111  [001] ....   234.551878: rtcpu_string: tstamp:7744296170 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/1:3-2111  [001] ....   234.551885: rtcpu_string: tstamp:7744296385 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/1:3-2111  [001] ....   234.551890: rtcpu_string: tstamp:7744296616 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/1:3-2111  [001] ....   234.551896: rtcpu_string: tstamp:7744296847 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/1:3-2111  [001] ....   234.551902: rtcpu_string: tstamp:7744304342 id:0x04010000 str:"vi5_hwinit: firmware CL2018101701 protocol vers"
     kworker/1:3-2111  [001] ....   234.551903: rtcpu_string: tstamp:7744304447 id:0x04010000 str:"on 2.2
"
     kworker/1:3-2111  [001] ....   234.667771: rtos_queue_peek_from_isr_failed: tstamp:7746699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   234.835803: rtos_queue_peek_from_isr_failed: tstamp:7751699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   234.951770: rtos_queue_peek_from_isr_failed: tstamp:7756699527 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   235.123783: rtos_queue_peek_from_isr_failed: tstamp:7761699527 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   235.295755: rtos_queue_peek_from_isr_failed: tstamp:7766699527 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   235.463772: rtos_queue_peek_from_isr_failed: tstamp:7771699527 queue:0x0bcbb8b8
    v4l2src0:src-15161 [003] ....   235.478996: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-15161 [003] ....   235.481319: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x1
    v4l2src0:src-15161 [003] ....   235.481322: csi_s_stream: enable : 0x1
    v4l2src0:src-15161 [003] ....   235.481465: tegra_channel_set_stream: rvnirp461 2-000a : 0x1
    v4l2src0:src-15161 [003] ....   235.481476: tegra_channel_set_stream: ar0231 2-0009 : 0x1
     kworker/1:3-2111  [001] ....   235.519754: rtos_queue_send_from_isr_failed: tstamp:7773367368 queue:0x0bcb2b38
     kworker/1:3-2111  [001] ....   235.519757: rtos_queue_send_from_isr_failed: tstamp:7773367507 queue:0x0bcb73a0
     kworker/1:3-2111  [001] ....   235.519758: rtos_queue_send_from_isr_failed: tstamp:7773367655 queue:0x0bcb8f20
     kworker/1:3-2111  [001] ....   235.519758: rtos_queue_send_from_isr_failed: tstamp:7773367788 queue:0x0bcb9ce0
     kworker/1:3-2111  [001] ....   235.519759: rtos_queue_send_from_isr_failed: tstamp:7773367923 queue:0x0bcbaaa0
     kworker/1:3-2111  [001] ....   235.519759: rtos_queue_send_from_isr_failed: tstamp:7773369339 queue:0x0bcb2b38
     kworker/1:3-2111  [001] ....   235.519760: rtos_queue_send_from_isr_failed: tstamp:7773369474 queue:0x0bcb73a0
     kworker/1:3-2111  [001] ....   235.519760: rtos_queue_send_from_isr_failed: tstamp:7773369610 queue:0x0bcb8f20
     kworker/1:3-2111  [001] ....   235.519761: rtos_queue_send_from_isr_failed: tstamp:7773369742 queue:0x0bcb9ce0
     kworker/1:3-2111  [001] ....   235.519761: rtos_queue_send_from_isr_failed: tstamp:7773369876 queue:0x0bcbaaa0
     kworker/1:3-2111  [001] ....   235.631773: rtos_queue_peek_from_isr_failed: tstamp:7776699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   235.799750: rtos_queue_peek_from_isr_failed: tstamp:7781699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   235.911793: rtos_queue_peek_from_isr_failed: tstamp:7786699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   236.087782: rtos_queue_peek_from_isr_failed: tstamp:7791699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   236.259775: rtos_queue_peek_from_isr_failed: tstamp:7796699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   236.427767: rtos_queue_peek_from_isr_failed: tstamp:7801699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   236.595769: rtos_queue_peek_from_isr_failed: tstamp:7806699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   236.763767: rtos_queue_peek_from_isr_failed: tstamp:7811699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   236.879768: rtos_queue_peek_from_isr_failed: tstamp:7816699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   237.055771: rtos_queue_peek_from_isr_failed: tstamp:7821699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   237.167785: rtos_queue_send_from_isr_failed: tstamp:7824953129 queue:0x0bcb2b38
     kworker/1:3-2111  [001] ....   237.167789: rtos_queue_send_from_isr_failed: tstamp:7824953268 queue:0x0bcb73a0
     kworker/1:3-2111  [001] ....   237.167790: rtos_queue_send_from_isr_failed: tstamp:7824953406 queue:0x0bcb8f20
     kworker/1:3-2111  [001] ....   237.167790: rtos_queue_send_from_isr_failed: tstamp:7824953540 queue:0x0bcb9ce0
     kworker/1:3-2111  [001] ....   237.167791: rtos_queue_send_from_isr_failed: tstamp:7824953674 queue:0x0bcbaaa0
     kworker/1:3-2111  [001] ....   237.167792: rtcpu_string: tstamp:7824954506 id:0x04010000 str:"VI GOS[#0] set to VM GOS[4] base 0xc2104000
"
     kworker/1:3-2111  [001] ....   237.167812: rtcpu_vinotify_event: tstamp:7824963875 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:9880404000 data:0x10000000
     kworker/1:3-2111  [001] ....   237.167813: rtcpu_vinotify_event: tstamp:7824964007 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:9880407808 data:0x31000001
     kworker/1:3-2111  [001] ....   237.167814: rtcpu_vinotify_event: tstamp:7824964151 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:9880408832 data:0x07020001
     kworker/1:3-2111  [001] ....   237.167818: rtcpu_vinotify_event: tstamp:7825278426 tag:FS channel:0x03 frame:0 vi_tstamp:7825044359 data:0x00000010
     kworker/1:3-2111  [001] ....   237.167819: rtcpu_vinotify_event: tstamp:7825278574 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:7825044359 data:0x00000000
     kworker/1:3-2111  [001] ....   237.167820: rtcpu_vinotify_event: tstamp:7825278701 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:7825044802 data:0x00000001
     kworker/1:3-2111  [001] ....   237.167837: rtcpu_vinotify_event: tstamp:7825278844 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:9883265632 data:0x08020001
     kworker/1:3-2111  [001] ....   237.227765: rtcpu_vinotify_error: tstamp:7826086734 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7826086076 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.227770: rtcpu_vinotify_event: tstamp:7826222252 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:7825996152 data:0x04b70002
     kworker/1:3-2111  [001] ....   237.227771: rtcpu_vinotify_event: tstamp:7826222401 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:7825996165 data:0x00000000
     kworker/1:3-2111  [001] ....   237.227771: rtcpu_vinotify_event: tstamp:7826222532 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:9913709216 data:0x02020001
     kworker/1:3-2111  [001] ....   237.227772: rtcpu_vinotify_event: tstamp:7826222673 tag:FE channel:0x03 frame:0 vi_tstamp:7825996178 data:0x00000020
     kworker/1:3-2111  [001] ....   237.227773: rtcpu_vinotify_event: tstamp:7826222800 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:7825996179 data:0x00000000
     kworker/1:3-2111  [001] ....   237.227774: rtcpu_vinotify_event: tstamp:7826222944 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:9913712128 data:0x00020001
     kworker/1:3-2111  [001] ....   237.227775: rtcpu_vinotify_event: tstamp:7826223067 tag:FS channel:0x03 frame:0 vi_tstamp:7826085632 data:0x00000010
     kworker/1:3-2111  [001] ....   237.227775: rtcpu_vinotify_event: tstamp:7826223207 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7826086076 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.227777: rtos_queue_peek_from_isr_failed: tstamp:7826699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   237.227778: rtcpu_vinotify_event: tstamp:7827037967 tag:FE channel:0x03 frame:0 vi_tstamp:7827037450 data:0x00000020
     kworker/1:3-2111  [001] ....   237.227799: rtcpu_vinotify_error: tstamp:7827128049 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7827127349 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.227801: rtcpu_vinotify_event: tstamp:7827323443 tag:FS channel:0x03 frame:0 vi_tstamp:7827126906 data:0x00000010
     kworker/1:3-2111  [001] ....   237.227802: rtcpu_vinotify_event: tstamp:7827323569 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7827127349 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.287762: rtcpu_vinotify_error: tstamp:7828169274 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7828168621 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.287768: rtcpu_vinotify_event: tstamp:7828267322 tag:FE channel:0x03 frame:0 vi_tstamp:7828078721 data:0x00000020
     kworker/1:3-2111  [001] ....   237.287768: rtcpu_vinotify_event: tstamp:7828267449 tag:FS channel:0x03 frame:0 vi_tstamp:7828168177 data:0x00000010
     kworker/1:3-2111  [001] ....   237.287769: rtcpu_vinotify_event: tstamp:7828267591 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7828168621 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.287770: rtcpu_vinotify_error: tstamp:7829210550 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7829209892 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.287771: rtcpu_vinotify_event: tstamp:7829211069 tag:FE channel:0x03 frame:0 vi_tstamp:7829119993 data:0x00000020
     kworker/1:3-2111  [001] ....   237.287771: rtcpu_vinotify_event: tstamp:7829211223 tag:FS channel:0x03 frame:0 vi_tstamp:7829209449 data:0x00000010
     kworker/1:3-2111  [001] ....   237.287772: rtcpu_vinotify_event: tstamp:7829211349 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7829209892 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.343756: rtcpu_vinotify_error: tstamp:7830251820 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7830251166 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.343760: rtcpu_vinotify_event: tstamp:7830469343 tag:FE channel:0x03 frame:0 vi_tstamp:7830161269 data:0x00000020
     kworker/1:3-2111  [001] ....   237.343761: rtcpu_vinotify_event: tstamp:7830469466 tag:FS channel:0x03 frame:0 vi_tstamp:7830250723 data:0x00000010
     kworker/1:3-2111  [001] ....   237.343762: rtcpu_vinotify_event: tstamp:7830469608 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7830251166 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.343762: rtcpu_vinotify_error: tstamp:7831293092 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7831292439 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.343763: rtcpu_vinotify_event: tstamp:7831412958 tag:FE channel:0x03 frame:0 vi_tstamp:7831202541 data:0x00000020
     kworker/1:3-2111  [001] ....   237.343764: rtcpu_vinotify_event: tstamp:7831413098 tag:FS channel:0x03 frame:0 vi_tstamp:7831291995 data:0x00000010
     kworker/1:3-2111  [001] ....   237.343764: rtcpu_vinotify_event: tstamp:7831413224 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7831292439 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.399753: rtos_queue_peek_from_isr_failed: tstamp:7831699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   237.399757: rtcpu_vinotify_event: tstamp:7832334018 tag:FE channel:0x03 frame:0 vi_tstamp:7832243813 data:0x00000020
     kworker/1:3-2111  [001] ....   237.399758: rtcpu_vinotify_event: tstamp:7832334146 tag:FS channel:0x03 frame:0 vi_tstamp:7832333267 data:0x00000010
     kworker/1:3-2111  [001] ....   237.399759: rtcpu_vinotify_error: tstamp:7832334516 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7832333711 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.399760: rtcpu_vinotify_event: tstamp:7832639674 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7832333711 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.455753: rtcpu_vinotify_error: tstamp:7833375635 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7833374982 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.455757: rtcpu_vinotify_event: tstamp:7833583185 tag:FE channel:0x03 frame:0 vi_tstamp:7833285084 data:0x00000020
     kworker/1:3-2111  [001] ....   237.455758: rtcpu_vinotify_event: tstamp:7833583326 tag:FS channel:0x03 frame:0 vi_tstamp:7833374539 data:0x00000010
     kworker/1:3-2111  [001] ....   237.455759: rtcpu_vinotify_event: tstamp:7833583452 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7833374982 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.455759: rtcpu_vinotify_error: tstamp:7834416907 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7834416256 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.455760: rtcpu_vinotify_event: tstamp:7834526798 tag:FE channel:0x03 frame:0 vi_tstamp:7834326358 data:0x00000020
     kworker/1:3-2111  [001] ....   237.455761: rtcpu_vinotify_event: tstamp:7834526923 tag:FS channel:0x03 frame:0 vi_tstamp:7834415813 data:0x00000010
     kworker/1:3-2111  [001] ....   237.455761: rtcpu_vinotify_event: tstamp:7834527066 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7834416256 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.511768: rtcpu_vinotify_error: tstamp:7835458185 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7835457529 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.511771: rtcpu_vinotify_event: tstamp:7835470221 tag:FE channel:0x03 frame:0 vi_tstamp:7835367630 data:0x00000020
     kworker/1:3-2111  [001] ....   237.511771: rtcpu_vinotify_event: tstamp:7835470362 tag:FS channel:0x03 frame:0 vi_tstamp:7835457086 data:0x00000010
     kworker/1:3-2111  [001] ....   237.511772: rtcpu_vinotify_event: tstamp:7835470488 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7835457529 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.511773: rtcpu_vinotify_event: tstamp:7836413373 tag:FE channel:0x03 frame:0 vi_tstamp:7836408902 data:0x00000020
     kworker/1:3-2111  [001] ....   237.511773: rtcpu_vinotify_error: tstamp:7836499456 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7836498800 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.511775: rtos_queue_peek_from_isr_failed: tstamp:7836699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   237.511776: rtcpu_vinotify_event: tstamp:7836727730 tag:FS channel:0x03 frame:0 vi_tstamp:7836498357 data:0x00000010
     kworker/1:3-2111  [001] ....   237.511776: rtcpu_vinotify_event: tstamp:7836727870 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7836498800 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.567751: rtcpu_vinotify_error: tstamp:7837540723 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7837540073 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.567753: rtcpu_vinotify_event: tstamp:7837540982 tag:FE channel:0x03 frame:0 vi_tstamp:7837450173 data:0x00000020
     kworker/1:3-2111  [001] ....   237.567754: rtcpu_vinotify_event: tstamp:7837541123 tag:FS channel:0x03 frame:0 vi_tstamp:7837539629 data:0x00000010
     kworker/1:3-2111  [001] ....   237.567755: rtcpu_vinotify_event: tstamp:7837541248 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7837540073 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.567755: rtcpu_vinotify_event: tstamp:7838520418 tag:FE channel:0x03 frame:0 vi_tstamp:7838491448 data:0x00000020
     kworker/1:3-2111  [001] ....   237.627764: rtcpu_vinotify_error: tstamp:7838582005 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7838581347 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.627766: rtcpu_vinotify_event: tstamp:7838834930 tag:FS channel:0x03 frame:0 vi_tstamp:7838580903 data:0x00000010
     kworker/1:3-2111  [001] ....   237.627767: rtcpu_vinotify_event: tstamp:7838835073 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7838581347 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.627768: rtcpu_vinotify_error: tstamp:7839623272 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7839622620 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.627768: rtcpu_vinotify_event: tstamp:7839778511 tag:FE channel:0x03 frame:0 vi_tstamp:7839532720 data:0x00000020
     kworker/1:3-2111  [001] ....   237.627769: rtcpu_vinotify_event: tstamp:7839778652 tag:FS channel:0x03 frame:0 vi_tstamp:7839622176 data:0x00000010
     kworker/1:3-2111  [001] ....   237.627770: rtcpu_vinotify_event: tstamp:7839778779 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7839622620 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.683752: rtcpu_vinotify_error: tstamp:7840664548 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7840663891 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.683754: rtcpu_vinotify_event: tstamp:7840722119 tag:FE channel:0x03 frame:0 vi_tstamp:7840573992 data:0x00000020
     kworker/1:3-2111  [001] ....   237.683755: rtcpu_vinotify_event: tstamp:7840722245 tag:FS channel:0x03 frame:0 vi_tstamp:7840663447 data:0x00000010
     kworker/1:3-2111  [001] ....   237.683756: rtcpu_vinotify_event: tstamp:7840722389 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7840663891 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.683757: rtcpu_vinotify_event: tstamp:7841665760 tag:FE channel:0x03 frame:0 vi_tstamp:7841615264 data:0x00000020
     kworker/1:3-2111  [001] ....   237.683759: rtos_queue_peek_from_isr_failed: tstamp:7841699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   237.683759: rtcpu_vinotify_error: tstamp:7841705815 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7841705162 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.683760: rtcpu_vinotify_event: tstamp:7841980165 tag:FS channel:0x03 frame:0 vi_tstamp:7841704718 data:0x00000010
     kworker/1:3-2111  [001] ....   237.683761: rtcpu_vinotify_event: tstamp:7841980291 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7841705162 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.739760: rtcpu_vinotify_error: tstamp:7842747087 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7842746434 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.739763: rtcpu_vinotify_event: tstamp:7842923651 tag:FE channel:0x03 frame:0 vi_tstamp:7842656536 data:0x00000020
     kworker/1:3-2111  [001] ....   237.739764: rtcpu_vinotify_event: tstamp:7842923778 tag:FS channel:0x03 frame:0 vi_tstamp:7842745990 data:0x00000010
     kworker/1:3-2111  [001] ....   237.739764: rtcpu_vinotify_event: tstamp:7842923920 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7842746434 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.739765: rtcpu_vinotify_event: tstamp:7843698325 tag:FE channel:0x03 frame:0 vi_tstamp:7843697810 data:0x00000020
     kworker/1:3-2111  [001] ....   237.739766: rtcpu_vinotify_error: tstamp:7843788361 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7843787709 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.795750: rtcpu_vinotify_event: tstamp:7843993232 tag:FS channel:0x03 frame:0 vi_tstamp:7843787266 data:0x00000010
     kworker/1:3-2111  [001] ....   237.795752: rtcpu_vinotify_event: tstamp:7843993359 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7843787709 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.795753: rtcpu_vinotify_error: tstamp:7844829634 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7844828981 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.795754: rtcpu_vinotify_event: tstamp:7844937102 tag:FE channel:0x03 frame:0 vi_tstamp:7844739082 data:0x00000020
     kworker/1:3-2111  [001] ....   237.795755: rtcpu_vinotify_event: tstamp:7844937228 tag:FS channel:0x03 frame:0 vi_tstamp:7844828538 data:0x00000010
     kworker/1:3-2111  [001] ....   237.795755: rtcpu_vinotify_event: tstamp:7844937370 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7844828981 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.855755: rtcpu_vinotify_error: tstamp:7845870908 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7845870252 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.855757: rtcpu_vinotify_event: tstamp:7845880754 tag:FE channel:0x03 frame:0 vi_tstamp:7845780354 data:0x00000020
     kworker/1:3-2111  [001] ....   237.855758: rtcpu_vinotify_event: tstamp:7845880895 tag:FS channel:0x03 frame:0 vi_tstamp:7845869809 data:0x00000010
     kworker/1:3-2111  [001] ....   237.855758: rtcpu_vinotify_event: tstamp:7845881021 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7845870252 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.855760: rtos_queue_peek_from_isr_failed: tstamp:7846699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   237.855761: rtcpu_vinotify_event: tstamp:7846824340 tag:FE channel:0x03 frame:0 vi_tstamp:7846821626 data:0x00000020
     kworker/1:3-2111  [001] ....   237.855761: rtcpu_vinotify_error: tstamp:7846912180 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7846911524 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.855762: rtcpu_vinotify_event: tstamp:7847138894 tag:FS channel:0x03 frame:0 vi_tstamp:7846911081 data:0x00000010
     kworker/1:3-2111  [001] ....   237.855763: rtcpu_vinotify_event: tstamp:7847139036 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7846911524 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.911755: rtcpu_vinotify_error: tstamp:7847953452 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7847952799 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.911757: rtcpu_vinotify_event: tstamp:7848082543 tag:FE channel:0x03 frame:0 vi_tstamp:7847862901 data:0x00000020
     kworker/1:3-2111  [001] ....   237.911758: rtcpu_vinotify_event: tstamp:7848082686 tag:FS channel:0x03 frame:0 vi_tstamp:7847952356 data:0x00000010
     kworker/1:3-2111  [001] ....   237.911759: rtcpu_vinotify_event: tstamp:7848082810 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7847952799 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.911760: rtcpu_vinotify_event: tstamp:7848994375 tag:FE channel:0x03 frame:0 vi_tstamp:7848904172 data:0x00000020
     kworker/1:3-2111  [001] ....   237.911760: rtcpu_vinotify_event: tstamp:7848994501 tag:FS channel:0x03 frame:0 vi_tstamp:7848993628 data:0x00000010
     kworker/1:3-2111  [001] ....   237.911762: rtcpu_vinotify_error: tstamp:7848994871 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7848994071 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.911762: rtcpu_vinotify_event: tstamp:7849277880 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7848994071 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.967788: rtcpu_vinotify_error: tstamp:7850035996 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7850035343 data:0x000003c9
     kworker/1:3-2111  [001] ....   237.967790: rtcpu_vinotify_event: tstamp:7850221655 tag:FE channel:0x03 frame:0 vi_tstamp:7849945444 data:0x00000020
     kworker/1:3-2111  [001] ....   237.967791: rtcpu_vinotify_event: tstamp:7850221796 tag:FS channel:0x03 frame:0 vi_tstamp:7850034899 data:0x00000010
     kworker/1:3-2111  [001] ....   237.967792: rtcpu_vinotify_event: tstamp:7850221922 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7850035343 data:0x000003c9
 vi-output, ar02-15192 [004] ....   238.016829: tegra_channel_capture_frame: sof:250.401433664
 vi-output, ar02-15192 [004] ....   238.016834: tegra_channel_capture_frame: eof:250.431877728
     kworker/1:3-2111  [001] ....   238.023765: rtcpu_vinotify_error: tstamp:7851077268 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7851076614 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.023768: rtcpu_vinotify_event: tstamp:7851165195 tag:FE channel:0x03 frame:0 vi_tstamp:7850986715 data:0x00000020
     kworker/1:3-2111  [001] ....   238.023769: rtcpu_vinotify_event: tstamp:7851165319 tag:FS channel:0x03 frame:0 vi_tstamp:7851076171 data:0x00000010
     kworker/1:3-2111  [001] ....   238.023770: rtcpu_vinotify_event: tstamp:7851165462 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7851076614 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.023772: rtos_queue_peek_from_isr_failed: tstamp:7851699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   238.023773: rtcpu_vinotify_event: tstamp:7852108902 tag:FE channel:0x03 frame:0 vi_tstamp:7852027991 data:0x00000020
     kworker/1:3-2111  [001] ....   238.023773: rtcpu_vinotify_error: tstamp:7852118538 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7852117890 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.023774: rtcpu_vinotify_event: tstamp:7852423465 tag:FS channel:0x03 frame:0 vi_tstamp:7852117446 data:0x00000010
     kworker/1:3-2111  [001] ....   238.023774: rtcpu_vinotify_event: tstamp:7852423591 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7852117890 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.079771: rtcpu_vinotify_error: tstamp:7853159812 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7853159161 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.079775: rtcpu_vinotify_event: tstamp:7853367110 tag:FE channel:0x03 frame:0 vi_tstamp:7853069262 data:0x00000020
     kworker/1:3-2111  [001] ....   238.079776: rtcpu_vinotify_event: tstamp:7853367236 tag:FS channel:0x03 frame:0 vi_tstamp:7853158717 data:0x00000010
     kworker/1:3-2111  [001] ....   238.079777: rtcpu_vinotify_event: tstamp:7853367377 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7853159161 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.079778: rtcpu_vinotify_error: tstamp:7854201083 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7854200433 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.079778: rtcpu_vinotify_event: tstamp:7854201340 tag:FE channel:0x03 frame:0 vi_tstamp:7854110534 data:0x00000020
     kworker/1:3-2111  [001] ....   238.079779: rtcpu_vinotify_event: tstamp:7854201485 tag:FS channel:0x03 frame:0 vi_tstamp:7854199989 data:0x00000010
     kworker/1:3-2111  [001] ....   238.079780: rtcpu_vinotify_event: tstamp:7854201608 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7854200433 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.135783: rtcpu_vinotify_event: tstamp:7855160217 tag:FE channel:0x03 frame:0 vi_tstamp:7855151806 data:0x00000020
     kworker/1:3-2111  [001] ....   238.135787: rtcpu_vinotify_error: tstamp:7855242359 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7855241704 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.135788: rtcpu_vinotify_event: tstamp:7855474771 tag:FS channel:0x03 frame:0 vi_tstamp:7855241261 data:0x00000010
     kworker/1:3-2111  [001] ....   238.135789: rtcpu_vinotify_event: tstamp:7855474912 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7855241704 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.135789: rtcpu_vinotify_error: tstamp:7856283630 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7856282978 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.191773: rtcpu_vinotify_event: tstamp:7856418365 tag:FE channel:0x03 frame:0 vi_tstamp:7856193080 data:0x00000020
     kworker/1:3-2111  [001] ....   238.191777: rtcpu_vinotify_event: tstamp:7856418508 tag:FS channel:0x03 frame:0 vi_tstamp:7856282535 data:0x00000010
     kworker/1:3-2111  [001] ....   238.191778: rtcpu_vinotify_event: tstamp:7856418632 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7856282978 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.191779: rtos_queue_peek_from_isr_failed: tstamp:7856699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   238.191780: rtcpu_vinotify_error: tstamp:7857324907 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7857324251 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.191781: rtcpu_vinotify_event: tstamp:7857361926 tag:FE channel:0x03 frame:0 vi_tstamp:7857234353 data:0x00000020
     kworker/1:3-2111  [001] ....   238.191782: rtcpu_vinotify_event: tstamp:7857362051 tag:FS channel:0x03 frame:0 vi_tstamp:7857323808 data:0x00000010
     kworker/1:3-2111  [001] ....   238.191782: rtcpu_vinotify_event: tstamp:7857362193 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7857324251 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.247756: rtcpu_vinotify_event: tstamp:7858305408 tag:FE channel:0x03 frame:0 vi_tstamp:7858275623 data:0x00000020
     kworker/1:3-2111  [001] ....   238.247763: rtcpu_vinotify_error: tstamp:7858366174 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7858365522 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.247764: rtcpu_vinotify_event: tstamp:7858619901 tag:FS channel:0x03 frame:0 vi_tstamp:7858365079 data:0x00000010
     kworker/1:3-2111  [001] ....   238.247765: rtcpu_vinotify_event: tstamp:7858620026 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7858365522 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.247766: rtcpu_vinotify_error: tstamp:7859407445 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7859406794 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.247766: rtcpu_vinotify_event: tstamp:7859563684 tag:FE channel:0x03 frame:0 vi_tstamp:7859316895 data:0x00000020
     kworker/1:3-2111  [001] ....   238.247767: rtcpu_vinotify_event: tstamp:7859563810 tag:FS channel:0x03 frame:0 vi_tstamp:7859406350 data:0x00000010
     kworker/1:3-2111  [001] ....   238.247767: rtcpu_vinotify_event: tstamp:7859563950 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7859406794 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.303757: rtcpu_vinotify_event: tstamp:7860358684 tag:FE channel:0x03 frame:0 vi_tstamp:7860358170 data:0x00000020
     kworker/1:3-2111  [001] ....   238.303760: rtcpu_vinotify_error: tstamp:7860448723 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7860448069 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.303761: rtcpu_vinotify_event: tstamp:7860664916 tag:FS channel:0x03 frame:0 vi_tstamp:7860447625 data:0x00000010
     kworker/1:3-2111  [001] ....   238.303762: rtcpu_vinotify_event: tstamp:7860665041 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7860448069 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.303762: rtcpu_vinotify_error: tstamp:7861489993 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7861489342 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.359750: rtcpu_vinotify_event: tstamp:7861608775 tag:FE channel:0x03 frame:0 vi_tstamp:7861399443 data:0x00000020
     kworker/1:3-2111  [001] ....   238.359753: rtcpu_vinotify_event: tstamp:7861608901 tag:FS channel:0x03 frame:0 vi_tstamp:7861488898 data:0x00000010
     kworker/1:3-2111  [001] ....   238.359754: rtcpu_vinotify_event: tstamp:7861609042 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7861489342 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.359756: rtos_queue_peek_from_isr_failed: tstamp:7861699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   238.359757: rtcpu_vinotify_error: tstamp:7862531267 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7862530613 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.359758: rtcpu_vinotify_event: tstamp:7862552482 tag:FE channel:0x03 frame:0 vi_tstamp:7862440714 data:0x00000020
     kworker/1:3-2111  [001] ....   238.359758: rtcpu_vinotify_event: tstamp:7862552625 tag:FS channel:0x03 frame:0 vi_tstamp:7862530170 data:0x00000010
     kworker/1:3-2111  [001] ....   238.359759: rtcpu_vinotify_event: tstamp:7862552749 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7862530613 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.359760: rtos_queue_send_from_isr_failed: tstamp:7862941860 queue:0x0bcb2b38
     kworker/1:3-2111  [001] ....   238.359761: rtos_queue_send_from_isr_failed: tstamp:7862942000 queue:0x0bcb73a0
     kworker/1:3-2111  [001] ....   238.359761: rtos_queue_send_from_isr_failed: tstamp:7862942140 queue:0x0bcb8f20
     kworker/1:3-2111  [001] ....   238.359762: rtos_queue_send_from_isr_failed: tstamp:7862942274 queue:0x0bcb9ce0
     kworker/1:3-2111  [001] ....   238.359762: rtos_queue_send_from_isr_failed: tstamp:7862942407 queue:0x0bcbaaa0
     kworker/1:3-2111  [001] ....   238.359763: rtcpu_vinotify_event: tstamp:7863181724 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:6801064992 data:0x10000000
     kworker/1:3-2111  [001] ....   238.359764: rtcpu_vinotify_event: tstamp:7863181853 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:6801068800 data:0x31000002
     kworker/1:3-2111  [001] ....   238.359764: rtcpu_vinotify_event: tstamp:7863181997 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:6801069792 data:0x07020002
     kworker/1:3-2111  [001] ....   238.415765: rtcpu_vinotify_event: tstamp:7863496365 tag:FE channel:0x03 frame:0 vi_tstamp:7863481986 data:0x00000020
     kworker/1:3-2111  [001] ....   238.415767: rtcpu_vinotify_event: tstamp:7863572726 tag:FS channel:0x03 frame:0 vi_tstamp:7863571441 data:0x00000010
     kworker/1:3-2111  [001] ....   238.415768: rtcpu_vinotify_event: tstamp:7863572856 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:7863571441 data:0x00000000
     kworker/1:3-2111  [001] ....   238.415769: rtcpu_vinotify_event: tstamp:7863573000 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:7863571885 data:0x00000001
     kworker/1:3-2111  [001] ....   238.415769: rtcpu_vinotify_event: tstamp:7863573129 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:6821164960 data:0x08020002
     kworker/1:3-2111  [001] ....   238.415775: rtcpu_vinotify_event: tstamp:7864534824 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:7864523231 data:0x04b70002
     kworker/1:3-2111  [001] ....   238.415776: rtcpu_vinotify_event: tstamp:7864534951 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:7864523245 data:0x00000000
     kworker/1:3-2111  [001] ....   238.415776: rtcpu_vinotify_event: tstamp:7864535096 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:6851608448 data:0x02020002
     kworker/1:3-2111  [001] ....   238.415777: rtcpu_vinotify_event: tstamp:7864535221 tag:FE channel:0x03 frame:0 vi_tstamp:7864523258 data:0x00000020
     kworker/1:3-2111  [001] ....   238.415778: rtcpu_vinotify_event: tstamp:7864535364 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:7864523258 data:0x00000000
     kworker/1:3-2111  [001] ....   238.415799: rtcpu_vinotify_event: tstamp:7864535487 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:6851611360 data:0x00020002
     kworker/1:3-2111  [001] ....   238.415799: rtcpu_vinotify_error: tstamp:7864613826 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7864613159 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.415800: rtcpu_vinotify_event: tstamp:7864849461 tag:FS channel:0x03 frame:0 vi_tstamp:7864612716 data:0x00000010
     kworker/1:3-2111  [001] ....   238.415801: rtcpu_vinotify_event: tstamp:7864849584 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7864613159 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.475766: rtcpu_vinotify_error: tstamp:7865655083 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7865654432 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.475769: rtcpu_vinotify_event: tstamp:7865793463 tag:FE channel:0x03 frame:0 vi_tstamp:7865564533 data:0x00000020
     kworker/1:3-2111  [001] ....   238.475770: rtcpu_vinotify_event: tstamp:7865793590 tag:FS channel:0x03 frame:0 vi_tstamp:7865653988 data:0x00000010
     kworker/1:3-2111  [001] ....   238.475771: rtcpu_vinotify_event: tstamp:7865793732 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7865654432 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.475772: rtcpu_vinotify_error: tstamp:7866696357 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7866695703 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.475773: rtos_queue_peek_from_isr_failed: tstamp:7866699525 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   238.475774: rtcpu_vinotify_event: tstamp:7866737613 tag:FE channel:0x03 frame:0 vi_tstamp:7866605805 data:0x00000020
     kworker/1:3-2111  [001] ....   238.475775: rtcpu_vinotify_event: tstamp:7866737755 tag:FS channel:0x03 frame:0 vi_tstamp:7866695260 data:0x00000010
     kworker/1:3-2111  [001] ....   238.475775: rtcpu_vinotify_event: tstamp:7866737882 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7866695703 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.531758: rtcpu_vinotify_event: tstamp:7867681534 tag:FE channel:0x03 frame:0 vi_tstamp:7867647076 data:0x00000020
     kworker/1:3-2111  [001] ....   238.531760: rtcpu_vinotify_event: tstamp:7867737281 tag:FS channel:0x03 frame:0 vi_tstamp:7867736531 data:0x00000010
     kworker/1:3-2111  [001] ....   238.531761: rtcpu_vinotify_error: tstamp:7867737652 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7867736975 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.531762: rtcpu_vinotify_event: tstamp:7868027503 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7867736975 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.591790: rtcpu_vinotify_error: tstamp:7868778901 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7868778249 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.591793: rtcpu_vinotify_event: tstamp:7868971231 tag:FE channel:0x03 frame:0 vi_tstamp:7868688348 data:0x00000020
     kworker/1:3-2111  [001] ....   238.591794: rtcpu_vinotify_event: tstamp:7868971371 tag:FS channel:0x03 frame:0 vi_tstamp:7868777806 data:0x00000010
     kworker/1:3-2111  [001] ....   238.591795: rtcpu_vinotify_event: tstamp:7868971538 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7868778249 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.591796: rtcpu_vinotify_error: tstamp:7869820174 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7869819521 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.591797: rtcpu_vinotify_event: tstamp:7869915125 tag:FE channel:0x03 frame:0 vi_tstamp:7869729623 data:0x00000020
     kworker/1:3-2111  [001] ....   238.591797: rtcpu_vinotify_event: tstamp:7869915248 tag:FS channel:0x03 frame:0 vi_tstamp:7869819078 data:0x00000010
     kworker/1:3-2111  [001] ....   238.591798: rtcpu_vinotify_event: tstamp:7869915388 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7869819521 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.647763: rtcpu_vinotify_event: tstamp:7870859071 tag:FE channel:0x03 frame:0 vi_tstamp:7870770895 data:0x00000020
     kworker/1:3-2111  [001] ....   238.647765: rtcpu_vinotify_error: tstamp:7870861441 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7870860793 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.647766: rtcpu_vinotify_event: tstamp:7871173652 tag:FS channel:0x03 frame:0 vi_tstamp:7870860350 data:0x00000010
     kworker/1:3-2111  [001] ....   238.647767: rtcpu_vinotify_event: tstamp:7871173775 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7870860793 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.647788: rtos_queue_peek_from_isr_failed: tstamp:7871699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   238.647789: rtcpu_vinotify_error: tstamp:7871902717 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7871902065 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.647790: rtcpu_vinotify_event: tstamp:7872117473 tag:FE channel:0x03 frame:0 vi_tstamp:7871812166 data:0x00000020
     kworker/1:3-2111  [001] ....   238.647790: rtcpu_vinotify_event: tstamp:7872117598 tag:FS channel:0x03 frame:0 vi_tstamp:7871901622 data:0x00000010
     kworker/1:3-2111  [001] ....   238.647791: rtcpu_vinotify_event: tstamp:7872117739 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7871902065 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.707773: rtcpu_vinotify_error: tstamp:7872943989 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7872943337 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.707775: rtcpu_vinotify_event: tstamp:7872944246 tag:FE channel:0x03 frame:0 vi_tstamp:7872853437 data:0x00000020
     kworker/1:3-2111  [001] ....   238.707776: rtcpu_vinotify_event: tstamp:7872944387 tag:FS channel:0x03 frame:0 vi_tstamp:7872942894 data:0x00000010
     kworker/1:3-2111  [001] ....   238.707777: rtcpu_vinotify_event: tstamp:7872944510 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7872943337 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.707777: rtcpu_vinotify_event: tstamp:7873910868 tag:FE channel:0x03 frame:0 vi_tstamp:7873894714 data:0x00000020
     kworker/1:3-2111  [001] ....   238.707778: rtcpu_vinotify_error: tstamp:7873985268 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7873984612 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.767752: rtcpu_vinotify_event: tstamp:7874225491 tag:FS channel:0x03 frame:0 vi_tstamp:7873984168 data:0x00000010
     kworker/1:3-2111  [001] ....   238.767754: rtcpu_vinotify_event: tstamp:7874225633 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7873984612 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.767755: rtcpu_vinotify_error: tstamp:7875026537 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7875025884 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.767756: rtcpu_vinotify_event: tstamp:7875169383 tag:FE channel:0x03 frame:0 vi_tstamp:7874935985 data:0x00000020
     kworker/1:3-2111  [001] ....   238.767757: rtcpu_vinotify_event: tstamp:7875169523 tag:FS channel:0x03 frame:0 vi_tstamp:7875025440 data:0x00000010
     kworker/1:3-2111  [001] ....   238.767757: rtcpu_vinotify_event: tstamp:7875169649 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7875025884 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.823751: rtcpu_vinotify_error: tstamp:7876067809 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7876067155 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.823754: rtcpu_vinotify_event: tstamp:7876113188 tag:FE channel:0x03 frame:0 vi_tstamp:7875977257 data:0x00000020
     kworker/1:3-2111  [001] ....   238.823755: rtcpu_vinotify_event: tstamp:7876113313 tag:FS channel:0x03 frame:0 vi_tstamp:7876066711 data:0x00000010
     kworker/1:3-2111  [001] ....   238.823755: rtcpu_vinotify_event: tstamp:7876113456 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7876067155 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.823757: rtos_queue_peek_from_isr_failed: tstamp:7876699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   238.823758: rtcpu_vinotify_event: tstamp:7877057032 tag:FE channel:0x03 frame:0 vi_tstamp:7877018528 data:0x00000020
     kworker/1:3-2111  [001] ....   238.823758: rtcpu_vinotify_error: tstamp:7877109081 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7877108427 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.823759: rtcpu_vinotify_event: tstamp:7877371704 tag:FS channel:0x03 frame:0 vi_tstamp:7877107984 data:0x00000010
     kworker/1:3-2111  [001] ....   238.823760: rtcpu_vinotify_event: tstamp:7877371830 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7877108427 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.879741: rtcpu_vinotify_error: tstamp:7878150357 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7878149701 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.879743: rtcpu_vinotify_event: tstamp:7878315603 tag:FE channel:0x03 frame:0 vi_tstamp:7878059804 data:0x00000020
     kworker/1:3-2111  [001] ....   238.879744: rtcpu_vinotify_event: tstamp:7878315729 tag:FS channel:0x03 frame:0 vi_tstamp:7878149258 data:0x00000010
     kworker/1:3-2111  [001] ....   238.879745: rtcpu_vinotify_event: tstamp:7878315871 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7878149701 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.879745: rtcpu_vinotify_event: tstamp:7879101591 tag:FE channel:0x03 frame:0 vi_tstamp:7879101075 data:0x00000020
     kworker/1:3-2111  [001] ....   238.879746: rtcpu_vinotify_error: tstamp:7879191629 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7879190974 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.879747: rtcpu_vinotify_event: tstamp:7879385375 tag:FS channel:0x03 frame:0 vi_tstamp:7879190531 data:0x00000010
     kworker/1:3-2111  [001] ....   238.879747: rtcpu_vinotify_event: tstamp:7879385501 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7879190974 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.935756: rtcpu_vinotify_error: tstamp:7880232900 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7880232245 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.935760: rtcpu_vinotify_event: tstamp:7880329221 tag:FE channel:0x03 frame:0 vi_tstamp:7880142347 data:0x00000020
     kworker/1:3-2111  [001] ....   238.935761: rtcpu_vinotify_event: tstamp:7880329347 tag:FS channel:0x03 frame:0 vi_tstamp:7880231803 data:0x00000010
     kworker/1:3-2111  [001] ....   238.935762: rtcpu_vinotify_event: tstamp:7880329492 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7880232245 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.935763: rtcpu_vinotify_event: tstamp:7881272838 tag:FE channel:0x03 frame:0 vi_tstamp:7881183618 data:0x00000020
     kworker/1:3-2111  [001] ....   238.935763: rtcpu_vinotify_error: tstamp:7881274163 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7881273517 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.991764: rtcpu_vinotify_event: tstamp:7881587403 tag:FS channel:0x03 frame:0 vi_tstamp:7881273074 data:0x00000010
     kworker/1:3-2111  [001] ....   238.991768: rtcpu_vinotify_event: tstamp:7881587529 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7881273517 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.991770: rtos_queue_peek_from_isr_failed: tstamp:7881699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   238.991771: rtcpu_vinotify_error: tstamp:7882315488 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7882314792 data:0x000003c9
     kworker/1:3-2111  [001] ....   238.991772: rtcpu_vinotify_event: tstamp:7882531026 tag:FE channel:0x03 frame:0 vi_tstamp:7882224894 data:0x00000020
     kworker/1:3-2111  [001] ....   238.991773: rtcpu_vinotify_event: tstamp:7882531152 tag:FS channel:0x03 frame:0 vi_tstamp:7882314348 data:0x00000010
     kworker/1:3-2111  [001] ....   238.991773: rtcpu_vinotify_event: tstamp:7882531292 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7882314792 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.047752: rtcpu_vinotify_error: tstamp:7883356716 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7883356064 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.047755: rtcpu_vinotify_event: tstamp:7883474858 tag:FE channel:0x03 frame:0 vi_tstamp:7883266165 data:0x00000020
     kworker/1:3-2111  [001] ....   239.047756: rtcpu_vinotify_event: tstamp:7883474999 tag:FS channel:0x03 frame:0 vi_tstamp:7883355621 data:0x00000010
     kworker/1:3-2111  [001] ....   239.047757: rtcpu_vinotify_event: tstamp:7883475125 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7883356064 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.047776: rtcpu_vinotify_event: tstamp:7884397641 tag:FE channel:0x03 frame:0 vi_tstamp:7884307436 data:0x00000020
     kworker/1:3-2111  [001] ....   239.047776: rtcpu_vinotify_event: tstamp:7884397768 tag:FS channel:0x03 frame:0 vi_tstamp:7884396892 data:0x00000010
     kworker/1:3-2111  [001] ....   239.047777: rtcpu_vinotify_error: tstamp:7884398138 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7884397336 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.047778: rtcpu_vinotify_event: tstamp:7884701817 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7884397336 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.103756: rtcpu_vinotify_error: tstamp:7885439261 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7885438608 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.103759: rtcpu_vinotify_event: tstamp:7885645640 tag:FE channel:0x03 frame:0 vi_tstamp:7885348708 data:0x00000020
     kworker/1:3-2111  [001] ....   239.103760: rtcpu_vinotify_event: tstamp:7885645782 tag:FS channel:0x03 frame:0 vi_tstamp:7885438164 data:0x00000010
     kworker/1:3-2111  [001] ....   239.103761: rtcpu_vinotify_event: tstamp:7885645908 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7885438608 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.103761: rtcpu_vinotify_error: tstamp:7886480535 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7886479882 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.159762: rtcpu_vinotify_event: tstamp:7886589545 tag:FE channel:0x03 frame:0 vi_tstamp:7886389983 data:0x00000020
     kworker/1:3-2111  [001] ....   239.159765: rtcpu_vinotify_event: tstamp:7886589671 tag:FS channel:0x03 frame:0 vi_tstamp:7886479438 data:0x00000010
     kworker/1:3-2111  [001] ....   239.159766: rtcpu_vinotify_event: tstamp:7886589811 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7886479882 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.159768: rtos_queue_peek_from_isr_failed: tstamp:7886699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   239.159769: rtcpu_vinotify_error: tstamp:7887521808 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7887521155 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.159769: rtcpu_vinotify_event: tstamp:7887533400 tag:FE channel:0x03 frame:0 vi_tstamp:7887431256 data:0x00000020
     kworker/1:3-2111  [001] ....   239.159786: rtcpu_vinotify_event: tstamp:7887533543 tag:FS channel:0x03 frame:0 vi_tstamp:7887520710 data:0x00000010
     kworker/1:3-2111  [001] ....   239.159786: rtcpu_vinotify_event: tstamp:7887533667 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7887521155 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.215780: rtcpu_vinotify_event: tstamp:7888477464 tag:FE channel:0x03 frame:0 vi_tstamp:7888472528 data:0x00000020
     kworker/1:3-2111  [001] ....   239.215782: rtcpu_vinotify_error: tstamp:7888563081 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7888562426 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.215783: rtcpu_vinotify_event: tstamp:7888792166 tag:FS channel:0x03 frame:0 vi_tstamp:7888561982 data:0x00000010
     kworker/1:3-2111  [001] ....   239.215784: rtcpu_vinotify_event: tstamp:7888792306 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7888562426 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.215785: rtcpu_vinotify_error: tstamp:7889604352 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7889603697 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.215785: rtcpu_vinotify_event: tstamp:7889604610 tag:FE channel:0x03 frame:0 vi_tstamp:7889513798 data:0x00000020
     kworker/1:3-2111  [001] ....   239.215786: rtcpu_vinotify_event: tstamp:7889604752 tag:FS channel:0x03 frame:0 vi_tstamp:7889603254 data:0x00000010
     kworker/1:3-2111  [001] ....   239.215786: rtcpu_vinotify_event: tstamp:7889604876 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7889603697 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.271764: rtcpu_vinotify_event: tstamp:7890585592 tag:FE channel:0x03 frame:0 vi_tstamp:7890555073 data:0x00000020
     kworker/1:3-2111  [001] ....   239.271768: rtcpu_vinotify_error: tstamp:7890645627 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7890644972 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.271769: rtcpu_vinotify_event: tstamp:7890900152 tag:FS channel:0x03 frame:0 vi_tstamp:7890644529 data:0x00000010
     kworker/1:3-2111  [001] ....   239.271770: rtcpu_vinotify_event: tstamp:7890900294 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7890644972 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.271770: rtcpu_vinotify_error: tstamp:7891686899 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7891686245 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.271772: rtos_queue_peek_from_isr_failed: tstamp:7891699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   239.327766: rtcpu_vinotify_event: tstamp:7891844077 tag:FE channel:0x03 frame:0 vi_tstamp:7891596346 data:0x00000020
     kworker/1:3-2111  [001] ....   239.327770: rtcpu_vinotify_event: tstamp:7891844221 tag:FS channel:0x03 frame:0 vi_tstamp:7891685802 data:0x00000010
     kworker/1:3-2111  [001] ....   239.327770: rtcpu_vinotify_event: tstamp:7891844348 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7891686245 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.327771: rtcpu_vinotify_error: tstamp:7892728170 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7892727516 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.327773: rtcpu_vinotify_event: tstamp:7892788113 tag:FE channel:0x03 frame:0 vi_tstamp:7892637617 data:0x00000020
     kworker/1:3-2111  [001] ....   239.327773: rtcpu_vinotify_event: tstamp:7892788238 tag:FS channel:0x03 frame:0 vi_tstamp:7892727073 data:0x00000010
     kworker/1:3-2111  [001] ....   239.327774: rtcpu_vinotify_event: tstamp:7892788382 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7892727516 data:0x000003c9
 vi-output, ar02-15192 [004] ....   239.343602: tegra_channel_capture_frame: sof:251.634300320
 vi-output, ar02-15192 [004] ....   239.349137: tegra_channel_capture_frame: eof:251.664744256
     kworker/1:3-2111  [001] ....   239.387760: rtcpu_vinotify_event: tstamp:7893731935 tag:FE channel:0x03 frame:0 vi_tstamp:7893678889 data:0x00000020
     kworker/1:3-2111  [001] ....   239.387763: rtcpu_vinotify_error: tstamp:7893769444 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7893768788 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.387764: rtcpu_vinotify_event: tstamp:7894046639 tag:FS channel:0x03 frame:0 vi_tstamp:7893768345 data:0x00000010
     kworker/1:3-2111  [001] ....   239.387765: rtcpu_vinotify_event: tstamp:7894046765 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7893768788 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.387766: rtcpu_vinotify_error: tstamp:7894810712 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7894810059 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.387766: rtcpu_vinotify_event: tstamp:7894990466 tag:FE channel:0x03 frame:0 vi_tstamp:7894720161 data:0x00000020
     kworker/1:3-2111  [001] ....   239.387767: rtcpu_vinotify_event: tstamp:7894990592 tag:FS channel:0x03 frame:0 vi_tstamp:7894809616 data:0x00000010
     kworker/1:3-2111  [001] ....   239.387767: rtcpu_vinotify_event: tstamp:7894990733 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7894810059 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.443760: rtcpu_vinotify_event: tstamp:7895761953 tag:FE channel:0x03 frame:0 vi_tstamp:7895761436 data:0x00000020
     kworker/1:3-2111  [001] ....   239.443765: rtcpu_vinotify_error: tstamp:7895851991 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7895851335 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.443766: rtcpu_vinotify_event: tstamp:7896060436 tag:FS channel:0x03 frame:0 vi_tstamp:7895850892 data:0x00000010
     kworker/1:3-2111  [001] ....   239.443767: rtcpu_vinotify_event: tstamp:7896060562 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7895851335 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.443769: rtos_queue_peek_from_isr_failed: tstamp:7896699534 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   239.443769: rtcpu_vinotify_error: tstamp:7896893263 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7896892607 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.443770: rtcpu_vinotify_event: tstamp:7897004304 tag:FE channel:0x03 frame:0 vi_tstamp:7896802708 data:0x00000020
     kworker/1:3-2111  [001] ....   239.443789: rtcpu_vinotify_event: tstamp:7897004430 tag:FS channel:0x03 frame:0 vi_tstamp:7896892163 data:0x00000010
     kworker/1:3-2111  [001] ....   239.443789: rtcpu_vinotify_event: tstamp:7897004571 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7896892607 data:0x000003c9
  gst-launch-1.0-15160 [005] ....   239.470438: tegra_channel_set_stream: enable : 0x0
  gst-launch-1.0-15160 [005] ....   239.470442: tegra_channel_set_stream: ar0231 2-0009 : 0x0
  gst-launch-1.0-15160 [001] ....   239.481016: tegra_channel_set_stream: rvnirp461 2-000a : 0x0
  gst-launch-1.0-15160 [001] ....   239.486446: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-15160 [001] ....   239.486448: csi_s_stream: enable : 0x0
     kworker/1:3-2111  [001] ....   239.500598: rtcpu_vinotify_error: tstamp:7897934531 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7897933878 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.500601: rtcpu_vinotify_event: tstamp:7897948549 tag:FE channel:0x03 frame:0 vi_tstamp:7897843979 data:0x00000020
     kworker/1:3-2111  [001] ....   239.500602: rtcpu_vinotify_event: tstamp:7897948695 tag:FS channel:0x03 frame:0 vi_tstamp:7897933434 data:0x00000010
     kworker/1:3-2111  [001] ....   239.500603: rtcpu_vinotify_event: tstamp:7897948820 tag:CHANSEL_NOMATCH channel:0xc1 frame:0 vi_tstamp:7897933878 data:0x000003c9
     kworker/1:3-2111  [001] ....   239.619763: rtos_queue_peek_from_isr_failed: tstamp:7901699535 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   239.677944: rtos_queue_send_from_isr_failed: tstamp:7902797346 queue:0x0bcb2b38
     kworker/1:3-2111  [001] ....   239.677946: rtos_queue_send_from_isr_failed: tstamp:7902797486 queue:0x0bcb73a0
     kworker/1:3-2111  [001] ....   239.677947: rtos_queue_send_from_isr_failed: tstamp:7902797626 queue:0x0bcb8f20
     kworker/1:3-2111  [001] ....   239.677948: rtos_queue_send_from_isr_failed: tstamp:7902797761 queue:0x0bcb9ce0
     kworker/1:3-2111  [001] ....   239.677949: rtos_queue_send_from_isr_failed: tstamp:7902797893 queue:0x0bcbaaa0
     kworker/1:3-2111  [001] ....   239.792565: rtos_queue_peek_from_isr_failed: tstamp:7906699535 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   239.912532: rtos_queue_peek_from_isr_failed: tstamp:7911699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   240.091638: rtos_queue_peek_from_isr_failed: tstamp:7916699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   240.260958: rtos_queue_peek_from_isr_failed: tstamp:7921699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   240.433598: rtos_queue_peek_from_isr_failed: tstamp:7926699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   240.553129: rtos_queue_peek_from_isr_failed: tstamp:7931699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   240.668533: rtos_queue_send_from_isr_failed: tstamp:7934478386 queue:0x0bcb2b38
     kworker/1:3-2111  [001] ....   240.668536: rtos_queue_send_from_isr_failed: tstamp:7934478525 queue:0x0bcb73a0
     kworker/1:3-2111  [001] ....   240.668536: rtos_queue_send_from_isr_failed: tstamp:7934478666 queue:0x0bcb8f20
     kworker/1:3-2111  [001] ....   240.668537: rtos_queue_send_from_isr_failed: tstamp:7934478801 queue:0x0bcb9ce0
     kworker/1:3-2111  [001] ....   240.668537: rtos_queue_send_from_isr_failed: tstamp:7934478933 queue:0x0bcbaaa0
     kworker/1:3-2111  [001] ....   240.668539: rtcpu_vinotify_event: tstamp:7934762428 tag:FE channel:0x03 frame:0 vi_tstamp:7934479425 data:0x00000020
     kworker/1:3-2111  [001] ....   240.724896: rtos_queue_peek_from_isr_failed: tstamp:7936699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   240.892349: rtos_queue_peek_from_isr_failed: tstamp:7941699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   241.063331: rtos_queue_peek_from_isr_failed: tstamp:7946699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   241.228091: rtos_queue_peek_from_isr_failed: tstamp:7951699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   241.397947: rtos_queue_peek_from_isr_failed: tstamp:7956699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   241.511688: rtos_queue_peek_from_isr_failed: tstamp:7961699528 queue:0x0bcbb8b8
  gst-launch-1.0-15160 [001] ....   241.583087: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-15160 [001] ....   241.583093: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-15160 [001] ....   241.583095: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-15160 [001] ....   241.583096: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-15160 [001] ....   241.583098: csi_s_power: enable : 0x0
     kworker/1:3-2111  [001] ....   241.675759: rtos_queue_peek_from_isr_failed: tstamp:7966699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   241.843809: rtos_queue_peek_from_isr_failed: tstamp:7971699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   242.011761: rtos_queue_peek_from_isr_failed: tstamp:7976699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   242.179807: rtos_queue_peek_from_isr_failed: tstamp:7981699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   242.347794: rtos_queue_peek_from_isr_failed: tstamp:7986699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   242.515757: rtos_queue_peek_from_isr_failed: tstamp:7991699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   242.683784: rtos_queue_peek_from_isr_failed: tstamp:7996699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   242.799767: rtos_queue_peek_from_isr_failed: tstamp:8001699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   242.971770: rtos_queue_peek_from_isr_failed: tstamp:8006699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   243.139758: rtos_queue_peek_from_isr_failed: tstamp:8011699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   243.314106: rtos_queue_peek_from_isr_failed: tstamp:8016699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   243.479759: rtos_queue_peek_from_isr_failed: tstamp:8021699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   243.591795: rtos_queue_peek_from_isr_failed: tstamp:8026699527 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   243.763760: rtos_queue_peek_from_isr_failed: tstamp:8031699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   243.935807: rtos_queue_peek_from_isr_failed: tstamp:8036699532 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   244.103782: rtos_queue_peek_from_isr_failed: tstamp:8041699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   244.271767: rtos_queue_peek_from_isr_failed: tstamp:8046699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   244.439778: rtos_queue_peek_from_isr_failed: tstamp:8051699527 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   244.551778: rtos_queue_peek_from_isr_failed: tstamp:8056699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   244.719762: rtos_queue_peek_from_isr_failed: tstamp:8061699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   244.887756: rtos_queue_peek_from_isr_failed: tstamp:8066699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   245.055781: rtos_queue_peek_from_isr_failed: tstamp:8071699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   245.223756: rtos_queue_peek_from_isr_failed: tstamp:8076699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   245.395773: rtos_queue_peek_from_isr_failed: tstamp:8081699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   245.511764: rtos_queue_peek_from_isr_failed: tstamp:8086699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   245.679797: rtos_queue_peek_from_isr_failed: tstamp:8091699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   245.847770: rtos_queue_peek_from_isr_failed: tstamp:8096699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   246.019755: rtos_queue_peek_from_isr_failed: tstamp:8101699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   246.195800: rtos_queue_peek_from_isr_failed: tstamp:8106699530 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   246.363776: rtos_queue_peek_from_isr_failed: tstamp:8111699531 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   246.475761: rtos_queue_peek_from_isr_failed: tstamp:8116699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   246.647756: rtos_queue_peek_from_isr_failed: tstamp:8121699528 queue:0x0bcbb8b8
     kworker/1:3-2111  [001] ....   246.647760: rtos_queue_peek_from_isr_failed: tstamp:8121941422 queue:0x0bcbb8b8
  gst-launch-1.0-16176 [006] ....   260.569749: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-16176 [001] ....   260.571192: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-16176 [001] ....   260.571197: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-16176 [001] ....   260.571198: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-16176 [001] ....   260.571200: csi_s_power: enable : 0x1
  gst-launch-1.0-16176 [001] ....   260.572623: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-16176 [001] ....   260.572630: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-16176 [001] ....   260.572631: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-16176 [001] ....   260.572632: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-16176 [001] ....   260.572633: csi_s_power: enable : 0x0
  gst-launch-1.0-16176 [001] ....   260.579251: tegra_channel_open: vi-output, ar0231 2-0009
  gst-launch-1.0-16176 [001] ....   260.579260: tegra_channel_set_power: ar0231 2-0009 : 0x1
  gst-launch-1.0-16176 [001] ....   260.579261: tegra_channel_set_power: rvnirp461 2-000a : 0x1
  gst-launch-1.0-16176 [001] ....   260.579262: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x1
  gst-launch-1.0-16176 [001] ....   260.579264: csi_s_power: enable : 0x1
    v4l2src0:src-16177 [007] ....   260.583918: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
     kworker/1:2-1686  [001] ....   260.603771: rtos_queue_peek_from_isr_failed: tstamp:8557892778 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   260.603776: rtcpu_start: tstamp:8557894243
     kworker/1:2-1686  [001] ....   260.715770: rtos_queue_send_from_isr_failed: tstamp:8560551248 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   260.715774: rtos_queue_send_from_isr_failed: tstamp:8560551385 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   260.715775: rtos_queue_send_from_isr_failed: tstamp:8560551524 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   260.715775: rtos_queue_send_from_isr_failed: tstamp:8560551658 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   260.715776: rtos_queue_send_from_isr_failed: tstamp:8560551792 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   260.715778: rtcpu_string: tstamp:8560552196 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/1:2-1686  [001] ....   260.715801: rtcpu_string: tstamp:8560552372 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/1:2-1686  [001] ....   260.715807: rtcpu_string: tstamp:8560552611 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/1:2-1686  [001] ....   260.715813: rtcpu_string: tstamp:8560552841 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/1:2-1686  [001] ....   260.715817: rtcpu_string: tstamp:8560553059 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/1:2-1686  [001] ....   260.715848: rtcpu_string: tstamp:8560553290 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/1:2-1686  [001] ....   260.715853: rtcpu_string: tstamp:8560553508 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/1:2-1686  [001] ....   260.715858: rtcpu_string: tstamp:8560560999 id:0x04010000 str:"vi5_hwinit: firmware CL2018101701 protocol vers"
     kworker/1:2-1686  [001] ....   260.715859: rtcpu_string: tstamp:8560561104 id:0x04010000 str:"on 2.2
"
     kworker/1:2-1686  [001] ....   260.771779: rtos_queue_peek_from_isr_failed: tstamp:8562893125 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   260.943760: rtos_queue_peek_from_isr_failed: tstamp:8567893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   261.115764: rtos_queue_peek_from_isr_failed: tstamp:8572893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   261.227757: rtos_queue_peek_from_isr_failed: tstamp:8577893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   261.395761: rtos_queue_peek_from_isr_failed: tstamp:8582893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   261.567765: rtos_queue_peek_from_isr_failed: tstamp:8587893123 queue:0x0bcbb8b8
    v4l2src0:src-16177 [002] ....   261.599565: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-16177 [001] ....   261.601056: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x1
    v4l2src0:src-16177 [001] ....   261.601060: csi_s_stream: enable : 0x1
    v4l2src0:src-16177 [001] ....   261.601217: tegra_channel_set_stream: rvnirp461 2-000a : 0x1
    v4l2src0:src-16177 [001] ....   261.601238: tegra_channel_set_stream: ar0231 2-0009 : 0x1
     kworker/1:2-1686  [001] ....   261.623755: rtos_queue_send_from_isr_failed: tstamp:8589609435 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   261.623757: rtos_queue_send_from_isr_failed: tstamp:8589609575 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   261.623758: rtos_queue_send_from_isr_failed: tstamp:8589609715 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   261.623758: rtos_queue_send_from_isr_failed: tstamp:8589609850 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   261.623759: rtos_queue_send_from_isr_failed: tstamp:8589609985 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   261.623759: rtos_queue_send_from_isr_failed: tstamp:8589610956 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   261.623760: rtos_queue_send_from_isr_failed: tstamp:8589611079 queue:0x0bcb7338
     kworker/1:2-1686  [001] ....   261.623761: rtos_queue_send_from_isr_failed: tstamp:8589611204 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   261.623761: rtos_queue_send_from_isr_failed: tstamp:8589611330 queue:0x0bcb8eb8
     kworker/1:2-1686  [001] ....   261.623781: rtos_queue_send_from_isr_failed: tstamp:8589611454 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   261.623782: rtos_queue_send_from_isr_failed: tstamp:8589611576 queue:0x0bcb9c78
     kworker/1:2-1686  [001] ....   261.623783: rtos_queue_send_from_isr_failed: tstamp:8589611697 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   261.623783: rtos_queue_send_from_isr_failed: tstamp:8589611818 queue:0x0bcbaa38
     kworker/1:2-1686  [001] ....   261.623784: rtos_queue_send_from_isr_failed: tstamp:8589611940 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   261.735776: rtos_queue_peek_from_isr_failed: tstamp:8592893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   261.903762: rtos_queue_peek_from_isr_failed: tstamp:8597893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   262.071764: rtos_queue_peek_from_isr_failed: tstamp:8602893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   262.239752: rtos_queue_peek_from_isr_failed: tstamp:8607893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   262.355764: rtos_queue_peek_from_isr_failed: tstamp:8612893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   262.523756: rtos_queue_peek_from_isr_failed: tstamp:8617893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   262.695759: rtos_queue_peek_from_isr_failed: tstamp:8622893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   262.867746: rtos_queue_peek_from_isr_failed: tstamp:8627893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   263.035757: rtos_queue_peek_from_isr_failed: tstamp:8632893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   263.147814: rtos_queue_peek_from_isr_failed: tstamp:8637893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   263.259757: rtos_queue_send_from_isr_failed: tstamp:8641201485 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   263.259760: rtos_queue_send_from_isr_failed: tstamp:8641201624 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   263.259761: rtos_queue_send_from_isr_failed: tstamp:8641201763 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   263.259762: rtos_queue_send_from_isr_failed: tstamp:8641201898 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   263.259763: rtos_queue_send_from_isr_failed: tstamp:8641202032 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   263.259764: rtcpu_string: tstamp:8641202840 id:0x04010000 str:"VI GOS[#0] set to VM GOS[4] base 0xc2104000
"
     kworker/1:2-1686  [001] ....   263.259783: rtcpu_vinotify_event: tstamp:8641216607 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:10230547424 data:0x10000000
     kworker/1:2-1686  [001] ....   263.259784: rtcpu_vinotify_event: tstamp:8641216738 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:10230551232 data:0x31000001
     kworker/1:2-1686  [001] ....   263.259785: rtcpu_vinotify_event: tstamp:8641216882 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:10230552224 data:0x07020001
     kworker/1:2-1686  [001] ....   263.315765: rtos_queue_peek_from_isr_failed: tstamp:8642893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   263.483769: rtos_queue_peek_from_isr_failed: tstamp:8647893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   263.539798: rtos_queue_send_from_isr_failed: tstamp:8649726983 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   263.539802: rtos_queue_send_from_isr_failed: tstamp:8649727131 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   263.539802: rtos_queue_send_from_isr_failed: tstamp:8649727270 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   263.539803: rtos_queue_send_from_isr_failed: tstamp:8649727405 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   263.539803: rtos_queue_send_from_isr_failed: tstamp:8649727539 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   263.539805: rtcpu_vinotify_event: tstamp:8650022936 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:10503350592 data:0x10000000
     kworker/1:2-1686  [001] ....   263.539806: rtcpu_vinotify_event: tstamp:8650023085 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:10503354432 data:0x31000002
     kworker/1:2-1686  [001] ....   263.651785: rtos_queue_peek_from_isr_failed: tstamp:8652893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   263.823758: rtos_queue_peek_from_isr_failed: tstamp:8657893118 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   263.991760: rtos_queue_peek_from_isr_failed: tstamp:8662893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   264.159790: rtos_queue_peek_from_isr_failed: tstamp:8667893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   264.275753: rtos_queue_peek_from_isr_failed: tstamp:8672893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   264.451762: rtos_queue_peek_from_isr_failed: tstamp:8677893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   264.619759: rtos_queue_peek_from_isr_failed: tstamp:8682893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   264.791757: rtos_queue_peek_from_isr_failed: tstamp:8687893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   264.959755: rtos_queue_peek_from_isr_failed: tstamp:8692893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   265.071763: rtos_queue_peek_from_isr_failed: tstamp:8697893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   265.239753: rtos_queue_peek_from_isr_failed: tstamp:8702893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   265.407753: rtos_queue_peek_from_isr_failed: tstamp:8707893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   265.583750: rtos_queue_peek_from_isr_failed: tstamp:8712893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   265.751765: rtos_queue_peek_from_isr_failed: tstamp:8717893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   265.979341: rtos_queue_peek_from_isr_failed: tstamp:8722893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   266.034961: rtos_queue_peek_from_isr_failed: tstamp:8727893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   266.148094: rtos_queue_send_from_isr_failed: tstamp:8730867268 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   266.148096: rtos_queue_send_from_isr_failed: tstamp:8730867408 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   266.148097: rtos_queue_send_from_isr_failed: tstamp:8730867547 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   266.148098: rtos_queue_send_from_isr_failed: tstamp:8730867685 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   266.148098: rtos_queue_send_from_isr_failed: tstamp:8730867818 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   266.208731: rtos_queue_peek_from_isr_failed: tstamp:8732893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   266.322999: rtos_queue_send_from_isr_failed: tstamp:8736439369 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   266.323002: rtos_queue_send_from_isr_failed: tstamp:8736439505 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   266.323002: rtos_queue_send_from_isr_failed: tstamp:8736439643 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   266.323003: rtos_queue_send_from_isr_failed: tstamp:8736439776 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   266.323004: rtos_queue_send_from_isr_failed: tstamp:8736439910 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   266.377244: rtos_queue_peek_from_isr_failed: tstamp:8737893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   266.547788: rtos_queue_peek_from_isr_failed: tstamp:8742893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   266.720860: rtos_queue_peek_from_isr_failed: tstamp:8747893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   266.835053: rtos_queue_peek_from_isr_failed: tstamp:8752893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   267.000693: rtos_queue_peek_from_isr_failed: tstamp:8757893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   267.167754: rtos_queue_peek_from_isr_failed: tstamp:8762893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   267.342258: rtos_queue_peek_from_isr_failed: tstamp:8767893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   267.342261: rtos_queue_send_from_isr_failed: tstamp:8768418216 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   267.342262: rtos_queue_send_from_isr_failed: tstamp:8768418354 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   267.342262: rtos_queue_send_from_isr_failed: tstamp:8768418492 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   267.342263: rtos_queue_send_from_isr_failed: tstamp:8768418626 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   267.342263: rtos_queue_send_from_isr_failed: tstamp:8768418758 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   267.515077: rtos_queue_peek_from_isr_failed: tstamp:8772893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   267.685962: rtos_queue_peek_from_isr_failed: tstamp:8777893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   267.800425: rtos_queue_peek_from_isr_failed: tstamp:8782893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   267.970603: rtos_queue_peek_from_isr_failed: tstamp:8787893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   268.135801: rtos_queue_peek_from_isr_failed: tstamp:8792893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   268.308706: rtos_queue_peek_from_isr_failed: tstamp:8797893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   268.423499: rtos_queue_send_from_isr_failed: tstamp:8802070131 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   268.423501: rtos_queue_send_from_isr_failed: tstamp:8802070270 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   268.423502: rtos_queue_send_from_isr_failed: tstamp:8802070409 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   268.423502: rtos_queue_send_from_isr_failed: tstamp:8802070542 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   268.423503: rtos_queue_send_from_isr_failed: tstamp:8802070676 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   268.478004: rtos_queue_peek_from_isr_failed: tstamp:8802893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   268.592456: rtos_queue_peek_from_isr_failed: tstamp:8807893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   268.763344: rtos_queue_peek_from_isr_failed: tstamp:8812893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   268.936157: rtos_queue_peek_from_isr_failed: tstamp:8817893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   269.109319: rtos_queue_peek_from_isr_failed: tstamp:8822893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   269.227279: rtos_queue_peek_from_isr_failed: tstamp:8827893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   269.392062: rtos_queue_peek_from_isr_failed: tstamp:8832893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   269.448646: rtos_queue_send_from_isr_failed: tstamp:8833862411 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   269.448648: rtos_queue_send_from_isr_failed: tstamp:8833862550 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   269.448649: rtos_queue_send_from_isr_failed: tstamp:8833862687 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   269.448649: rtos_queue_send_from_isr_failed: tstamp:8833862821 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   269.448650: rtos_queue_send_from_isr_failed: tstamp:8833862953 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   269.563354: rtos_queue_peek_from_isr_failed: tstamp:8837893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   269.729603: rtos_queue_peek_from_isr_failed: tstamp:8842893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   269.901460: rtos_queue_peek_from_isr_failed: tstamp:8847893127 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   270.077857: rtos_queue_peek_from_isr_failed: tstamp:8852893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   270.192831: rtos_queue_peek_from_isr_failed: tstamp:8857893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   270.363990: rtos_queue_peek_from_isr_failed: tstamp:8862893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   270.482458: rtos_queue_send_from_isr_failed: tstamp:8865639190 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   270.482460: rtos_queue_send_from_isr_failed: tstamp:8865639331 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   270.482461: rtos_queue_send_from_isr_failed: tstamp:8865639470 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   270.482461: rtos_queue_send_from_isr_failed: tstamp:8865639604 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   270.482462: rtos_queue_send_from_isr_failed: tstamp:8865639737 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   270.537056: rtos_queue_peek_from_isr_failed: tstamp:8867893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   270.711883: rtos_queue_peek_from_isr_failed: tstamp:8872893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   270.832547: rtos_queue_peek_from_isr_failed: tstamp:8877893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   271.013248: rtos_queue_peek_from_isr_failed: tstamp:8882893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   271.188235: rtos_queue_peek_from_isr_failed: tstamp:8887893124 queue:0x0bcbb8b8
 vi-output, ar02-16208 [001] ....   271.316258: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1208 fmt 13
     kworker/1:2-1686  [001] ....   271.361930: rtos_queue_peek_from_isr_failed: tstamp:8892893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   271.471768: rtos_queue_send_from_isr_failed: tstamp:8897710340 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   271.471771: rtos_queue_send_from_isr_failed: tstamp:8897710480 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   271.471771: rtos_queue_send_from_isr_failed: tstamp:8897710620 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   271.471772: rtos_queue_send_from_isr_failed: tstamp:8897710755 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   271.471773: rtos_queue_send_from_isr_failed: tstamp:8897710889 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   271.471774: rtcpu_string: tstamp:8897711292 id:0x04010000 str:"Configuring VI GoS.
"
     kworker/1:2-1686  [001] ....   271.471790: rtcpu_string: tstamp:8897711466 id:0x04010000 str:"VM GOS[#0] addr=0xc2100000
"
     kworker/1:2-1686  [001] ....   271.471796: rtcpu_string: tstamp:8897711707 id:0x04010000 str:"VM GOS[#1] addr=0xc2101000
"
     kworker/1:2-1686  [001] ....   271.471802: rtcpu_string: tstamp:8897711935 id:0x04010000 str:"VM GOS[#2] addr=0xc2102000
"
     kworker/1:2-1686  [001] ....   271.471810: rtcpu_string: tstamp:8897712152 id:0x04010000 str:"VM GOS[#3] addr=0xc2103000
"
     kworker/1:2-1686  [001] ....   271.471816: rtcpu_string: tstamp:8897712384 id:0x04010000 str:"VM GOS[#4] addr=0xc2104000
"
     kworker/1:2-1686  [001] ....   271.471820: rtcpu_string: tstamp:8897712600 id:0x04010000 str:"VM GOS[#5] addr=0xc2105000
"
     kworker/1:2-1686  [001] ....   271.471826: rtos_queue_peek_from_isr_failed: tstamp:8897893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   271.639776: rtos_queue_peek_from_isr_failed: tstamp:8902893119 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   271.807760: rtos_queue_peek_from_isr_failed: tstamp:8907893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   271.979766: rtos_queue_peek_from_isr_failed: tstamp:8912893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   272.151754: rtos_queue_peek_from_isr_failed: tstamp:8917893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   272.319763: rtos_queue_peek_from_isr_failed: tstamp:8922893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   272.435768: rtos_queue_peek_from_isr_failed: tstamp:8927893124 queue:0x0bcbb8b8
  gst-launch-1.0-16176 [003] ....   272.512422: tegra_channel_set_stream: enable : 0x0
  gst-launch-1.0-16176 [003] ....   272.512426: tegra_channel_set_stream: ar0231 2-0009 : 0x0
  gst-launch-1.0-16176 [001] ....   272.522931: tegra_channel_set_stream: rvnirp461 2-000a : 0x0
  gst-launch-1.0-16176 [001] ....   272.528692: tegra_channel_set_stream: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-16176 [001] ....   272.528696: csi_s_stream: enable : 0x0
     kworker/1:2-1686  [001] ....   272.606461: rtos_queue_peek_from_isr_failed: tstamp:8932893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   272.721422: rtos_queue_send_from_isr_failed: tstamp:8935403645 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   272.721425: rtos_queue_send_from_isr_failed: tstamp:8935403784 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   272.721426: rtos_queue_send_from_isr_failed: tstamp:8935403923 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   272.721426: rtos_queue_send_from_isr_failed: tstamp:8935404055 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   272.721427: rtos_queue_send_from_isr_failed: tstamp:8935404189 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   272.781539: rtos_queue_peek_from_isr_failed: tstamp:8937893125 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   272.956535: rtos_queue_peek_from_isr_failed: tstamp:8942893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   273.076859: rtos_queue_peek_from_isr_failed: tstamp:8947893132 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   273.245990: rtos_queue_peek_from_isr_failed: tstamp:8952893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   273.415810: rtos_queue_peek_from_isr_failed: tstamp:8957893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   273.585318: rtos_queue_peek_from_isr_failed: tstamp:8962893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   273.702990: rtos_queue_send_from_isr_failed: tstamp:8967196127 queue:0x0bcb2b38
     kworker/1:2-1686  [001] ....   273.702993: rtos_queue_send_from_isr_failed: tstamp:8967196267 queue:0x0bcb73a0
     kworker/1:2-1686  [001] ....   273.702993: rtos_queue_send_from_isr_failed: tstamp:8967196407 queue:0x0bcb8f20
     kworker/1:2-1686  [001] ....   273.702994: rtos_queue_send_from_isr_failed: tstamp:8967196542 queue:0x0bcb9ce0
     kworker/1:2-1686  [001] ....   273.702995: rtos_queue_send_from_isr_failed: tstamp:8967196676 queue:0x0bcbaaa0
     kworker/1:2-1686  [001] ....   273.756271: rtos_queue_peek_from_isr_failed: tstamp:8967893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   273.876230: rtos_queue_peek_from_isr_failed: tstamp:8972893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   274.047119: rtos_queue_peek_from_isr_failed: tstamp:8977893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   274.219933: rtos_queue_peek_from_isr_failed: tstamp:8982893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   274.393095: rtos_queue_peek_from_isr_failed: tstamp:8987893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   274.509479: rtos_queue_peek_from_isr_failed: tstamp:8992893122 queue:0x0bcbb8b8
  gst-launch-1.0-16176 [001] ....   274.611251: tegra_channel_close: vi-output, ar0231 2-0009
  gst-launch-1.0-16176 [001] ....   274.611257: tegra_channel_set_power: ar0231 2-0009 : 0x0
  gst-launch-1.0-16176 [001] ....   274.611259: tegra_channel_set_power: rvnirp461 2-000a : 0x0
  gst-launch-1.0-16176 [001] ....   274.611260: tegra_channel_set_power: 15a00000.nvcsi--5 : 0x0
  gst-launch-1.0-16176 [001] ....   274.611262: csi_s_power: enable : 0x0
     kworker/1:2-1686  [001] ....   274.675769: rtos_queue_peek_from_isr_failed: tstamp:8997893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   274.847762: rtos_queue_peek_from_isr_failed: tstamp:9002893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   275.015775: rtos_queue_peek_from_isr_failed: tstamp:9007893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   275.187761: rtos_queue_peek_from_isr_failed: tstamp:9012893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   275.359774: rtos_queue_peek_from_isr_failed: tstamp:9017893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   275.471758: rtos_queue_peek_from_isr_failed: tstamp:9022893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   275.639776: rtos_queue_peek_from_isr_failed: tstamp:9027893123 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   275.807771: rtos_queue_peek_from_isr_failed: tstamp:9032893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   275.975772: rtos_queue_peek_from_isr_failed: tstamp:9037893120 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   276.143809: rtos_queue_peek_from_isr_failed: tstamp:9042893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   276.315758: rtos_queue_peek_from_isr_failed: tstamp:9047893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   276.427766: rtos_queue_peek_from_isr_failed: tstamp:9052893116 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   276.599806: rtos_queue_peek_from_isr_failed: tstamp:9057893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   276.771779: rtos_queue_peek_from_isr_failed: tstamp:9062893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   276.939777: rtos_queue_peek_from_isr_failed: tstamp:9067893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   277.111807: rtos_queue_peek_from_isr_failed: tstamp:9072893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   277.279765: rtos_queue_peek_from_isr_failed: tstamp:9077893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   277.391756: rtos_queue_peek_from_isr_failed: tstamp:9082893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   277.563768: rtos_queue_peek_from_isr_failed: tstamp:9087893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   277.731761: rtos_queue_peek_from_isr_failed: tstamp:9092893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   277.903800: rtos_queue_peek_from_isr_failed: tstamp:9097893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   278.071774: rtos_queue_peek_from_isr_failed: tstamp:9102893118 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   278.187762: rtos_queue_peek_from_isr_failed: tstamp:9107893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   278.355751: rtos_queue_peek_from_isr_failed: tstamp:9112893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   278.527781: rtos_queue_peek_from_isr_failed: tstamp:9117893124 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   278.695753: rtos_queue_peek_from_isr_failed: tstamp:9122893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   278.867790: rtos_queue_peek_from_isr_failed: tstamp:9127893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   279.039791: rtos_queue_peek_from_isr_failed: tstamp:9132893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   279.151778: rtos_queue_peek_from_isr_failed: tstamp:9137893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   279.319773: rtos_queue_peek_from_isr_failed: tstamp:9142893122 queue:0x0bcbb8b8
     kworker/1:2-1686  [001] ....   279.491766: rtos_queue_peek_from_isr_failed: tstamp:9147893122 queue:0x0bcbb8b8
     kworker/1:1-1185  [001] ....   279.659825: rtos_queue_peek_from_isr_failed: tstamp:9152893122 queue:0x0bcbb8b8
     kworker/1:1-1185  [001] ....   279.659829: rtos_queue_peek_from_isr_failed: tstamp:9153942302 queue:0x0bcbb8b8

Here are all the trace log files for single capture and double capture with and without the fpga (rvnirp).

trace_only_ar0231_1_image.txt (68.4 KB)
trace_only_ar0231_2_images.txt (133.3 KB)
trace_with_rvnirp_1_image.txt (69.4 KB)
trace_with_rvnirp_2_images.txt (96.9 KB)

The CHANSEL_NOMATCH could be undefine data type or embedded data incorrect size define in device tree, please check the embedded_data_height in device tree.

Thanks for your response. I have attached the live device tree from the system. I am not quite sure what you mean. I found that embedded_metadata_height is set to [30 00]. What should this value be? The FPGA node starts at line 9854. And the sensor node starts at line 10007.
with_rvnirp_live_dtc.txt (322.3 KB)

I also found that

v4l2-compliance -d /dev/video0

in both cases (with and without the external fpga) die in some null pointer referencing. I don’t know if that’s related at all. If it’s not related, I can deal with this later. I don’t want to cause distractions.

Make sure the line of embedded data output from FPGA confingure the embedded_metadata_height, your current set to 0

Hey Shane, thank you for your response. The embedded_metadata_height is being set to 0. In the live_dtc.txt file I shared, the embedded_metadata_height is set to [30 00]. 30 hexadecimal is 48 decimal, ASCII for ‘0’, and 00 is ASCII for the null pointer.

Without the FPGA in the pipeline, the sensor’s embedded_metadata_height also gets set to [30 00] and that setup works. But somehow adding the FPGA in the pipeline doesn’t work.

Please let me know what you think. Thank you!

That tell the FPGA output un-recognized data cause the CHANSEL_NOMATCH

Okay, is there something I can do or check or change to get more information about what might be causing the unrecognized data?

I have all these callback functions defined in the FPGA driver as part of the v4l2 framework.

static const struct v4l2_subdev_pad_ops rvnirp_pad_ops = {
	.enum_mbus_code = rvnirp_enum_mbus_code,
	.enum_frame_size = rvnirp_enum_frame_size,
	.get_fmt = rvnirp_get_format,
	.set_fmt = rvnirp_set_format,
};

static const struct v4l2_subdev_video_ops rvnirp_video_ops = {
	.g_frame_interval = rvnirp_g_frame_interval,
	.s_frame_interval = rvnirp_s_frame_interval,
	.s_stream = rvnirp_s_stream,
};


static const struct v4l2_subdev_ops rvnirp_subdev_ops = {
	.video = &rvnirp_video_ops,
    .pad = &rvnirp_pad_ops,
};


static const struct media_entity_operations rvnirp_media_ops = {
	.link_validate = v4l2_subdev_link_validate,
};

I am finding that v4l2_subdev_link_validate isn’t being called. (It is also not being called for the sensor driver.) I also found that none of the callback functions in rvnirp_pad_ops are being called either in the FPGA driver. They are being called for the sensor driver.

I started following the common camera initialization for the sensor and the FPGA, and so far, they seem to be similar to each other.

I guess I am wondering where to look at this point. I don’t know where the problem might be.

Suppose you need consult with vendor instead of modify the tegracam framework.

Thanks

Thanks, Shane. We develop our own boards and so we are the vendor. Haha.

It might be helpful to note that the image-taking works fine if we don’t try to include the FPGA in the V4L2 pipeline. In the hardware, the FPGA stays connected all the time. If there is only the sensor in the V4L2 pipeline connected to the csi node in software, the images can be taken just fine even with the FPGA connected in the hardware. It is only when I try to add the FPGA between the sensor and the csi node in the V4L2 pipeline in software that it stops working. This tells me that it is not a board problem.

Let me know what you think. Thank you for your time and help!

Does the FPGA able output test pattern to try.

Hi Shane, sorry about the delay in replying.

There are v4l2 controls defined for the sensor. I changed the v4l2-ctrls to produce a solid yellow image.

This is the result of v4l2-ctl --list-ctrls

Camera Controls

           sensor_configuration 0x009a2032 (u32)    : min=0 max=4294967295 step=1 default=0 [22] flags=read-only, volatile, has-payload
         sensor_mode_i2c_packet 0x009a2033 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
      sensor_control_i2c_packet 0x009a2034 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
                    bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0
                override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0
                   height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0
               write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
       sensor_signal_properties 0x009a2069 (u32)    : min=0 max=4294967295 step=1 default=0 [30][18] flags=read-only, has-payload
        sensor_image_properties 0x009a206a (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
      sensor_control_properties 0x009a206b (u32)    : min=0 max=4294967295 step=1 default=0 [30][36] flags=read-only, has-payload
              sensor_dv_timings 0x009a206c (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
               low_latency_mode 0x009a206d (bool)   : default=0 value=0
               preferred_stride 0x009a206e (int)    : min=0 max=65535 step=1 default=0 value=0
                   sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=1 flags=read-only

Image Source Controls

                       test_red 0x009e0904 (int)    : min=0 max=65535 step=1 default=65535 value=65535 flags=slider
                  test_greenred 0x009e0905 (int)    : min=0 max=65535 step=1 default=65535 value=65535 flags=slider
                      test_blue 0x009e0906 (int)    : min=0 max=65535 step=1 default=65535 value=0 flags=slider
                 test_greenblue 0x009e0907 (int)    : min=0 max=65535 step=1 default=65535 value=65535 flags=slider

Image Processing Controls

                   test_pattern 0x009f0903 (menu)   : min=0 max=5 default=0 value=1

Scenario 1: FPGA not in the pipeline: I can produce multiple yellow images.
Scenario 2: FPGA in the pipeline: The first image is yellow, but then the subsequent images are all green (which is what I had been calling ‘no image’)

So, the behavior with or without the test pattern stuff enabled is the same.

Another thing I found that might be helpful:
I enabled the sensor to produce embedded metadata with each frame output. Then I changed the embedded_metadata_height to 2, since according to the sensor’s datasheet, 2 rows of data are prepended to the frame output containing the metadata.

I noticed the following:

  1. The ivc_queue counters were acting exactly like with the FPGA. I mean the tx_channel’s r_count and w_count were incrementing but rx_channel’s r_count and w_count were not.
  2. That led to the capture request timing out. Here is a screenshot that shows that:

Even though the request timed out, there were still successful images! I don’t know what to think about that.

Thank you for your time on this! I really appreciate your help!

Does the trace log still show the CHANSEL_NOMATCH for modify the embedded data?