Simulataneous capture failure with NX-V4l2

Hi,

On performing simultaneous capturing of images from three cameras sensors on NX module, infrequently we see the capture failing for one of the sensor and gradually all camera capture fails. v4l2 throws V4L2_BUF_FLAG_ERROR, when this happens. We are using v4l2+ userspace + memory allocated via cudaMallocManaged(…) for the capture.

We are unsure of the where the issue is. Please provide pointers to debug this issue

Further details are as follows

Tried following and still did not help

  1. Adding serdes_pix_clk_hz = “833333333” to dts file
  2. Push clock to maximum
    cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate ==> 314000000
    cat /sys/kernel/debug/bpmp/debug/clk/vi/rate ==> 460800000

System details:

  • R32.4.3
  • Kernel version 4.9.140-tegra

Below is the DTS that is being used

  		mode0 {
  			mclk_khz = "74250";
  			num_lanes = "4";
  			tegra_sinterface = "serial_c";
  			phy_mode = "DPHY";
  			discontinuous_clk = "yes";
  			dpcm_enable = "false";
  			cil_settletime = "0";

  			active_w = "1472"; //1440+32
  			active_h = "1080"; 
  			mode_type = "bayer";
  			pixel_phase = "bggr";
  			csi_pixel_bit_depth = "10";
  			readout_orientation = "0";
  			line_length = "1856";
  			inherent_gain = "1";
  			mclk_multiplier = "6.4"; //"12.53"; 
  			pix_clk_hz = "475201702"; //"465061632"; //1130x1856x226.58

  			gain_factor = "16";
  			framerate_factor = "1000000";
  			exposure_factor = "1000000";
  			min_gain_val = "0"; 
  			max_gain_val = "48"; 
  			step_gain_val = "1";
  			default_gain = "48"; 
  			min_hdr_ratio = "1";
  			max_hdr_ratio = "1";
  			min_framerate = "2000000"; /* 2.0 fps */
  			max_framerate = "226580000"; /* 226.5 fps */
  			step_framerate = "1";
  			default_framerate = "226580000"; /* 226.5 fps */
  			min_exp_time = "1"; /* us */
  			max_exp_time = "100000"; /* us */
  			step_exp_time = "1";
  			default_exp_time = "100"; /* us */
  			embedded_metadata_height = "0";
  		};

Following are the trace logs.

vi-output, imx2-6002 [001] … 464.220947: tegra_channel_capture_frame: eof:474.709343616
vi-output, imx2-6002 [001] … 464.239520: tegra_channel_capture_frame: sof:474.722082880
vi-output, imx2-6002 [001] … 464.239521: tegra_channel_capture_frame: eof:474.726303680
vi-output, imx2-6002 [001] … 464.239682: tegra_channel_capture_frame: sof:474.727042976
vi-output, imx2-6002 [001] … 464.239682: tegra_channel_capture_frame: eof:474.731263744

CPU 0 buffer started
 kworker/0:1-787   [000] ....   464.246924: rtcpu_nvcsi_intr: tstamp:14833916154 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246924: rtcpu_nvcsi_intr: tstamp:14833917443 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246924: rtcpu_nvcsi_intr: tstamp:14833917443 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246925: rtcpu_nvcsi_intr: tstamp:14833917443 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246925: rtcpu_nvcsi_intr: tstamp:14833917443 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246925: rtcpu_nvcsi_intr: tstamp:14833917443 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246925: rtcpu_nvcsi_intr: tstamp:14833917443 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246925: rtcpu_nvcsi_intr: tstamp:14833918779 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246925: rtcpu_nvcsi_intr: tstamp:14833918779 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246926: rtcpu_nvcsi_intr: tstamp:14833918779 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246926: rtcpu_nvcsi_intr: tstamp:14833918779 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246926: rtcpu_nvcsi_intr: tstamp:14833918779 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246926: rtcpu_nvcsi_intr: tstamp:14833918779 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246926: rtcpu_nvcsi_intr: tstamp:14833920074 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246926: rtcpu_nvcsi_intr: tstamp:14833920074 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246926: rtcpu_nvcsi_intr: tstamp:14833920074 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246927: rtcpu_nvcsi_intr: tstamp:14833920074 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246927: rtcpu_nvcsi_intr: tstamp:14833920074 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246927: rtcpu_nvcsi_intr: tstamp:14833920074 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246927: rtcpu_nvcsi_intr: tstamp:14833921367 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246927: rtcpu_nvcsi_intr: tstamp:14833921367 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246927: rtcpu_nvcsi_intr: tstamp:14833921367 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246927: rtcpu_nvcsi_intr: tstamp:14833921367 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246928: rtcpu_nvcsi_intr: tstamp:14833921367 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246928: rtcpu_nvcsi_intr: tstamp:14833921367 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246928: rtcpu_nvcsi_intr: tstamp:14833922666 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246928: rtcpu_nvcsi_intr: tstamp:14833922666 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246928: rtcpu_nvcsi_intr: tstamp:14833922666 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246928: rtcpu_nvcsi_intr: tstamp:14833922666 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246929: rtcpu_nvcsi_intr: tstamp:14833922666 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246929: rtcpu_nvcsi_intr: tstamp:14833922666 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246929: rtcpu_nvcsi_intr: tstamp:14833923959 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246929: rtcpu_nvcsi_intr: tstamp:14833923959 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246929: rtcpu_nvcsi_intr: tstamp:14833923959 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246929: rtcpu_nvcsi_intr: tstamp:14833923959 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246930: rtcpu_nvcsi_intr: tstamp:14833923959 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246930: rtcpu_nvcsi_intr: tstamp:14833923959 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246930: rtcpu_nvcsi_intr: tstamp:14833925249 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246930: rtcpu_nvcsi_intr: tstamp:14833925249 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246930: rtcpu_nvcsi_intr: tstamp:14833925249 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246930: rtcpu_nvcsi_intr: tstamp:14833925249 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246930: rtcpu_nvcsi_intr: tstamp:14833925249 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246931: rtcpu_nvcsi_intr: tstamp:14833925249 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246931: rtcpu_nvcsi_intr: tstamp:14833926547 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246931: rtcpu_nvcsi_intr: tstamp:14833926547 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246931: rtcpu_nvcsi_intr: tstamp:14833926547 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246931: rtcpu_nvcsi_intr: tstamp:14833926547 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246931: rtcpu_nvcsi_intr: tstamp:14833926547 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246931: rtcpu_nvcsi_intr: tstamp:14833926547 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246932: rtcpu_nvcsi_intr: tstamp:14833927840 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246932: rtcpu_nvcsi_intr: tstamp:14833927840 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246932: rtcpu_nvcsi_intr: tstamp:14833927840 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246932: rtcpu_nvcsi_intr: tstamp:14833927840 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246932: rtcpu_nvcsi_intr: tstamp:14833927840 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246932: rtcpu_nvcsi_intr: tstamp:14833927840 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246933: rtcpu_nvcsi_intr: tstamp:14833929134 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246933: rtcpu_nvcsi_intr: tstamp:14833929134 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246933: rtcpu_nvcsi_intr: tstamp:14833929134 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246933: rtcpu_nvcsi_intr: tstamp:14833929134 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246933: rtcpu_nvcsi_intr: tstamp:14833929134 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246933: rtcpu_nvcsi_intr: tstamp:14833929134 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246933: rtcpu_nvcsi_intr: tstamp:14833930428 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246934: rtcpu_nvcsi_intr: tstamp:14833930428 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246934: rtcpu_nvcsi_intr: tstamp:14833930428 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246934: rtcpu_nvcsi_intr: tstamp:14833930428 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246934: rtcpu_nvcsi_intr: tstamp:14833930428 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246934: rtcpu_nvcsi_intr: tstamp:14833930428 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246934: rtcpu_nvcsi_intr: tstamp:14833931722 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246935: rtcpu_nvcsi_intr: tstamp:14833931722 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246935: rtcpu_nvcsi_intr: tstamp:14833931722 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246935: rtcpu_nvcsi_intr: tstamp:14833931722 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246935: rtcpu_nvcsi_intr: tstamp:14833931722 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246935: rtcpu_nvcsi_intr: tstamp:14833931722 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246935: rtcpu_nvcsi_intr: tstamp:14833933025 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246935: rtcpu_nvcsi_intr: tstamp:14833933025 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246936: rtcpu_nvcsi_intr: tstamp:14833933025 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246936: rtcpu_nvcsi_intr: tstamp:14833933025 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246936: rtcpu_nvcsi_intr: tstamp:14833933025 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246936: rtcpu_nvcsi_intr: tstamp:14833933025 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246936: rtcpu_nvcsi_intr: tstamp:14833934318 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246936: rtcpu_nvcsi_intr: tstamp:14833934318 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246937: rtcpu_nvcsi_intr: tstamp:14833934318 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246937: rtcpu_nvcsi_intr: tstamp:14833934318 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246937: rtcpu_nvcsi_intr: tstamp:14833934318 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246937: rtcpu_nvcsi_intr: tstamp:14833934318 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246937: rtcpu_nvcsi_intr: tstamp:14833935616 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246937: rtcpu_nvcsi_intr: tstamp:14833935616 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246937: rtcpu_nvcsi_intr: tstamp:14833935616 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246938: rtcpu_nvcsi_intr: tstamp:14833935616 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246938: rtcpu_nvcsi_intr: tstamp:14833935616 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246938: rtcpu_nvcsi_intr: tstamp:14833935616 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246938: rtcpu_nvcsi_intr: tstamp:14833936918 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246938: rtcpu_nvcsi_intr: tstamp:14833936918 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246938: rtcpu_nvcsi_intr: tstamp:14833936918 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246938: rtcpu_nvcsi_intr: tstamp:14833936918 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246939: rtcpu_nvcsi_intr: tstamp:14833936918 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246939: rtcpu_nvcsi_intr: tstamp:14833936918 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246939: rtcpu_nvcsi_intr: tstamp:14833938221 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246939: rtcpu_nvcsi_intr: tstamp:14833938221 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246939: rtcpu_nvcsi_intr: tstamp:14833938221 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246939: rtcpu_nvcsi_intr: tstamp:14833938221 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246940: rtcpu_nvcsi_intr: tstamp:14833938221 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246940: rtcpu_nvcsi_intr: tstamp:14833938221 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246940: rtcpu_nvcsi_intr: tstamp:14833939527 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246940: rtcpu_nvcsi_intr: tstamp:14833939527 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246940: rtcpu_nvcsi_intr: tstamp:14833939527 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246940: rtcpu_nvcsi_intr: tstamp:14833939527 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246940: rtcpu_nvcsi_intr: tstamp:14833939527 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246941: rtcpu_nvcsi_intr: tstamp:14833939527 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246941: rtcpu_nvcsi_intr: tstamp:14833940825 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246941: rtcpu_nvcsi_intr: tstamp:14833940825 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246941: rtcpu_nvcsi_intr: tstamp:14833940825 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246941: rtcpu_nvcsi_intr: tstamp:14833940825 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246942: rtcpu_nvcsi_intr: tstamp:14833940825 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246942: rtcpu_nvcsi_intr: tstamp:14833940825 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246942: rtcpu_nvcsi_intr: tstamp:14833942119 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246943: rtcpu_nvcsi_intr: tstamp:14833942119 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246943: rtcpu_nvcsi_intr: tstamp:14833942119 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246943: rtcpu_nvcsi_intr: tstamp:14833942119 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246943: rtcpu_nvcsi_intr: tstamp:14833942119 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246943: rtcpu_nvcsi_intr: tstamp:14833942119 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246943: rtcpu_nvcsi_intr: tstamp:14833943413 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246943: rtcpu_nvcsi_intr: tstamp:14833943413 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246944: rtcpu_nvcsi_intr: tstamp:14833943413 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246944: rtcpu_nvcsi_intr: tstamp:14833943413 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246944: rtcpu_nvcsi_intr: tstamp:14833943413 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246944: rtcpu_nvcsi_intr: tstamp:14833943413 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246944: rtcpu_nvcsi_intr: tstamp:14833944703 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246944: rtcpu_nvcsi_intr: tstamp:14833944703 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246945: rtcpu_nvcsi_intr: tstamp:14833944703 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246945: rtcpu_nvcsi_intr: tstamp:14833944703 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246945: rtcpu_nvcsi_intr: tstamp:14833944703 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246945: rtcpu_nvcsi_intr: tstamp:14833944703 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246945: rtcpu_nvcsi_intr: tstamp:14833945992 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246945: rtcpu_nvcsi_intr: tstamp:14833945992 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246945: rtcpu_nvcsi_intr: tstamp:14833945992 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246946: rtcpu_nvcsi_intr: tstamp:14833945992 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246946: rtcpu_nvcsi_intr: tstamp:14833945992 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246946: rtcpu_nvcsi_intr: tstamp:14833945992 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246946: rtcpu_nvcsi_intr: tstamp:14833947286 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246946: rtcpu_nvcsi_intr: tstamp:14833947286 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246946: rtcpu_nvcsi_intr: tstamp:14833947286 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246947: rtcpu_nvcsi_intr: tstamp:14833947286 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246947: rtcpu_nvcsi_intr: tstamp:14833947286 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246947: rtcpu_nvcsi_intr: tstamp:14833947286 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246947: rtcpu_nvcsi_intr: tstamp:14833948583 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000006
 kworker/0:1-787   [000] ....   464.246947: rtcpu_nvcsi_intr: tstamp:14833948583 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246947: rtcpu_nvcsi_intr: tstamp:14833948583 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246947: rtcpu_nvcsi_intr: tstamp:14833948583 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000006
 kworker/0:1-787   [000] ....   464.246948: rtcpu_vinotify_error: tstamp:14833951228 tag:CSIMUX_FRAME channel:0x00 frame:1793 vi_tstamp:14833948276 data:0x00400260
 kworker/0:1-787   [000] ....   464.246949: rtcpu_vinotify_event: tstamp:14833951700 tag:CSIMUX_FRAME channel:0x00 frame:1794 vi_tstamp:14833793276 data:0x00400260
 kworker/0:1-787   [000] ....   464.246949: rtcpu_vinotify_event: tstamp:14833951842 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:1794 vi_tstamp:14833793276 data:0x01000000
 kworker/0:1-787   [000] ....   464.246949: rtcpu_vinotify_event: tstamp:14833952009 tag:RESERVED_19 channel:0x23 frame:2 vi_tstamp:15119901952 data:0x01021fce
 kworker/0:1-787   [000] ....   464.246949: rtcpu_vinotify_event: tstamp:14833952148 tag:FS channel:0x00 frame:1 vi_tstamp:14833793277 data:0x00000014
 kworker/0:1-787   [000] ....   464.246950: rtcpu_vinotify_event: tstamp:14833952312 tag:ATOMP_FE channel:0x00 frame:1794 vi_tstamp:14833793277 data:0x00000000
 kworker/0:1-787   [000] ....   464.246950: rtcpu_vinotify_event: tstamp:14833952448 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:15119911680 data:0x07021fcf
 kworker/0:1-787   [000] ....   464.246950: rtcpu_vinotify_event: tstamp:14833952612 tag:ATOMP_FS channel:0x00 frame:1 vi_tstamp:14833793278 data:0x00000000
 kworker/0:1-787   [000] ....   464.246950: rtcpu_vinotify_event: tstamp:14833952748 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:15120138912 data:0x10000000
 kworker/0:1-787   [000] ....   464.246950: rtcpu_vinotify_event: tstamp:14833952906 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:15120147552 data:0x31001fd0
 kworker/0:1-787   [000] ....   464.246950: rtcpu_vinotify_event: tstamp:14833953041 tag:FS channel:0x00 frame:1793 vi_tstamp:14833816349 data:0x00000010
 kworker/0:1-787   [000] ....   464.246950: rtcpu_vinotify_event: tstamp:14833953195 tag:ATOMP_FS channel:0x00 frame:1793 vi_tstamp:14833816349 data:0x00000000
 kworker/0:1-787   [000] ....   464.246951: rtcpu_vinotify_event: tstamp:14833953330 tag:CHANSEL_PXL_SOF channel:0x22 frame:1 vi_tstamp:14833816351 data:0x00000001
 kworker/0:1-787   [000] ....   464.246951: rtcpu_vinotify_event: tstamp:14833953487 tag:RESERVED_19 channel:0x22 frame:1 vi_tstamp:15120626784 data:0x08021947
 kworker/0:1-787   [000] ....   464.246951: rtcpu_vinotify_event: tstamp:14833953621 tag:CHANSEL_PXL_SOF channel:0x23 frame:1793 vi_tstamp:14833816473 data:0x00000001
 kworker/0:1-787   [000] ....   464.246951: rtcpu_vinotify_event: tstamp:14833953779 tag:RESERVED_19 channel:0x23 frame:1 vi_tstamp:15120632608 data:0x08021fcf
 kworker/0:1-787   [000] ....   464.246951: rtcpu_vinotify_event: tstamp:14833953912 tag:FE channel:0x00 frame:1 vi_tstamp:14833826981 data:0x00000022
 kworker/0:1-787   [000] ....   464.246951: rtcpu_vinotify_event: tstamp:14833954068 tag:FS channel:0x00 frame:2 vi_tstamp:14833827008 data:0x00000012
 kworker/0:1-787   [000] ....   464.246952: rtcpu_vinotify_event: tstamp:14833954204 tag:ATOMP_FS channel:0x00 frame:2 vi_tstamp:14833827008 data:0x00000000
 kworker/0:1-787   [000] ....   464.246952: rtcpu_vinotify_event: tstamp:14833954361 tag:CHANSEL_PXL_SOF channel:0x21 frame:2 vi_tstamp:14833850085 data:0x00000001
 kworker/0:1-787   [000] ....   464.246952: rtcpu_vinotify_event: tstamp:14833954496 tag:RESERVED_19 channel:0x21 frame:2 vi_tstamp:15121703104 data:0x08021940
 kworker/0:1-787   [000] ....   464.246952: rtcpu_vinotify_event: tstamp:14833954651 tag:CHANSEL_PXL_EOF channel:0x22 frame:1 vi_tstamp:14833948142 data:0x04370002
 kworker/0:1-787   [000] ....   464.246952: rtcpu_vinotify_event: tstamp:14833954786 tag:ATOMP_FRAME_DONE channel:0x22 frame:1 vi_tstamp:14833948171 data:0x00000000
 kworker/0:1-787   [000] ....   464.246952: rtcpu_vinotify_event: tstamp:14833954943 tag:RESERVED_19 channel:0x22 frame:1 vi_tstamp:15124841888 data:0x02021947
 kworker/0:1-787   [000] ....   464.246953: rtcpu_vinotify_event: tstamp:14833955077 tag:FE channel:0x00 frame:1 vi_tstamp:14833948250 data:0x00000024
 kworker/0:1-787   [000] ....   464.246953: rtcpu_vinotify_event: tstamp:14833955232 tag:ATOMP_FE channel:0x00 frame:1 vi_tstamp:14833948250 data:0x00000000
 kworker/0:1-787   [000] ....   464.246953: rtcpu_vinotify_event: tstamp:14833955369 tag:RESERVED_19 channel:0x22 frame:1 vi_tstamp:15124848160 data:0x00021947
 kworker/0:1-787   [000] ....   464.246953: rtcpu_vinotify_event: tstamp:14833955525 tag:CSIMUX_FRAME channel:0x00 frame:1793 vi_tstamp:14833948276 data:0x00400260
 kworker/0:1-787   [000] ....   464.246953: rtcpu_vinotify_event: tstamp:14833955664 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:1793 vi_tstamp:14833948276 data:0x01000000
 kworker/0:1-787   [000] ....   464.246953: rtcpu_vinotify_event: tstamp:14833955822 tag:RESERVED_19 channel:0x23 frame:1 vi_tstamp:15124859296 data:0x01021fcf
 kworker/0:1-787   [000] ....   464.246954: rtcpu_vinotify_event: tstamp:14833955958 tag:ATOMP_FE channel:0x00 frame:1793 vi_tstamp:14833948276 data:0x00000000
 kworker/0:1-787   [000] ....   464.246954: rtcpu_vinotify_event: tstamp:14833956114 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:15124868416 data:0x07021fd0
 kworker/0:1-787   [000] ....   464.246954: rtcpu_vinotify_event: tstamp:14833956248 tag:FS channel:0x00 frame:2 vi_tstamp:14833948276 data:0x00000014
 kworker/0:1-787   [000] ....   464.246957: rtcpu_nvcsi_intr: tstamp:14833971429 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000002
 kworker/0:1-787   [000] ....   464.246957: rtcpu_nvcsi_intr: tstamp:14833971429 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246957: rtcpu_nvcsi_intr: tstamp:14833971429 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000006
 kworker/0:1-787   [000] ....   464.246957: rtcpu_nvcsi_intr: tstamp:14833971429 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246957: rtcpu_nvcsi_intr: tstamp:14833971429 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246957: rtcpu_nvcsi_intr: tstamp:14833972601 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246958: rtcpu_nvcsi_intr: tstamp:14833972601 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246958: rtcpu_nvcsi_intr: tstamp:14833972601 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246958: rtcpu_nvcsi_intr: tstamp:14833972601 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246958: rtcpu_nvcsi_intr: tstamp:14833972601 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246958: rtcpu_nvcsi_intr: tstamp:14833972601 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246958: rtcpu_nvcsi_intr: tstamp:14833973902 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246959: rtcpu_nvcsi_intr: tstamp:14833973902 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246959: rtcpu_nvcsi_intr: tstamp:14833973902 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246959: rtcpu_nvcsi_intr: tstamp:14833973902 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246959: rtcpu_nvcsi_intr: tstamp:14833973902 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246959: rtcpu_nvcsi_intr: tstamp:14833973902 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246959: rtcpu_nvcsi_intr: tstamp:14833975195 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246959: rtcpu_nvcsi_intr: tstamp:14833975195 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246960: rtcpu_nvcsi_intr: tstamp:14833975195 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246960: rtcpu_nvcsi_intr: tstamp:14833975195 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246960: rtcpu_nvcsi_intr: tstamp:14833975195 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246960: rtcpu_nvcsi_intr: tstamp:14833975195 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246960: rtcpu_nvcsi_intr: tstamp:14833976490 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246960: rtcpu_nvcsi_intr: tstamp:14833976490 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246961: rtcpu_nvcsi_intr: tstamp:14833976490 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246961: rtcpu_nvcsi_intr: tstamp:14833976490 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246961: rtcpu_nvcsi_intr: tstamp:14833976490 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246961: rtcpu_nvcsi_intr: tstamp:14833976490 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246961: rtcpu_nvcsi_intr: tstamp:14833977784 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246961: rtcpu_nvcsi_intr: tstamp:14833977784 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246962: rtcpu_nvcsi_intr: tstamp:14833977784 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246962: rtcpu_nvcsi_intr: tstamp:14833977784 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246962: rtcpu_nvcsi_intr: tstamp:14833977784 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246962: rtcpu_nvcsi_intr: tstamp:14833977784 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246962: rtcpu_nvcsi_intr: tstamp:14833979082 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246962: rtcpu_nvcsi_intr: tstamp:14833979082 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246963: rtcpu_nvcsi_intr: tstamp:14833979082 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246963: rtcpu_nvcsi_intr: tstamp:14833979082 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246963: rtcpu_nvcsi_intr: tstamp:14833979082 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246964: rtcpu_nvcsi_intr: tstamp:14833979082 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246964: rtcpu_nvcsi_intr: tstamp:14833980376 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246964: rtcpu_nvcsi_intr: tstamp:14833980376 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246964: rtcpu_nvcsi_intr: tstamp:14833980376 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246964: rtcpu_nvcsi_intr: tstamp:14833980376 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246964: rtcpu_nvcsi_intr: tstamp:14833980376 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246965: rtcpu_nvcsi_intr: tstamp:14833980376 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246965: rtcpu_nvcsi_intr: tstamp:14833981707 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246965: rtcpu_nvcsi_intr: tstamp:14833981707 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246965: rtcpu_nvcsi_intr: tstamp:14833981707 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246965: rtcpu_nvcsi_intr: tstamp:14833981707 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246965: rtcpu_nvcsi_intr: tstamp:14833981707 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246966: rtcpu_nvcsi_intr: tstamp:14833981707 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246966: rtcpu_nvcsi_intr: tstamp:14833982998 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246966: rtcpu_nvcsi_intr: tstamp:14833982998 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246966: rtcpu_nvcsi_intr: tstamp:14833982998 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246966: rtcpu_nvcsi_intr: tstamp:14833982998 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246966: rtcpu_nvcsi_intr: tstamp:14833982998 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246966: rtcpu_nvcsi_intr: tstamp:14833982998 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246967: rtcpu_nvcsi_intr: tstamp:14833984288 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246967: rtcpu_nvcsi_intr: tstamp:14833984288 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246967: rtcpu_nvcsi_intr: tstamp:14833984288 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246967: rtcpu_nvcsi_intr: tstamp:14833984288 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246967: rtcpu_nvcsi_intr: tstamp:14833984288 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246967: rtcpu_nvcsi_intr: tstamp:14833984288 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246968: rtcpu_nvcsi_intr: tstamp:14833985577 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246968: rtcpu_nvcsi_intr: tstamp:14833985577 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246968: rtcpu_nvcsi_intr: tstamp:14833985577 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246968: rtcpu_nvcsi_intr: tstamp:14833985577 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246968: rtcpu_nvcsi_intr: tstamp:14833985577 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246968: rtcpu_nvcsi_intr: tstamp:14833985577 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246968: rtcpu_nvcsi_intr: tstamp:14833986875 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246969: rtcpu_nvcsi_intr: tstamp:14833986875 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246969: rtcpu_nvcsi_intr: tstamp:14833986875 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246969: rtcpu_nvcsi_intr: tstamp:14833986875 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246969: rtcpu_nvcsi_intr: tstamp:14833986875 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246969: rtcpu_nvcsi_intr: tstamp:14833986875 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246969: rtcpu_nvcsi_intr: tstamp:14833988171 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246970: rtcpu_nvcsi_intr: tstamp:14833988171 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246970: rtcpu_nvcsi_intr: tstamp:14833988171 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246970: rtcpu_nvcsi_intr: tstamp:14833988171 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246970: rtcpu_nvcsi_intr: tstamp:14833988171 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246970: rtcpu_nvcsi_intr: tstamp:14833988171 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246970: rtcpu_nvcsi_intr: tstamp:14833989467 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246970: rtcpu_nvcsi_intr: tstamp:14833989467 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246971: rtcpu_nvcsi_intr: tstamp:14833989467 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246971: rtcpu_nvcsi_intr: tstamp:14833989467 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246971: rtcpu_nvcsi_intr: tstamp:14833989467 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246971: rtcpu_nvcsi_intr: tstamp:14833989467 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246971: rtcpu_nvcsi_intr: tstamp:14833990760 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246971: rtcpu_nvcsi_intr: tstamp:14833990760 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246971: rtcpu_nvcsi_intr: tstamp:14833990760 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246972: rtcpu_nvcsi_intr: tstamp:14833990760 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246972: rtcpu_nvcsi_intr: tstamp:14833990760 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246972: rtcpu_nvcsi_intr: tstamp:14833990760 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246972: rtcpu_nvcsi_intr: tstamp:14833992049 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246972: rtcpu_nvcsi_intr: tstamp:14833992049 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246972: rtcpu_nvcsi_intr: tstamp:14833992049 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246973: rtcpu_nvcsi_intr: tstamp:14833992049 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246973: rtcpu_nvcsi_intr: tstamp:14833992049 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246973: rtcpu_nvcsi_intr: tstamp:14833992049 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246973: rtcpu_nvcsi_intr: tstamp:14833993346 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246973: rtcpu_nvcsi_intr: tstamp:14833993346 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246973: rtcpu_nvcsi_intr: tstamp:14833993346 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246973: rtcpu_nvcsi_intr: tstamp:14833993346 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246974: rtcpu_nvcsi_intr: tstamp:14833993346 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246974: rtcpu_nvcsi_intr: tstamp:14833993346 class:CORRECTABLE_ERR type:PHY_INTR phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246974: rtcpu_nvcsi_intr: tstamp:14833994638 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
 kworker/0:1-787   [000] ....   464.246974: rtcpu_nvcsi_intr: tstamp:14833994638 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....   464.246974: rtcpu_nvcsi_intr: tstamp:14833994638 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000110
 kworker/0:1-787   [000] ....

Additional Info on the v4l2 method we employ for capture:

  1. Memory pool for a total of 100 plus frames pre-allocated using single cudaMallocManaged call.
  2. Config V4l2 for userspace IO method & queue size set to 4 using ioctl VIDIOC_REQBUFS call
  3. Queue 4 of the buffer pointers from the memory pool using ioctl VIDIOC_QBUF & issue VIDIOC_STREAMON
  4. After the first succesfull dequeue, the 5th buffer pointer from 100 frame memory pool is enqueued instead of the 4 buffer,
  5. Continue feeding buffers from the pool cyclically ie after every successful dequeue of nth buffer, (n+1)%100 bufffer pointer gets enqueued.

We followed above method , so the implementation is pure zero copy, without any memcpy. The cpu usage is also considerably reduced with the above method.

Am I missing anything?

Thanks
Hari

Could you verify by v4l2-ctl to verify it.

Thanks

Thanks for the reply.
v4l2-ctl, will only use the 4 buffers which were initially queued and will also perform memcpy. We want to avoid this step and still be able to buffer frames of longer duration.

Between, if we restrict ourselves to only using those 4 buffers initially used and use memcpy to copy to the main bigger memory pool, we dont see any issues. I think this is what you were trying to verify and expect v4l2-ctl will also work. Let me know if I understood correctly

So to stress on the big difference in the way we perform capture:

  • enqueue 5th buffer pointer from 100 frames memory pool instead of re queuing the same buffer.
    (but queue size set to 4 using ioctl VIDIOC_REQBUFS call & the ioctl call does not allow setting bigger queue size such as 100+)
  • Using pinned memory created using cudaMallocManaged api

Just want to confirm if v4l2-ctl can capture continuously for multiple cameras.

v4l2-ctl --stream-mmap --set-ctrl bypass_mode=0 -d /dev/video0 & v4l2-ctl --stream-mmap --set-ctrl bypass_mode=0 -d /dev/video1

I am unable to verify using v4l2-ctl, as we have additional gpio controls.

But I verified using our custom application with IO methods such as mmap or user-pointer which uses memcpy and it works fine.

Capture get in to issue only with the method which i explained before.

We are striving for the method to work, as it is resulting in reduced CPU usage for the capture.

Could you reference to the MMAPI sample code …/jetson_multimedia_api/samples/12_camera_v4l2_cuda to implement your use case to try.

Thanks, will check and return

Hi Shane,

Was able to crosscheck, still capture fails.
Between I would like to stress again, the following difference in the flow.

  1. Memory pool for a total of 100 frames pre-allocated using single cudaMallocManaged call.
  2. Config V4l2 for userspace IO method & queue size set to 4 using ioctl VIDIOC_REQBUFS call
  3. Queue 4 buffer pointers from the memory pool using ioctl VIDIOC_QBUF & issue VIDIOC_STREAMON
  4. After the first successful dequeue, the 5th buffer pointer from 100 frame memory pool is enqueued instead of the passing the same buffer again.
  5. Continue feeding buffers from the pool cyclically ie after every successful dequeue of nth buffer, (n+1)%100 bufffer pointer gets enqueued.

Other difference is we dont have any process_image, after dequeuing.

Can you confirm if the above method of queuing back different buffer other than what it was initialized with will work?

Could you boost the nvcsi/vi/isp clocks and run the below command to verify.

sudo nvpmodel -m 2
sudo jetson_clocks

https://elinux.org/Jetson_TX2_Camera_BringUp

nvpmodel -q
NV Fan Mode:quiet
NV Power Mode: MODE_15W_6CORE
2

jetson_clocks --show
SOC family:tegra194 Machine:NVIDIA Jetson Xavier NX Developer Kit
Online CPUs: 0-5
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1420800 CurrentFreq=1420800 IdleStates: C1=1 c6=1
cpu1: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1420800 CurrentFreq=1420800 IdleStates: C1=1 c6=1
cpu2: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1420800 CurrentFreq=1420800 IdleStates: C1=1 c6=1
cpu3: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1420800 CurrentFreq=1420800 IdleStates: C1=1 c6=1
cpu4: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1420800 CurrentFreq=1420800 IdleStates: C1=1 c6=1
cpu5: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1420800 CurrentFreq=1420800 IdleStates: C1=1 c6=1
GPU MinFreq=114750000 MaxFreq=1109250000 CurrentFreq=114750000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=0
Fan: speed=0
NV Power Mode: MODE_15W_6CORE

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked

vi max 460800000 current 460800000
nvcsi max 314000000 current 314000000
isp max 576000000 current 576000000

Another observation, capture gets in to issue after 8 sec of capture

You may need to go through your APP to narrow down to find clue.
Also what the trace log after boost the clocks?

Issued trace enabling commands & then issued commands to maximize the nvcsi, vi, isp clocks
captured trace log after boosting the nvcsi, vi ,isp clocks.

rtos_queue_peek_from_isr_failed ??

tracer: nop

entries-in-buffer/entries-written: 35/35 #P:6

_-----=> irqs-off

/ _----=> need-resched

| / _—=> hardirq/softirq

|| / _–=> preempt-depth

||| / delay

TASK-PID CPU# |||| TIMESTAMP FUNCTION

| | | |||| | |

 kworker/0:0-4     [000] ....   366.195249: rtos_queue_peek_from_isr_failed: tstamp:11769323976 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   366.363204: rtos_queue_peek_from_isr_failed: tstamp:11774323979 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   366.531231: rtos_queue_peek_from_isr_failed: tstamp:11779323998 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   366.699214: rtos_queue_peek_from_isr_failed: tstamp:11784323979 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   366.867195: rtos_queue_peek_from_isr_failed: tstamp:11789323984 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   367.035186: rtos_queue_peek_from_isr_failed: tstamp:11794323984 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   367.147168: rtos_queue_peek_from_isr_failed: tstamp:11799323985 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   367.315162: rtos_queue_peek_from_isr_failed: tstamp:11804323979 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   367.483148: rtos_queue_peek_from_isr_failed: tstamp:11809323976 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   367.651151: rtos_queue_peek_from_isr_failed: tstamp:11814323980 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   367.819177: rtos_queue_peek_from_isr_failed: tstamp:11819323984 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   367.987167: rtos_queue_peek_from_isr_failed: tstamp:11824323976 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   368.155163: rtos_queue_peek_from_isr_failed: tstamp:11829323979 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   368.267142: rtos_queue_peek_from_isr_failed: tstamp:11834323984 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   368.435434: rtos_queue_peek_from_isr_failed: tstamp:11839323976 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   368.603152: rtos_queue_peek_from_isr_failed: tstamp:11844323979 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   368.771273: rtos_queue_peek_from_isr_failed: tstamp:11849323976 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   368.939159: rtos_queue_peek_from_isr_failed: tstamp:11854323988 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   369.107190: rtos_queue_peek_from_isr_failed: tstamp:11859323998 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   369.275146: rtos_queue_peek_from_isr_failed: tstamp:11864323980 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   369.387174: rtos_queue_peek_from_isr_failed: tstamp:11869323987 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   369.555150: rtos_queue_peek_from_isr_failed: tstamp:11874323976 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   369.723144: rtos_queue_peek_from_isr_failed: tstamp:11879323983 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   369.891136: rtos_queue_peek_from_isr_failed: tstamp:11884323974 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   370.059115: rtos_queue_peek_from_isr_failed: tstamp:11889323983 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   370.227109: rtos_queue_peek_from_isr_failed: tstamp:11894323980 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   370.395264: rtos_queue_peek_from_isr_failed: tstamp:11899323979 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   370.507126: rtos_queue_peek_from_isr_failed: tstamp:11904323982 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   370.675124: rtos_queue_peek_from_isr_failed: tstamp:11909323983 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   370.843116: rtos_queue_peek_from_isr_failed: tstamp:11914323982 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   371.011109: rtos_queue_peek_from_isr_failed: tstamp:11919323979 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   371.179194: rtos_queue_peek_from_isr_failed: tstamp:11924323796 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   371.347116: rtos_queue_peek_from_isr_failed: tstamp:11929323975 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   371.515129: rtos_queue_peek_from_isr_failed: tstamp:11934323975 queue:0x0bcbcf78
 kworker/0:0-4     [000] ....   371.627102: rtos_queue_peek_from_isr_failed: tstamp:11939155442 queue:0x0bcbcf78

Errors are reported after running the trace enabling commands (shown below) as referred in the wiki & not when enabling the clock

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace

This log looks didn’t receive any validate data from MIPI bus.
That’s weird it’s worse than early trace log?

Hi @ShaneCCC
Misunderstood your test. Attached the trace logs with camera enabled.
three-cam-capture-variable-framerate.trace (15.4 MB)

Thanks

Looks like the trace is different with earlier. This trace CHANSEL_NOMATCH after a while but looks like it still working. Beside this there’s no any others error from the trace.

Yes I noticed, but I did check clock were set to maximum, when I collected the logs.
But I may not have set the mrq_rate_locked flag. Not sure if it would had made any difference.
Mentioned capture failed because, at userspace v4l2 returns V4L2_BUF_FLAG_ERROR. There are 3 camera captured happening simultaneously. For the first 8 sec or so the capture continues without issue, but after that camera start returning the buf flag error and eventually all cameras fail at the end. We dont want any of the frames to be missed.

Anyother details from the log which could help us debug this issue will be appreciated. Also are there documents on understanding trace logs, as I see errors like RESERVED_18, RESERVED_19 etc

Between, I am using poll instead of select. Not sure if this is of relevance to the issue

Thanks

You can get some information from below link. The RESERVED_xx should ignore it.

https://elinux.org/Jetson_TX2_Camera_BringUp