Data Error of CSI Camera Capture on Jetson TX2

Hi,
We are attempting to use the 720p 60fps SOC Camera in conjunction with the Lattice Crosslink FPGA doing SubLVDS to CSI conversion.

However, we are not able to solve a serious problem.
The problem is that the tx2 system does not receive the data perfectly.
After the number of mmap buffers (req.count = 8 in my case), the current frame data and the previous frame data were mixed.

1th YUV Data File (1280x720) - attached yuv.zip
https://drive.google.com/open?id=1IYy3AdA41LYZ2lb1UzKSDO9soJFPdKU3
9th YUV Data File (1280x720)
https://drive.google.com/open?id=1CitndQVFyyEl-Vr31pz8zV74OjTv2W5i

The following shows “Frame Buffer Data” (only the first 4 bytes and the last 4 bytes.)

Byte Data 1 and 3 is Pixel Count, and Byte Data 2,4 is Frame Count. In one frame data, the frame count is constant.

1/2/3/4____1_/2_/3_/4
H:01 06 00 06 T:00 00 00 00 - Frame is 6 and No data, Error
H:01 07 00 07 T:00 00 00 00 - Frame is 7 and No data, Error
H:01 08 00 08 T:00 00 00 00 - Frame is 8 and No data, Error
H:01 09 00 09 T:00 00 00 00 - Frame is 9 and No data, Error
H:01 0A 00 0A T:00 00 00 00 - Frame is A and No data, Error
H:01 0B 00 0B T:00 00 00 00 - Frame is B and No data, Error
H:01 0C 00 0C T:00 00 00 00 - Frame is C and No data, Error
H:01 0D 00 0D T:00 00 00 00 - Frame is D and No data, Error
H:01 0E 00 0E T:FF 06 FE 06 - Frame is E and 6, Error , After eight buffers have passed, the remaining data appears.
H:01 0F 00 0F T:FF 07 FE 07 - Frame is F and 7 , Error
H:01 10 00 10 T:FF 08 FE 08 - Frame is 10 and 8 , Error
H:01 11 00 11 T:FF 09 FE 09 - Frame is 11 and 9 , Error
H:01 12 00 12 T:FF 0A FE 0A - Frame is 12 and A , Error
H:01 13 00 13 T:FF 0B FE 0B - Frame is 13 and B , Error
H:01 14 00 14 T:FF 0C FE 0C
H:01 15 00 15 T:FF 0D FE 0D
H:01 16 00 16 T:FF 0E FE 0E
H:01 17 00 17 T:FF 0F FE 0F
H:01 18 00 18 T:FF 10 FE 10
H:01 19 00 19 T:FF 11 FE 11

In many test situations, I noticed one peculiar point.
If the system is busy, the data is received normally. Of course, data loss occurs.
The following shows “Frame Buffer Data” when the system is busy,

H:01 07 00 07 T:00 00 00 00 - Frame is 7 and No data
H:01 08 00 08 T:00 00 00 00 - Frame is 8 and No data
H:01 09 00 09 T:00 00 00 00 - Frame is 9 and No data
H:01 0A 00 0A T:00 00 00 00 - Frame is A and No data
H:01 0B 00 0B T:00 00 00 00 - Frame is B and No data
H:01 0C 00 0C T:00 00 00 00 - Frame is C and No data
H:01 0D 00 0D T:00 00 00 00 - Frame is D and No data
H:01 0E 00 0E T:00 00 00 00 - Frame is E and No data
H:01 0F 00 0F T:FF 07 FE 07 - Frame is F and 7, Error, After eight buffers have passed, the remaining data appears.
H:01 10 00 10 T:FF 10 FE 10 - Frame is 10 and 10, Good. When the system is busy, data is normally received.
H:01 11 00 11 T:FF 11 FE 11 - Frame is 11 and 11, Good.
H:01 12 00 12 T:FF 12 FE 12 - Frame is 12 and 12, Good.
H:01 13 00 13 T:FF 13 FE 13 - Frame is 13 and 13, Good.
H:01 14 00 14 T:FF 14 FE 14 - Frame is 14 and 14, Good.
H:01 15 00 15 T:FF 15 FE 15 , Good.
H:01 16 00 16 T:FF 16 FE 16 , Good.
H:01 17 00 17 T:FF 17 FE 17 , Good.
H:01 18 00 18 T:FF 18 FE 18 , Good.
H:01 19 00 19 T:FF 19 FE 19 , Good.
H:01 1A 00 1A T:FF 1A FE 1A , Good.
H:01 1B 00 1B T:FF 1B FE 1B , Good.
H:01 1D 00 1D T:FF 1D FE 1D , Good.
H:01 20 00 20 T:FF 20 FE 20 - Frame is 20 and 20, Because the system is busy, the frame is missing (Frame 1E & 1F missed).
H:01 22 00 22 T:FF 22 FE 22
H:01 26 00 26 T:FF 26 FE 26
H:01 28 00 28 T:FF 28 FE 28
H:01 2A 00 2A T:FF 2A FE 2A

The trace error log is here:

# entries-in-buffer/entries-written: 1153/1153   #P:6
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/3:2-2621  [003] ...1  2981.372705: rtos_queue_peek_from_isr_failed: tstamp:93356815067 queue:0x0b4a3c58
     kworker/3:2-2621  [003] ...1  2981.372719: rtcpu_start: tstamp:93356817475
     kworker/3:2-2621  [003] ...1  2981.532704: rtcpu_vinotify_handle_msg: tstamp:93361472157 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:3167158420 data:0x00000100
     kworker/3:2-2621  [003] ...1  2981.532718: rtcpu_vinotify_handle_msg: tstamp:93361545842 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3167232034 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.532721: rtcpu_vinotify_handle_msg: tstamp:93361546109 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3167232044 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.532725: rtcpu_vinotify_handle_msg: tstamp:93361555465 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3167241801 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.532733: rtos_queue_peek_from_isr_failed: tstamp:93361816032 queue:0x0b4a3c58
     kworker/3:2-2621  [003] ...1  2981.532736: rtcpu_vinotify_handle_msg: tstamp:93362045435 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3167731762 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.532740: rtcpu_vinotify_handle_msg: tstamp:93362049577 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3167735877 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.532743: rtcpu_vinotify_handle_msg: tstamp:93362066672 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3167752884 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.532746: rtcpu_vinotify_handle_msg: tstamp:93362066916 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3167752895 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.532749: rtcpu_vinotify_handle_msg: tstamp:93362076654 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3167762975 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.585334: rtcpu_vinotify_handle_msg: tstamp:93362566309 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3168252612 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.585352: rtcpu_vinotify_handle_msg: tstamp:93362570426 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3168256727 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.585356: rtcpu_vinotify_handle_msg: tstamp:93362587537 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3168273735 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.585360: rtcpu_vinotify_handle_msg: tstamp:93362587781 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3168273745 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.585364: rtcpu_vinotify_handle_msg: tstamp:93362593846 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3168280171 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.585368: rtcpu_vinotify_handle_msg: tstamp:93363087155 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3168773463 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.585373: rtcpu_vinotify_handle_msg: tstamp:93363091278 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3168777578 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.585377: rtcpu_vinotify_handle_msg: tstamp:93363108388 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3168794586 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.585381: rtcpu_vinotify_handle_msg: tstamp:93363108637 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3168794595 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.585385: rtcpu_vinotify_handle_msg: tstamp:93363115242 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3168801573 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.585390: rtcpu_vinotify_handle_msg: tstamp:93363608026 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3169294314 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.585394: rtcpu_vinotify_handle_msg: tstamp:93363612141 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3169298427 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.585398: rtcpu_vinotify_handle_msg: tstamp:93363629245 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3169315436 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.585402: rtcpu_vinotify_handle_msg: tstamp:93363629488 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3169315447 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.585406: rtcpu_vinotify_handle_msg: tstamp:93363635450 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3169321773 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.585411: rtcpu_vinotify_handle_msg: tstamp:93364128838 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3169815164 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.585414: rtcpu_vinotify_handle_msg: tstamp:93364132996 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3169819279 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.585419: rtcpu_vinotify_handle_msg: tstamp:93364150115 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3169836286 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.585423: rtcpu_vinotify_handle_msg: tstamp:93364150403 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3169836297 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.585427: rtcpu_vinotify_handle_msg: tstamp:93364155879 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3169842209 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.636673: rtcpu_vinotify_handle_msg: tstamp:93364649703 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3170336014 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.636683: rtcpu_vinotify_handle_msg: tstamp:93364653828 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3170340129 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.636687: rtcpu_vinotify_handle_msg: tstamp:93364670927 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3170357137 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.636690: rtcpu_vinotify_handle_msg: tstamp:93364671163 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3170357147 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.636693: rtcpu_vinotify_handle_msg: tstamp:93364677907 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3170364243 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.636696: rtcpu_vinotify_handle_msg: tstamp:93365170549 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3170856865 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.636699: rtcpu_vinotify_handle_msg: tstamp:93365174676 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3170860979 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.636702: rtcpu_vinotify_handle_msg: tstamp:93365191782 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3170877987 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.636705: rtcpu_vinotify_handle_msg: tstamp:93365192030 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3170877998 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.636708: rtcpu_vinotify_handle_msg: tstamp:93365198573 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3170884911 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.636711: rtcpu_vinotify_handle_msg: tstamp:93365691383 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3171377716 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.636714: rtcpu_vinotify_handle_msg: tstamp:93365695532 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3171381830 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.636718: rtcpu_vinotify_handle_msg: tstamp:93365712645 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3171398837 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.636721: rtcpu_vinotify_handle_msg: tstamp:93365712909 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3171398849 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.636724: rtcpu_vinotify_handle_msg: tstamp:93365717814 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3171404138 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.688695: rtcpu_vinotify_handle_msg: tstamp:93366212258 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3171898566 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.688705: rtcpu_vinotify_handle_msg: tstamp:93366216377 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3171902681 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.688709: rtcpu_vinotify_handle_msg: tstamp:93366233490 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3171919688 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.688712: rtcpu_vinotify_handle_msg: tstamp:93366233746 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3171919698 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.688715: rtcpu_vinotify_handle_msg: tstamp:93366238446 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3171924771 data:0x08000000
     kworker/3:2-2621  [003] ...1  2981.688718: rtcpu_vinotify_handle_msg: tstamp:93366733109 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:3172419416 data:0x02cf0002
     kworker/3:2-2621  [003] ...1  2981.688721: rtcpu_vinotify_handle_msg: tstamp:93366737229 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:3172423531 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.688724: rtcpu_vinotify_handle_msg: tstamp:93366754321 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:3172440539 data:0x00000001
     kworker/3:2-2621  [003] ...1  2981.688728: rtcpu_vinotify_handle_msg: tstamp:93366754548 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:3172440549 data:0x00000000
     kworker/3:2-2621  [003] ...1  2981.688731: rtcpu_vinotify_handle_msg: tstamp:93366759651 tag:CHANSEL_LOAD_FRAMED channel:0x04 frame:0 vi_tstamp:3172445990 data:0x08000000

When does CHANSEL_LOAD_FRAMED occur in the log?
If CHANSEL_LOAD_FRAMED means that the system will take the buffer,
Should CHANSEL_LOAD_FRAMED be followed by CHANSEL_PXL_EOF or ATOMP_FE?

  • Operation Sequence ( by tstamp )
  1. SOF : Start of Frame
  2. FS : Frame Start
  3. LOAD : Load Frame
  4. EOF : End of Frame
  5. FE

Suggested Operation Sequence

  1. SOF : Start of Frame
  2. FS : Frame Start
  3. EOF : End of Frame
  4. FE:
  5. LOAD : Load Frame

I have summarized the contents above. (is same with the attached file)
https://drive.google.com/file/d/1IUaHhifU9tliw6PRqDD8E1S2ZTyqvd2g/view?usp=sharing

Device tree is here:

/ {
	host1x {
		vi@15700000 {
			status = "okay";
			num-channels = <1>;
			ports {
				status = "okay";
				#address-cells = <1>;
				#size-cells = <0>;
				port@0 {
					status = "okay";
					reg = <0>;
					fusion_camera_vi_in0: endpoint {
						status = "okay";
						csi-port = <2>;
						bus-width = <2>;
						remote-endpoint = <&fusion_camera_csi_out0>;
					};
				};
			};
		};

		nvcsi@150c0000 {
			status = "okay";
			num-channels = <1>;
			#address-cells = <1>;
			#size-cells = <0>;
			channel@0 {
				status = "okay";
				reg = <0>;
				ports {
					status = "okay";
					#address-cells = <1>;
					#size-cells = <0>;
					/* port@0 - module ports*/
					port@0 {
						status = "okay";
						reg = <0>;
						ir_csi_in0: endpoint@0 {
							csi-port = <2>;
							bus-width = <2>;
							remote-endpoint = <&fusion_camera_out0>;
						};
					};
					/* port@1 -> vi endpoint*/
					port@1 {
						status = "okay";
						reg = <1>;
						fusion_camera_csi_out0: endpoint@1 {
							remote-endpoint = <&fusion_camera_vi_in0>;
						};
					};
				};
			};
		};
	};

	i2c@3180000 { 
		status = "okay";
		crosslink_camera_c@7c {
			status = "okay";
			compatible = "nvidia,crosslink_camera";
			reg = <0x7c>;

			physical_w = "3.674";
			physical_h = "2.738";

			sensor_model ="crosslink_camera";
			devnode = "video0";

			avdd-reg = "vana";
			iovdd-reg = "vif";

			post_crop_frame_drop = "0";
			mode0 {
				mclk_khz = "24000";
				num_lanes = "2";
				tegra_sinterface = "serial_c";
				discontinuous_clk = "yes";
				dpcm_enable = "false";
				cil_settletime = "0";

				active_w = "1280";
				active_h = "720";
				pixel_t = "uyvy";
				readout_orientation = "0";
				line_length = "2560";
				inherent_gain = "1";
				mclk_multiplier = "17.0";
				pix_clk_hz = "288000000";

				min_gain_val = "1.0";
				max_gain_val = "16";
				min_hdr_ratio = "1";
				max_hdr_ratio = "64";
				min_framerate = "1.816577";
				max_framerate = "60";
				min_exp_time = "17";
				max_exp_time = "275192";
				embedded_metadata_height = "0";
			};

			ports {
				#address-cells = <1>;
				#size-cells = <0>;

				port@0 {
					reg = <0>;
					fusion_camera_out0: endpoint {
						status = "okay";
						csi-port = <2>;
						bus-width = <2>;
						remote-endpoint = <&ir_csi_in0>;
					};
				};

			};
		};

	};

	crosslink_3cam_lens_model1@P5V27C {
		min_focus_distance = "0.0";
		hyper_focal = "0.0";
		focal_length = "2.67";
		f_number = "2.0";
		aperture = "2.0";
	};

	tegra-camera-platform {
		compatible = "nvidia, tegra-camera-platform";
		
		num_csi_lanes = <2>;
		max_lane_speed = <1500000>;
		min_bits_per_pixel = <10>;
		vi_peak_byte_per_pixel = <2>;
		vi_bw_margin_pct = <25>;
		max_pixel_rate = <160000>;
		isp_peak_byte_per_pixel = <5>;
		isp_bw_margin_pct = <25>;

		modules {
			module0 {
				badge = "crosslink_camera_front_P5V27C";
				position = "rear";
				orientation = "1";
				drivernode0 {
					/* Declare PCL support driver (classically known as guid)  */
					pcl_id = "v4l2_sensor";
					/* Driver v4l2 device name */
					devname = "crosslink_camera 2-007c";
					/* Declare the device-tree hierarchy to driver instance */
					proc-device-tree = "/proc/device-tree/i2c@3180000/crosslink_camera_c@7c";
				};
				drivernode1 {
					/* Declare PCL support driver (classically known as guid)  */
					pcl_id = "v4l2_lens";
					proc-device-tree = "/proc/device-tree/crosslink_3cam_lens_model1@P5V27C/";
				};
			};
		};
	};
};

Tx2_FrameError.pdf (187 KB)
yuv.zip (12.1 KB)

hello KIMKS,

FYI,

  • CHANSEL_LOAD_FRAMED is emitted when a LOAD command is received for a channel while that channel is currently in a frame.
  • ATOMP_FE event occur once the NVCSI FE that was matched in CHANSEL have reached ATOMP.

may I know which JetPack release you’re working on?
thanks

I am using JetPack v3.2.

hello KIMKS,

could you please update your kernel driver with apply the patches in the attachment,
thanks
Aug6_Topic1038067.tar.gz (1.82 KB)

It works very well.
The first data was not broken, and there were no errors for four hours.
Thanks!!!

Is there a patch for JetPack 3.1?