TX2 NX wrong camera data while test pattern works good

Hi,

I’m facing issue with camera driver. The driver is working fine with Nano, trying to make it work on TX2 NX with custom carrier (carrier board used for both Nano and TX2 NX).

Here is part of DTS.

cam_port_0@1a {
				compatible = "framos,imx485";
				reg = <0x1a>;
				devnode = "video0";
				physical_w = "20.00";
				physical_h = "16.80";
				sensor_model ="imx485";
				post_crop_frame_drop = "0";
				use_decibel_gain = "true";
				delayed_gain = "false";
				use_sensor_mode_id = "false";

				mode0 { //
					mclk_khz = "37125";
					num_lanes = "4";
					tegra_sinterface = "serial_a";
					phy_mode = "DPHY";
					discontinuous_clk = "no";
					dpcm_enable = "false";
					cil_settletime = "0";
					active_w = "3864";
					active_h = "2180";

					dynamic_pixel_bit_depth = "12";
					csi_pixel_bit_depth = "12";
					mode_type = "bayer";
					pixel_phase = "rggb";

					readout_orientation = "0";
					line_length = "6600";
					inherent_gain = "1";
					mclk_multiplier = "20.48";
					pix_clk_hz = "760000000";
					embedded_metadata_height = "1";
				};

Capturing the frame by v4l2-ctl:
v4l2-ctl --verbose --set-fmt-video=width=3864,height=2180,pixelformat=RG12 --set-ctrl bypass_mode=0,frame_rate=5000000,exposure=30000,gain=16 --stream-mmap --stream-count=1 --stream-to=imx485_4k.raw -d /dev/video0
works good with image sensor pattern generator, but real data (pattern generator off) looks almost dark, the frame data is almost 0xbx or 0xcx, even with long exposure time. Average of all pixels is 0xbb.

root@orbit-tx2nx:~# hexdump  imx485_4k.raw | head
0000000 00bc 00c0 00c0 00c0 00bc 00c0 00c0 00c0
0000010 00c0 00bc 00c0 00bc 00c0 00c0 00c8 00c0
0000020 00c0 00c4 00c0 00c0 00c0 00c4 00c0 00c0
0000030 00bc 00bc 00c4 00c4 00c8 00c0 00c4 00c0
0000040 00c4 00c0 00b8 00c0 00c4 00c0 00bc 00c0
0000050 00c4 00c0 00c0 00bc 00bc 00c0 00c4 00c0
0000060 00c4 00c0 00c4 00c0 00c0 00c0 00c4 00bc
0000070 00b8 00c0 00bc 00bc 00bc 00c0 00c0 00c0
0000080 00bc 00bc 00c0 00c0 00c0 00bc 00c0 00bc
0000090 00b8 00c0 00c4 00c4 00bc 00bc 00bc 00c4

The trace buffer shows many isr_failed

root@orbit-tx2nx:~# cat /sys/kernel/debug/tracing/trace                                                                                                           
# tracer: nop                                                                                                                                                            
#                                                                                                                                                                            
# entries-in-buffer/entries-written: 180/180   #P:4                                                                                                                      
#                                                                                                                                                                 
#                              _-----=> irqs-off                                                                     
#                             / _----=> need-resched                                                                 
#                            | / _---=> hardirq/softirq                                                                                                           
#                            || / _--=> preempt-depth                                                                                                                    
#                            ||| /     delay                                                                                                                                 
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION                                                                                                                 
#              | |       |   ||||       |         |                                                                                                               
     kworker/5:0-6722  [005] ....  3239.485317: rtos_queue_peek_from_isr_failed: tstamp:101492290712 queue:0x0b4b4500                                                                                        
     kworker/5:0-6722  [005] ....  3239.597320: rtos_queue_peek_from_isr_failed: tstamp:101497290707 queue:0x0b4b4500                                   
     kworker/5:0-6722  [005] ....  3239.765320: rtos_queue_peek_from_isr_failed: tstamp:101502290700 queue:0x0b4b4500                                             
     kworker/5:0-6722  [005] ....  3239.933344: rtos_queue_peek_from_isr_failed: tstamp:101507290694 queue:0x0b4b4500                                                    
     kworker/5:0-6722  [005] ....  3240.045332: rtos_queue_peek_from_isr_failed: tstamp:101510201756 queue:0x0b4b4500                                                        
        v4l2-ctl-8558  [005] ....  3273.325561: tegra_channel_open: vi-output, imx485 9-001a                         
        v4l2-ctl-8558  [000] ....  3273.326963: tegra_channel_set_power: imx485 9-001a : 0x1                                                                                                                 
        v4l2-ctl-8558  [000] ....  3273.326975: camera_common_s_power: status : 0x1                                                                               
        v4l2-ctl-8558  [000] ....  3273.384458: tegra_channel_set_power: 150c0000.nvcsi--6 : 0x1                     
        v4l2-ctl-8558  [000] ....  3273.384462: csi_s_power: enable : 0x1                                            
        v4l2-ctl-8558  [004] ....  3273.420788: tegra_channel_capture_setup: vnc_id 0 W 3864 H 2180 fmt 20                                                        
 vi-output, imx4-8559  [003] ....  3273.429013: tegra_channel_set_stream: enable : 0x1                                                                                   
 vi-output, imx4-8559  [003] ....  3273.430662: tegra_channel_set_stream: 150c0000.nvcsi--6 : 0x1                                                                            
 vi-output, imx4-8559  [003] ....  3273.430665: csi_s_stream: enable : 0x1                                                                                               
     kworker/5:0-6722  [005] ....  3273.433256: rtos_queue_peek_from_isr_failed: tstamp:102554420778 queue:0x0b4b4500                                             
     kworker/5:0-6722  [005] ....  3273.433258: rtcpu_start: tstamp:102554421732                                     
     kworker/5:0-6722  [005] ....  3273.433260: rtos_queue_send_from_isr_failed: tstamp:102554428225 queue:0x0b4a7258                                                                                        
     kworker/5:0-6722  [005] ....  3273.433261: rtos_queue_send_from_isr_failed: tstamp:102554428331 queue:0x0b4aad68                                             
     kworker/5:0-6722  [005] ....  3273.433262: rtos_queue_send_from_isr_failed: tstamp:102554428439 queue:0x0b4ac998                                                    
     kworker/5:0-6722  [005] ....  3273.433262: rtos_queue_send_from_isr_failed: tstamp:102554428546 queue:0x0b4af2d8                                                        
     kworker/5:0-6722  [005] ....  3273.433263: rtos_queue_send_from_isr_failed: tstamp:102554428650 queue:0x0b4b0098                                                    
     kworker/5:0-6722  [005] ....  3273.433263: rtos_queue_send_from_isr_failed: tstamp:102554428755 queue:0x0b4b0e58                                             
     kworker/5:0-6722  [005] ....  3273.433264: rtos_queue_send_from_isr_failed: tstamp:102554428860 queue:0x0b4b1c18
     kworker/5:0-6722  [005] ....  3273.433266: rtos_queue_send_failed: tstamp:102554429318 queue:0x0b4a7258         
     kworker/5:0-6722  [005] ....  3273.433266: rtos_queue_send_from_isr_failed: tstamp:102554431350 queue:0x0b4a7258                                             
     kworker/5:0-6722  [005] ....  3273.433267: rtos_queue_send_from_isr_failed: tstamp:102554431456 queue:0x0b4aad68                                                    
     kworker/5:0-6722  [005] ....  3273.433268: rtos_queue_send_from_isr_failed: tstamp:102554431562 queue:0x0b4ac998                                                        
     kworker/5:0-6722  [005] ....  3273.433268: rtos_queue_send_from_isr_failed: tstamp:102554431670 queue:0x0b4af2d8                                                    
     kworker/5:0-6722  [005] ....  3273.433269: rtos_queue_send_from_isr_failed: tstamp:102554431774 queue:0x0b4b0098                                             
     kworker/5:0-6722  [005] ....  3273.433269: rtos_queue_send_from_isr_failed: tstamp:102554431878 queue:0x0b4b0e58
     kworker/5:0-6722  [005] ....  3273.433270: rtos_queue_send_from_isr_failed: tstamp:102554431981 queue:0x0b4b1c18                                             
     kworker/5:0-6722  [005] ....  3273.433271: rtos_queue_send_failed: tstamp:102554432905 queue:0x0b4a7258                                                             
 vi-output, imx4-8559  [003] ....  3273.485968: tegra_channel_set_stream: imx485 9-001a : 0x1                        
     kworker/5:0-6722  [005] ....  3273.601264: rtos_queue_peek_from_isr_failed: tstamp:102559421109 queue:0x0b4b4500                                                    
     kworker/5:0-6722  [005] ....  3273.769269: rtos_queue_peek_from_isr_failed: tstamp:102564421099 queue:0x0b4b4500                                             
     kworker/5:0-6722  [005] ....  3273.937319: rtos_queue_peek_from_isr_failed: tstamp:102569421093 queue:0x0b4b4500
 vi-output, imx4-8559  [004] ....  3273.972653: tegra_channel_capture_frame: sof:3273.859038048                      
    kworker/5:0-6722  [005] ....  3273.993307: rtcpu_vinotify_event: tstamp:102571412844 tag:ATOMP_FS channel:0x00 frame:1 vi_tstamp:102571412458 data:0x00000000
     kworker/5:0-6722  [005] ....  3273.993313: rtcpu_vinotify_event: tstamp:102571417709 tag:CHANSEL_PXL_SOF channel:0x00 frame:1 vi_tstamp:102571417322 data:0x00000001
     kworker/5:0-6722  [005] ....  3273.993315: rtcpu_vinotify_event: tstamp:102571680542 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:1 vi_tstamp:102571680153 data:0x08000000
     kworker/5:0-6722  [005] ....  3273.993317: rtcpu_vinotify_event: tstamp:102571923088 tag:CHANSEL_PXL_EOF channel:0x00 frame:1 vi_tstamp:102571922386 data:0x08830002
     kworker/5:0-6722  [005] ....  3273.993319: rtcpu_vinotify_event: tstamp:102571923211 tag:ATOMP_FE channel:0x00 frame:1 vi_tstamp:102571922402 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.105314: rtos_queue_peek_from_isr_failed: tstamp:102574421094 queue:0x0b4b4500
 vi-output, imx4-8559  [004] ....  3274.172804: tegra_channel_capture_frame: sof:3274.59236416
     kworker/5:0-6722  [005] ....  3274.217310: rtcpu_vinotify_event: tstamp:102577669039 tag:ATOMP_FS channel:0x00 frame:2 vi_tstamp:102577668657 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.217316: rtcpu_vinotify_event: tstamp:102577673903 tag:CHANSEL_PXL_SOF channel:0x00 frame:2 vi_tstamp:102577673521 data:0x00000001
     kworker/5:0-6722  [005] ....  3274.217318: rtcpu_vinotify_event: tstamp:102577935268 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:2 vi_tstamp:102577934877 data:0x08000000
     kworker/5:0-6722  [005] ....  3274.217320: rtcpu_vinotify_event: tstamp:102578179282 tag:CHANSEL_PXL_EOF channel:0x00 frame:2 vi_tstamp:102578178585 data:0x08830002
     kworker/5:0-6722  [005] ....  3274.217323: rtcpu_vinotify_event: tstamp:102578179406 tag:ATOMP_FE channel:0x00 frame:2 vi_tstamp:102578178601 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.273303: rtos_queue_peek_from_isr_failed: tstamp:102579421080 queue:0x0b4b4500
 vi-output, imx4-8559  [004] ....  3274.372998: tegra_channel_capture_frame: sof:3274.259434816
     kworker/5:0-6722  [005] ....  3274.385329: rtcpu_vinotify_event: tstamp:102583925238 tag:ATOMP_FS channel:0x00 frame:3 vi_tstamp:102583924856 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.385336: rtcpu_vinotify_event: tstamp:102583930094 tag:CHANSEL_PXL_SOF channel:0x00 frame:3 vi_tstamp:102583929721 data:0x00000001
     kworker/5:0-6722  [005] ....  3274.385338: rtcpu_vinotify_event: tstamp:102584191305 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:3 vi_tstamp:102584190921 data:0x08000000
     kworker/5:0-6722  [005] ....  3274.385342: rtos_queue_peek_from_isr_failed: tstamp:102584421075 queue:0x0b4b4500
     kworker/5:0-6722  [005] ....  3274.385344: rtcpu_vinotify_event: tstamp:102584435476 tag:CHANSEL_PXL_EOF channel:0x00 frame:3 vi_tstamp:102584434784 data:0x08830002
     kworker/5:0-6722  [005] ....  3274.385346: rtcpu_vinotify_event: tstamp:102584435592 tag:ATOMP_FE channel:0x00 frame:3 vi_tstamp:102584434800 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.553320: rtos_queue_peek_from_isr_failed: tstamp:102589421064 queue:0x0b4b4500
 vi-output, imx4-8559  [004] ....  3274.573274: tegra_channel_capture_frame: sof:3274.459633184
     kworker/5:0-6722  [005] ....  3274.609316: rtcpu_vinotify_event: tstamp:102590181441 tag:ATOMP_FS channel:0x00 frame:4 vi_tstamp:102590181056 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.609322: rtcpu_vinotify_event: tstamp:102590186294 tag:CHANSEL_PXL_SOF channel:0x00 frame:4 vi_tstamp:102590185920 data:0x00000001
     kworker/5:0-6722  [005] ....  3274.609324: rtcpu_vinotify_event: tstamp:102590449933 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:4 vi_tstamp:102590449542 data:0x08000000
     kworker/5:0-6722  [005] ....  3274.609326: rtcpu_vinotify_event: tstamp:102590691685 tag:CHANSEL_PXL_EOF channel:0x00 frame:4 vi_tstamp:102590690984 data:0x08830002
     kworker/5:0-6722  [005] ....  3274.609328: rtcpu_vinotify_event: tstamp:102590691807 tag:ATOMP_FE channel:0x00 frame:4 vi_tstamp:102590691000 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.721300: rtos_queue_peek_from_isr_failed: tstamp:102594421059 queue:0x0b4b4500
 vi-output, imx4-8559  [004] ....  3274.773433: tegra_channel_capture_frame: sof:3274.659831552
     kworker/5:0-6722  [005] ....  3274.777317: rtcpu_vinotify_event: tstamp:102596437637 tag:ATOMP_FS channel:0x00 frame:5 vi_tstamp:102596437255 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.777323: rtcpu_vinotify_event: tstamp:102596442494 tag:CHANSEL_PXL_SOF channel:0x00 frame:5 vi_tstamp:102596442119 data:0x00000001
     kworker/5:0-6722  [005] ....  3274.777325: rtcpu_vinotify_event: tstamp:102596704913 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:5 vi_tstamp:102596704523 data:0x08000000
     kworker/5:0-6722  [005] ....  3274.833323: rtcpu_vinotify_event: tstamp:102596947886 tag:CHANSEL_PXL_EOF channel:0x00 frame:5 vi_tstamp:102596947184 data:0x08830002
     kworker/5:0-6722  [005] ....  3274.833328: rtcpu_vinotify_event: tstamp:102596948009 tag:ATOMP_FE channel:0x00 frame:5 vi_tstamp:102596947199 data:0x00000000
     kworker/5:0-6722  [005] ....  3274.893285: rtos_queue_peek_from_isr_failed: tstamp:102599421052 queue:0x0b4b4500
 vi-output, imx4-8559  [004] ....  3274.973614: tegra_channel_capture_frame: sof:3274.860029920
     kworker/5:0-6722  [005] ....  3275.005319: rtcpu_vinotify_event: tstamp:102602693837 tag:ATOMP_FS channel:0x00 frame:6 vi_tstamp:102602693454 data:0x00000000
     kworker/5:0-6722  [005] ....  3275.005326: rtcpu_vinotify_event: tstamp:102602698697 tag:CHANSEL_PXL_SOF channel:0x00 frame:6 vi_tstamp:102602698318 data:0x00000001
     kworker/5:0-6722  [005] ....  3275.005329: rtcpu_vinotify_event: tstamp:102602961691 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:6 vi_tstamp:102602961300 data:0x08000000
     kworker/5:0-6722  [005] ....  3275.005331: rtcpu_vinotify_event: tstamp:102603204083 tag:CHANSEL_PXL_EOF channel:0x00 frame:6 vi_tstamp:102603203383 data:0x08830002
     kworker/5:0-6722  [005] ....  3275.005333: rtcpu_vinotify_event: tstamp:102603204211 tag:ATOMP_FE channel:0x00 frame:6 vi_tstamp:102603203398 data:0x00000000
     kworker/5:0-6722  [005] ....  3275.061319: rtos_queue_peek_from_isr_failed: tstamp:102604421049 queue:0x0b4b4500
     kworker/5:0-6722  [005] ....  3275.173317: rtcpu_vinotify_event: tstamp:102608950037 tag:ATOMP_FS channel:0x00 frame:7 vi_tstamp:102608949654 data:0x00000000
     kworker/5:0-6722  [005] ....  3275.173324: rtcpu_vinotify_event: tstamp:102608954893 tag:CHANSEL_PXL_SOF channel:0x00 frame:7 vi_tstamp:102608954518 data:0x00000001
     kworker/5:0-6722  [005] ....  3275.233418: rtos_queue_peek_from_isr_failed: tstamp:102609421047 queue:0x0b4b4500
     kworker/5:0-6722  [005] ....  3275.233426: rtcpu_vinotify_event: tstamp:102609460274 tag:CHANSEL_PXL_EOF channel:0x00 frame:7 vi_tstamp:102609459582 data:0x08830002
     kworker/5:0-6722  [005] ....  3275.233428: rtcpu_vinotify_event: tstamp:102609460391 tag:ATOMP_FE channel:0x00 frame:7 vi_tstamp:102609459597 data:0x00000000
     kworker/5:0-6722  [005] ....  3275.233461: rtos_queue_send_from_isr_failed: tstamp:102609809619 queue:0x0b4b0098
     kworker/5:0-6722  [005] ....  3275.233464: rtos_queue_send_from_isr_failed: tstamp:102609809722 queue:0x0b4b0e58
     kworker/5:0-6722  [005] ....  3275.233466: rtos_queue_send_from_isr_failed: tstamp:102609809826 queue:0x0b4b1c18
     kworker/5:0-6722  [005] ....  3275.233468: rtos_queue_send_failed: tstamp:102609810248 queue:0x0b4a7258
        v4l2-ctl-8558  [005] ....  3275.276929: tegra_channel_capture_setup: vnc_id 0 W 3864 H 2180 fmt 20
        v4l2-ctl-8558  [005] ....  3275.284973: tegra_channel_capture_done: eof:-271621375232.-549610857612
        v4l2-ctl-8558  [004] ....  3275.285142: tegra_channel_set_stream: enable : 0x0
        v4l2-ctl-8558  [004] ....  3275.285144: tegra_channel_set_stream: imx485 9-001a : 0x0

Kernel message shows:

[ 3273.697092] imx485 9-001a: fm_update_ctrl_range:  mode: 0, exposure range [14, 199989]
[ 3273.706089] imx485 9-001a: imx485_set_frame_rate: val: 5000000, frame_length set: 27002
[ 3273.714884] imx485 9-001a: imx485_set_black_level: black level: 50
[ 3273.964425] video4linux video0: vi_notify_wait: vi4 got SOF syncpt buf[ffffffc0cfc55000]
[ 3274.164618] video4linux video0: vi_notify_wait: vi4 got SOF syncpt buf[ffffffc0cfc55c00]
[ 3274.364803] video4linux video0: vi_notify_wait: vi4 got SOF syncpt buf[ffffffc0cfc56400]
[ 3274.565008] video4linux video0: vi_notify_wait: vi4 got SOF syncpt buf[ffffffc0cfc56800]
[ 3274.765196] video4linux video0: vi_notify_wait: vi4 got SOF syncpt buf[ffffffc0cfc55000]
[ 3274.965424] video4linux video0: vi_notify_wait: vi4 got SOF syncpt buf[ffffffc0cfc55c00]
[ 3275.177360] tegra-vi4 15700000.vi: ATOMP_FE syncpt timeout!
[ 3275.183109] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[ 3275.192915] nvcsi 150c0000.nvcsi: csi4_stop_streaming port_idx=0, lanes=4
[ 3275.199787] nvcsi 150c0000.nvcsi: csi4_phy_config
[ 3275.204591] nvcsi 150c0000.nvcsi: NVCSI_CIL_CONFIG = 00000004
[ 3275.210448] nvcsi 150c0000.nvcsi: csi4_stream_check_status
[ 3275.215981] nvcsi 150c0000.nvcsi: csi4_cil_check_status 397
[ 3275.221634] nvcsi 150c0000.nvcsi: csi4_start_streaming port_idx=0, lanes=4
[ 3275.228542] nvcsi 150c0000.nvcsi: csi4_stream_init
[ 3275.233517] nvcsi 150c0000.nvcsi: csi4_stream_config
[ 3275.238528] nvcsi 150c0000.nvcsi: csi4_stream_config (0) read VC0_DPCM_CTRL = 00000000
[ 3275.246545] nvcsi 150c0000.nvcsi: csi4_phy_config
[ 3275.251283] nvcsi 150c0000.nvcsi: NVCSI_CIL_CONFIG = 00000004
[ 3275.257047] nvcsi 150c0000.nvcsi: cil core clock: 204, csi clock: 760
[ 3275.263491] nvcsi 150c0000.nvcsi: cil_settingtime was autocalculated
[ 3275.269866] nvcsi 150c0000.nvcsi: csi settle time: 33, cil settle time: 19
[ 3275.276957] tegra-vi4 15700000.vi: Create Surface with imgW=3864, imgH=2180, memFmt=32
[ 3275.539103] nvcsi 150c0000.nvcsi: csi4_stop_streaming port_idx=0, lanes=4
[ 3275.546045] nvcsi 150c0000.nvcsi: csi4_phy_config
[ 3275.550834] nvcsi 150c0000.nvcsi: NVCSI_CIL_CONFIG = 00000004
[ 3275.556658] nvcsi 150c0000.nvcsi: csi4_stream_check_status
[ 3275.562270] nvcsi 150c0000.nvcsi: csi4_cil_check_status 397

The ATOMP_FE syncpt timeout! at end of the capturing session is abnormal, but I’m not sure it is relevant.

Can you please point me out what could be the reason? Thank you so much.

Try --stream-skip=10 to check the raw image again.

Hi @ShaneCCC ,
Thank you for your reponse.

With --stream-skip=10 added, the raw image is still same value range. The messages on tracing and dmesg is same (just the number of frame is increased).
The content of the raw image between two raw files are not the same, but the value range is still same. Median and average of pixel value is about 0xb8.

Have a check the gain/exposure/frame rate configure

1 Like

Hi @ShaneCCC ,
Thanks. It looks like the testing room is totally dark and the IRIS is fully closed then the image is dark. I increase gain to very high and can see noise in output. I will check more.

However, The messages “ATOMP_FE syncpt timeout!” in dmesg and “queue _isr_failed” in trace is normal? I don’t see them in Jetson Nano.

Time out may could be lost the frame due to frame rate too low or something else.

1 Like

Thank you for kindly help @ShaneCCC .
After adjusting light and IRIS, the camera works nice.
I have tried to increase frame rate, and the timeout message was gone.