Imx477 on devkit and custom pcb haunted

Hi,

We have an imx477 single camera working on the P3518 Xavier NX devkit with a TX2-NX module both with gstreamer (nvarguscamerasrc) and v4l2-ctl.

When we try to get the imx477 running on our custom pcb, also with TX2-NX module, we’re starting to get errors when using nvarguscamerasrc.

When using v4l2-ctl, we are able to stream and capture picture to disk which looks good.

When trying gstreamer, we get following msgs:

Streaming camera 0 at 4032x3040 to 10.0.200.194 at port 5000 ... 
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[  115.540506] imx477 2-001a: imx477_set_exposure: exposure limited by frame_length: 3092 [lines]
[  115.635798] imx477 2-001a: imx477_set_exposure: exposure limited by frame_length: 3092 [lines]
[  117.157533] fence timeout on [ffffffc0ca5b2e40] after 1500ms
[  117.157668] fence timeout on [ffffffc0ca5b2480] after 1500ms
[  117.157684] name=[nvhost_sync:22], current value=0 waiting value=1
[  117.157697] ---- mlocks ----
[  117.157718] 8: locked by channel 1
[  117.157743] 
[  117.157749] ---- syncpts ----
[  117.157778] id 8 (vblank0) min 6977 max 0 refs 1 (previous client : )
[  117.157812] id 22 (15600000.isp_nvargus-daemon_0) min 0 max 4 refs 4 (previous client : )
[  117.157824] id 23 (15600000.isp_nvargus-daemon_1) min 0 max 2 refs 4 (previous client : )
[  117.157835] id 24 (15600000.isp_nvargus-daemon_2) min 0 max 2 refs 4 (previous client : )
[  117.157846] id 25 (15600000.isp_nvargus-daemon_3) min 0 max 2 refs 4 (previous client : )
[  117.157860] id 28 (15600000.isp_nvargus-daemon_4) min 15 max 26 refs 9 (previous client : )
[  117.157871] id 29 (15600000.isp_nvargus-daemon_5) min 0 max 2 refs 4 (previous client : )
[  117.157883] id 31 (gp10b_506) min 52 max 52 refs 1 (previous client : )
[  117.157900] id 35 (15700000.vi_0) min 14 max 14 refs 1 (previous client : 15700000.vi_0)
[  117.157911] id 36 (15700000.vi_0) min 1 max 2 refs 4 (previous client : 15700000.vi_0)
[  117.157922] id 37 (15700000.vi_1) min 0 max 2 refs 4 (previous client : 15700000.vi_1)
[  117.157932] id 38 (15700000.vi_2) min 0 max 2 refs 4 (previous client : 15700000.vi_2)
[  117.157944] id 39 (150c0000.nvcsi_0) min 26 max 26 refs 1 (previous client : 150c0000.nvcsi_0)
[  117.157957] id 42 (gp10b_502) min 8 max 8 refs 1 (previous client : )
[  117.157968] id 43 (gp10b_501) min 6 max 6 refs 1 (previous client : )
[  117.157978] id 44 (gp10b_500) min 6 max 6 refs 1 (previous client : )
[  117.157988] id 45 (gp10b_499) min 6 max 6 refs 1 (previous client : )
[  117.157998] id 46 (gp10b_498) min 6 max 6 refs 1 (previous client : )
[  117.158008] id 47 (gp10b_497) min 6 max 6 refs 1 (previous client : )

We looked around for fence timeout, but can’t find any solution. The board seems to get the ‘start of frame’ as no PXL_SOF is seen in the error messages.

A low-level message in the trace is “rtos_queue_peek_from_isr_failed”.

Our camera device tree in attachment.

On the PCB, we checked all power-supplies to the camera, reset/pwdn pins and they all work like on the devkit where the camera works.

We also used a spectrum analyser to see if data is flowing over the CSI lanes.

What more can we verify to get the IMX477 running with the TX2-NX board on our pcb?

nobi-mainboard-uno-cam-imx477.dtsi (5.5 KB)

Try modify the sensor CID function like set exposure as dummy to try.

Hi Shane,

Thanks for the suggestion; I altered the imx477.c function imx477_set_exposure() to do nothing and just return 0.

But this gave the same behaviour and errors as descibed above.
The v4l2 images look the same as before the change.

Open to more suggestions.

Please also modify set_frame_rate() as dummy function too.
And dump the device tree to confirm the setting.

sudo dtc -I fs -O dts -o extracted_proc.dts /proc/device-tree

Putting the set_frame_rate() as a dummy, did not change the behaviour as well.

I checked the device tree with the dtc tool, and nothing specific stands out; grepping for imx477 gives:

		nobi_imx477_cam0 = "/i2c@3180000/nobi_imx477_a@1a";
		nobi_imx477_csi_in0 = "/host1x/nvcsi@150c0000/channel@0/ports/port@0/endpoint@0";
		nobi_imx477_csi_out0 = "/host1x/nvcsi@150c0000/channel@0/ports/port@1/endpoint@1";
		nobi_imx477_vi_in0 = "/host1x/vi@15700000/ports/port@0/endpoint";
		nobi_imx477_out0 = "/i2c@3180000/nobi_imx477_a@1a/ports/port@0/endpoint";
		nobi_imx477_a@1a {
			compatible = "ridgerun,imx477";
			sensor_model = "imx477";
				badge = "nobi_imx477_front";
					devname = "imx477 2-001a";
					proc-device-tree = "/proc/device-tree/i2c@3180000/nobi_imx477_a@1a";

View the camera specifc sections:

                nobi_imx477_a@1a {                                                                              
                        compatible = "ridgerun,imx477";                                                         
                        sensor_model = "imx477";                                                                
                        physical_w = "7.564";                                                                   
                        set_mode_delay_ms = "100";                                                              
                        devnode = "video0";                                                                                
                        reset-gpios = <0x21 0x68 0x01>;                                                         
                        use_sensor_mode_id = "true";                                                                       
                        phandle = <0xdc>;                                                                                  
                        reg = <0x1a>;                                            
                        linux,phandle = <0xdc>;                               
                        physical_h = "5.476";                                                                   
                                                                                                                
                        mode0 {                                               
                                default_framerate = "30000000";                 
                                step_exp_time = [31 00];                        
                                line_length = "9024";                                                                      
                                active_w = "4032";                              
                                embedded_metadata_height = [32 00];             
                                pixel_phase = "rggb";                                      
                                pix_clk_hz = "840000000";                                                                  
                                dpcm_enable = "false";            
                                mode_type = "bayer";                                                        
                                num_lanes = [32 00];                                                                       
                                inherent_gain = [31 00];                                                        
                                max_gain_val = "356";                                                           
                                min_hdr_ratio = [31 00];                                                        
                                discontinuous_clk = "no";                                                       
                                readout_orientation = "90";                        
                                min_gain_val = "16";                                  
                                max_hdr_ratio = [31 00];                      
                                tegra_sinterface = "serial_a";                                                             
                                step_gain_val = [31 00];                                                    
                                default_gain = "16";                                                        
                                csi_pixel_bit_depth = "10";                                                                
                                min_framerate = "2000000";                        
                                default_exp_time = "2495";   
                                max_framerate = "30000000";               
                                exposure_factor = "1000000";                        
                                step_framerate = [31 00];                           
                                phy_mode = "DPHY";                                  
                                mclk_khz = "24000";                                 
                                cil_settletime = [30 00];                           
                                gain_factor = "16";                                 
                                framerate_factor = "1000000";                       
                                active_h = "3040";                                  
                                max_exp_time = "684709";         `
                                mclk_multiplier = "80";                                                         
                                min_exp_time = "13";                                                            
                        };                                                                                      
                                                                                                                
                        ports {                                                                                 
                                status = "okay";                                                                           
                                #address-cells = <0x01>;                                                        
                                #size-cells = <0x00>;                                                                      
                                                                                                                           
                                port@0 {                                         
                                        status = "okay";                      
                                        reg = <0x00>;                                                           
                                                                                                                
                                        endpoint {                            
                                                port-index = <0x00>;            
                                                remote-endpoint = <0x26>;       
                                                status = "okay";                                                           
                                                bus-width = <0x02>;             
                                                phandle = <0x4e>;               
                                                linux,phandle = <0x4e>;                    
                                        };                                                                                 
                                };                                
                        };                                                                                  
                               mode1 {                                                                                 
                                default_framerate = "60000000";                                                 
                                step_exp_time = [31 00];                                                        
                                line_length = "9024";                                                           
                                active_w = "1920";                                                                         
                                embedded_metadata_height = [32 00];                                             
                                pixel_phase = "rggb";                                                                      
                                pix_clk_hz = "840000000";                                                                  
                                dpcm_enable = "false";                           
                                mode_type = "bayer";                          
                                num_lanes = [32 00];                                                            
                                inherent_gain = [31 00];                                                        
                                max_gain_val = "356";                         
                                min_hdr_ratio = [31 00];                        
                                discontinuous_clk = "yes";                      
                                readout_orientation = "90";                                                                
                                min_gain_val = "16";                            
                                max_hdr_ratio = [31 00];                        
                                tegra_sinterface = "serial_a";                             
                                step_gain_val = [31 00];                                                                   
                                default_gain = "16";              
                                csi_pixel_bit_depth = "10";                                                 
                                min_framerate = "2000000";                                                                 
                                default_exp_time = "2495";                                                      
                                max_framerate = "60000000";                                                     
                                exposure_factor = "1000000";                                                    
                                step_framerate = [31 00];                                                       
                                phy_mode = "DPHY";                                 
                                mclk_khz = "24000";                                   
                                cil_settletime = [30 00];                     
                                gain_factor = "16";                                                                        
                                framerate_factor = "1000000";                                               
                                active_h = "1080";                                                          
                                max_exp_time = "500000";                                                                   
                                mclk_multiplier = "80";                           
                                min_exp_time = "11";         
                        };                                                
                };                                                                  
        };                                                                                                                                                                                           

And the tegra-camera-platform:

        tegra-camera-platform {                                                                                     
                isp_bw_margin_pct = <0x19>;                                                                                
                compatible = "nvidia, tegra-camera-platform";                                                              
                max_pixel_rate = <0x7270e0>;                                                                               
                num_csi_lanes = <0x02>;                                       
                vi_peak_byte_per_pixel = <0x02>;                                                                
                max_lane_speed = <0x16e360>;                                                                    
                phandle = <0x150>;                                            
                vi_bw_margin_pct = <0x19>;                                      
                linux,phandle = <0x150>;                                        
                min_bits_per_pixel = <0x0a>;                                                                               
                isp_peak_byte_per_pixel = <0x05>;                               
                                                                                
                modules {                                                                  
                                                                                                                           
                        module0 {                                                                                          
                                badge = "nobi_imx477_front";                                                
                                status = "okay";                                                                           
                                phandle = <0x151>;                                                              
                                position = "front";                                                             
                                linux,phandle = <0x151>;                                                        
                                orientation = [31 00];                                                          
                                                                                   
                                drivernode0 {                                              
                                        devname = "imx477 2-001a";            
                                        pcl_id = "v4l2_sensor";                                                            
                                        status = "okay";                                                    
                                        proc-device-tree = "/proc/device-tree/i2c@3180000/nobi_imx477_a@1a";               
                                        phandle = <0x152>;                                                                 
                                        linux,phandle = <0x152>;                  
                                };                           
                        };                                                
                };                                                                  
        };                                                                          

Nothing specific stands out to me as missing/different from the device-tree dts file we made.

Trying with a script that streams to a file:

#!/bin/sh
if [[ -z $1 || -z $2 || -z $3 ]]; then
  echo "gstreamer_test.sh XRES YRES OUTPUTFILE "
  exit 1
fi
XRES=$1
YRES=$2
FILE=$3
echo "Streaming camera 0 at ${XRES}x${YRES} to file ${FILE} ... "
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=(int)${XRES},height=(int)${YRES},format=NV12" ! nvvidconv ! jpegenc ! filesink location=${FILE}

This gives me following output:

Streaming camera 0 at 4032x3040 to file /dev/null ... 
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
Execution ended after 0:00:01.942234274
Setting pipeline to NULL ...
[ 2059.658464] imx477 2-001a: imx477_set_frame_rate: Jan's dummy set framerate function
[ 2061.182782] fence timeout on [ffffffc0b6a2f840] after 1500ms
[ 2061.182794] fence timeout on [ffffffc0b6a2f6c0] after 1500ms
[ 2061.182813] name=[nvhost_sync:25], current value=7 waiting value=8
[ 2061.182824] ---- mlocks ----
[ 2061.182845] 8: locked by channel 1
[ 2061.182870] 
[ 2061.182875] ---- syncpts ----
[ 2061.182906] id 8 (vblank0) min 123663 max 0 refs 1 (previous client : )
[ 2061.182938] id 20 (15340000.vic_gst-launch-1.0_0) min 103 max 103 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[ 2061.182951] id 21 (15700000.vi_0) min 125 max 125 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[ 2061.182963] id 22 (15600000.isp_nvargus-daemon_0) min 14 max 18 refs 4 (previous client : 15600000.isp_nvargus-daemon_0)
[ 2061.182976] id 23 (15600000.isp_nvargus-daemon_1) min 7 max 9 refs 4 (previous client : 15600000.isp_nvargus-daemon_1)
[ 2061.182987] id 24 (15600000.isp_nvargus-daemon_2) min 7 max 9 refs 4 (previous client : 15600000.isp_nvargus-daemon_2)
[ 2061.182998] id 25 (15600000.isp_nvargus-daemon_3) min 7 max 9 refs 4 (previous client : 15600000.isp_nvargus-daemon_3)
[ 2061.183013] id 28 (15600000.isp_nvargus-daemon_4) min 88 max 99 refs 9 (previous client : 15600000.isp_nvargus-daemon_4)
[ 2061.183024] id 29 (15600000.isp_nvargus-daemon_5) min 7 max 9 refs 4 (previous client : 15600000.isp_nvargus-daemon_5)
[ 2061.183037] id 31 (gp10b_506) min 164 max 164 refs 1 (previous client : gp10b_506)
[ 2061.183053] id 35 (15700000.vi_0) min 29 max 30 refs 4 (previous client : 15700000.vi_0)
[ 2061.183064] id 36 (15700000.vi_1) min 7 max 9 refs 4 (previous client : 15700000.vi_0)
[ 2061.183074] id 37 (15700000.vi_2) min 7 max 9 refs 4 (previous client : 15700000.vi_1)
[ 2061.183085] id 38 (150c0000.nvcsi_0) min 30 max 30 refs 1 (previous client : 15700000.vi_2)
[ 2061.183100] id 41 (gp10b_502) min 22 max 22 refs 1 (previous client : gp10b_502)
[ 2061.183112] id 43 (gp10b_501) min 20 max 20 refs 1 (previous client : gp10b_501)
[ 2061.183123] id 44 (gp10b_500) min 18 max 18 refs 1 (previous client : gp10b_500)
[ 2061.183133] id 45 (gp10b_499) min 18 max 18 refs 1 (previous client : gp10b_499)
[ 2061.183143] id 46 (gp10b_498) min 18 max 18 refs 1 (previous client : gp10b_498)
[ 2061.183154] id 47 (gp10b_497) min 18 max 18 refs 1 (previous client : gp10b_497)
[ 2061.183978] 
[ 2061.183984] ---- channels ----
[ 2061.184019] 
[ 2061.184019] channel 2 - 15820000.se
[ 2061.184019] 
[ 2061.184025] NvHost basic channel registers:
[ 2061.184034] CMDFIFO_STAT_0:  00002040
[ 2061.184041] CMDFIFO_RDATA_0: 003c0028
[ 2061.184052] CMDP_OFFSET_0:   00000000
[ 2061.184059] CMDP_CLASS_0:    00000000
[ 2061.184066] CHANNELSTAT_0:   00000000
[ 2061.184072] The CDMA sync queue is empty.
[ 2061.184075] 
[ 2061.184086] 
[ 2061.184086] channel 3 - 15830000.se
[ 2061.184086] 
[ 2061.184091] NvHost basic channel registers:
[ 2061.184098] CMDFIFO_STAT_0:  00002040
[ 2061.184105] CMDFIFO_RDATA_0: b0502411
[ 2061.184114] CMDP_OFFSET_0:   00000000
[ 2061.184120] CMDP_CLASS_0:    00000000
[ 2061.184126] CHANNELSTAT_0:   00000000
[ 2061.184131] The CDMA sync queue is empty.
[ 2061.184135] 
[ 2061.184145] 
[ 2061.184145] channel 4 - 15840000.se
[ 2061.184145] 
[ 2061.184150] NvHost basic channel registers:
[ 2061.184157] CMDFIFO_STAT_0:  00002040
[ 2061.184163] CMDFIFO_RDATA_0: 840a0100
[ 2061.184172] CMDP_OFFSET_0:   00000000
[ 2061.184177] CMDP_CLASS_0:    00000000
[ 2061.184183] CHANNELSTAT_0:   00000000
[ 2061.184189] The CDMA sync queue is empty.
[ 2061.184192] 
[ 2061.184202] 
[ 2061.184202] channel 1 - 15600000.isp
[ 2061.184202] 
[ 2061.184207] NvHost basic channel registers:
[ 2061.184214] CMDFIFO_STAT_0:  00004000
[ 2061.184220] CMDFIFO_RDATA_0: 004e0041
[ 2061.184229] CMDP_OFFSET_0:   00000050
[ 2061.184235] CMDP_CLASS_0:    00000001
[ 2061.184241] CHANNELSTAT_0:   00000000
[ 2061.184257] 
[ 2061.184257] ffffffc0edfbb400: JOB, syncpt_id=22, syncpt_val=16, first_get=00000ca0, timeout=10000, num_slots=19
[ 2061.184282]     GATHER at ffd28000+3c20, 12 words
[ 2061.184294] 20000001 
[ 2061.184299] 00006416 
[ 2061.184305] 20000001 
[ 2061.184310] 00006817 
[ 2061.184314] 20000001 
[ 2061.184319] 00006c18 
[ 2061.184323] 20000001 
[ 2061.184328] 00007c19 
[ 2061.184333] 20000001 
[ 2061.184337] 0000741d 
[ 2061.184342] 200c0001 
[ 2061.184346] 00000005 
[ 2061.184347] 
[ 2061.184358]     GATHER at ffd28000+3c50, 2 words
[ 2061.184368] 20000001 
[ 2061.184372] 0000001c 
[ 2061.184373] 
[ 2061.184377] 
[ 2061.184392] 
[ 2061.184392] channel 5 - 15700000.vi
[ 2061.184392] 
[ 2061.184397] NvHost basic channel registers:
[ 2061.184404] CMDFIFO_STAT_0:  0000001b
[ 2061.184411] CMDFIFO_RDATA_0: 304e0005
[ 2061.184420] CMDP_OFFSET_0:   00000050
[ 2061.184426] CMDP_CLASS_0:    00000001
[ 2061.184432] CHANNELSTAT_0:   00000000
[ 2061.184446] 
[ 2061.184446] ffffffc0e541c400: JOB, syncpt_id=37, syncpt_val=8, first_get=00000188, timeout=10000, num_slots=7
[ 2061.184461]     GATHER at ffd60000+0000, 6 words
[ 2061.184472] 304e0005 
[ 2061.184476] 00000058 
[ 2061.184481] 0000001c 
[ 2061.184486] 304e0005 
[ 2061.184490] 00000007 
[ 2061.184494] 00000024 
[ 2061.184495] 
[ 2061.184505]     GATHER at ffd60000+0018, 29 words
[ 2061.184514] 90000002 
[ 2061.184518] a0004008 
[ 2061.184523] 00007f1f 
[ 2061.184528] 00000aff 
[ 2061.184532] 9000000c 
[ 2061.184537] a000400c 
[ 2061.184541] 00000fc0 
[ 2061.184546] 00000be0 
[ 2061.184551] 00000001 
[ 2061.184555] 00000001 
[ 2061.184560] 00000000 
[ 2061.184564] 00000000 
[ 2061.184569] 00000000 
[ 2061.184573] 00000000 
[ 2061.184578] 00000000 
[ 2061.184582] 00000000 
[ 2061.184587] 00000fc0 
[ 2061.184592] 00000be0 
[ 2061.184596] 90000002 
[ 2061.184601] a0004018 
[ 2061.184606] 007f0000 
[ 2061.184610] 00000104 
[ 2061.184615] 90000002 
[ 2061.184619] a0004021 
[ 2061.184624] 00000000 
[ 2061.184628] 00000000 
[ 2061.184633] 90000001 
[ 2061.184638] a0004045 
[ 2061.184642] 0003d5ea 
[ 2061.184643] 
[ 2061.184652]     GATHER at ffd60000+008c, 26 words
[ 2061.184661] 90000001 
[ 2061.184666] a0000401 
[ 2061.184670] 00000000 
[ 2061.184675] 90000001 
[ 2061.184679] a000404d 
[ 2061.184684] 00000001 
[ 2061.184689] 90000001 
[ 2061.184693] a0004020 
[ 2061.184698] 00000000 
[ 2061.184702] 90000003 
[ 2061.184707] a0004042 
[ 2061.184712] ffcbf000 
[ 2061.184716] 0000001f 
[ 2061.184721] 00000001 
[ 2061.184725] 90000001 
[ 2061.184730] a0004048 
[ 2061.184734] 00000000 
[ 2061.184739] 90000001 
[ 2061.184743] a000400a 
[ 2061.184748] 00000000 
[ 2061.184753] 90000001 
[ 2061.184757] a0004001 
[ 2061.184762] 00000001 
[ 2061.184766] 90000001 
[ 2061.184771] a0004007 
[ 2061.184775] 00000003 
[ 2061.184776] 
[ 2061.184784]     GATHER at ffd60000+00f4, 2 words
[ 2061.184793] 20000001 
[ 2061.184797] 00000023 
[ 2061.184797] 
[ 2061.184802] 
[ 2061.184811] 
[ 2061.184811] ---- host general irq ----
[ 2061.184811] 
[ 2061.184818] sync_intc0mask = 0x00000001
[ 2061.184825] sync_intmask = 0x50000003
[ 2061.184829] 
[ 2061.184829] ---- host syncpt irq mask ----
[ 2061.184829] 
[ 2061.184834] 
[ 2061.184834] ---- host syncpt irq status ----
[ 2061.184834] 
[ 2061.184843] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 2061.184851] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 2061.184858] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 2061.184866] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 2061.184873] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 2061.184880] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 2061.184887] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 2061.184894] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 2061.184901] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 2061.184908] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 2061.184915] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 2061.184922] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 2061.184930] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 2061.184937] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 2061.184944] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 2061.184952] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 2061.184959] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 2061.184966] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[ 2061.209369] fence timeout on [ffffffc0ecf9ff00] after 1500ms
[ 2061.209372] name=[nvhost_sync:36], current value=7 waiting value=8
[ 2061.209374] ---- mlocks ----
[ 2061.209385] 8: locked by channel 1
[ 2061.209402] 
[ 2061.209403] ---- syncpts ----
[ 2061.209415] id 8 (vblank0) min 123664 max 0 refs 1 (previous client : )
[ 2061.209430] id 20 (15340000.vic_gst-launch-1.0_0) min 103 max 103 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[ 2061.209434] id 21 (15700000.vi_0) min 125 max 125 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[ 2061.209437] id 22 (15600000.isp_nvargus-daemon_0) min 14 max 18 refs 4 (previous client : 15600000.isp_nvargus-daemon_0)
[ 2061.209440] id 23 (15600000.isp_nvargus-daemon_1) min 7 max 9 refs 4 (previous client : 15600000.isp_nvargus-daemon_1)
[ 2061.209443] id 24 (15600000.isp_nvargus-daemon_2) min 7 max 9 refs 4 (previous client : 15600000.isp_nvargus-daemon_2)
[ 2061.209446] id 25 (15600000.isp_nvargus-daemon_3) min 7 max 9 refs 4 (previous client : 15600000.isp_nvargus-daemon_3)
[ 2061.209451] id 28 (15600000.isp_nvargus-daemon_4) min 88 max 99 refs 9 (previous client : 15600000.isp_nvargus-daemon_4)
[ 2061.209454] id 29 (15600000.isp_nvargus-daemon_5) min 7 max 9 refs 4 (previous client : 15600000.isp_nvargus-daemon_5)
[ 2061.209458] id 31 (gp10b_506) min 164 max 164 refs 1 (previous client : gp10b_506)
[ 2061.209464] id 35 (15700000.vi_0) min 29 max 30 refs 4 (previous client : 15700000.vi_0)
[ 2061.209467] id 36 (15700000.vi_1) min 7 max 9 refs 4 (previous client : 15700000.vi_0)
[ 2061.209469] id 37 (15700000.vi_2) min 7 max 9 refs 4 (previous client : 15700000.vi_1)
[ 2061.209472] id 38 (150c0000.nvcsi_0) min 30 max 30 refs 1 (previous client : 15700000.vi_2)
[ 2061.209477] id 41 (gp10b_502) min 22 max 22 refs 1 (previous client : gp10b_502)
[ 2061.209481] id 43 (gp10b_501) min 20 max 20 refs 1 (previous client : gp10b_501)
[ 2061.209484] id 44 (gp10b_500) min 18 max 18 refs 1 (previous client : gp10b_500)
[ 2061.209486] id 45 (gp10b_499) min 18 max 18 refs 1 (previous client : gp10b_499)
[ 2061.209489] id 46 (gp10b_498) min 18 max 18 refs 1 (previous client : gp10b_498)
[ 2061.209492] id 47 (gp10b_497) min 18 max 18 refs 1 (previous client : gp10b_497)
[ 2061.210063] 
[ 2061.210064] ---- channels ----
[ 2061.210071] 
[ 2061.210071] channel 2 - 15820000.se
[ 2061.210071] 
[ 2061.210072] NvHost basic channel registers:
[ 2061.210074] CMDFIFO_STAT_0:  00002040
[ 2061.210076] CMDFIFO_RDATA_0: 003c0028
[ 2061.210080] CMDP_OFFSET_0:   00000000
[ 2061.210082] CMDP_CLASS_0:    00000000
[ 2061.210084] CHANNELSTAT_0:   00000000
[ 2061.210085] The CDMA sync queue is empty.
[ 2061.210085] 
[ 2061.210087] 
[ 2061.210087] channel 3 - 15830000.se
[ 2061.210087] 
[ 2061.210088] NvHost basic channel registers:
[ 2061.210090] CMDFIFO_STAT_0:  00002040
[ 2061.210092] CMDFIFO_RDATA_0: b0502411
[ 2061.210095] CMDP_OFFSET_0:   00000000
[ 2061.210097] CMDP_CLASS_0:    00000000
[ 2061.210099] CHANNELSTAT_0:   00000000
[ 2061.210100] The CDMA sync queue is empty.
[ 2061.210101] 
[ 2061.210103] 
[ 2061.210103] channel 4 - 15840000.se
[ 2061.210103] 
[ 2061.210104] NvHost basic channel registers:
[ 2061.210106] CMDFIFO_STAT_0:  00002040
[ 2061.210108] CMDFIFO_RDATA_0: 840a0100
[ 2061.210111] CMDP_OFFSET_0:   00000000
[ 2061.210113] CMDP_CLASS_0:    00000000
[ 2061.210115] CHANNELSTAT_0:   00000000
[ 2061.210116] The CDMA sync queue is empty.
[ 2061.210116] 
[ 2061.210118] 
[ 2061.210118] channel 1 - 15600000.isp
[ 2061.210118] 
[ 2061.210119] NvHost basic channel registers:
[ 2061.210121] CMDFIFO_STAT_0:  00004000
[ 2061.210123] CMDFIFO_RDATA_0: 004e0041
[ 2061.210126] CMDP_OFFSET_0:   00000050
[ 2061.210128] CMDP_CLASS_0:    00000001
[ 2061.210130] CHANNELSTAT_0:   00000000
[ 2061.210133] 
[ 2061.210133] ffffffc0edfbb400: JOB, syncpt_id=22, syncpt_val=16, first_get=00000ca0, timeout=10000, num_slots=19
[ 2061.210135]     GATHER at ffd28000+3c20, 12 words
[ 2061.210137] 20000001 
[ 2061.210139] 00006416 
[ 2061.210140] 20000001 
[ 2061.210141] 00006817 
[ 2061.210142] 20000001 
[ 2061.210143] 00006c18 
[ 2061.210144] 20000001 
[ 2061.210144] 00007c19 
[ 2061.210145] 20000001 
[ 2061.210146] 0000741d 
[ 2061.210147] 200c0001 
[ 2061.210148] 00000005 
[ 2061.210148] 
[ 2061.210150]     GATHER at ffd28000+3c50, 2 words
[ 2061.210152] 20000001 
[ 2061.210153] 0000001c 
[ 2061.210153] 
[ 2061.210154] 
[ 2061.210156] 
[ 2061.210156] channel 5 - 15700000.vi
[ 2061.210156] 
[ 2061.210157] NvHost basic channel registers:
[ 2061.210159] CMDFIFO_STAT_0:  0000001b
[ 2061.210161] CMDFIFO_RDATA_0: 304e0005
[ 2061.210164] CMDP_OFFSET_0:   00000050
[ 2061.210166] CMDP_CLASS_0:    00000001
[ 2061.210168] CHANNELSTAT_0:   00000000
[ 2061.210170] 
[ 2061.210170] ffffffc0e541c400: JOB, syncpt_id=37, syncpt_val=8, first_get=00000188, timeout=10000, num_slots=7
[ 2061.210172]     GATHER at ffd60000+0000, 6 words
[ 2061.210174] 304e0005 
[ 2061.210175] 00000058 
[ 2061.210176] 0000001c 
[ 2061.210177] 304e0005 
[ 2061.210177] 00000007 
[ 2061.210178] 00000024 
[ 2061.210178] 
[ 2061.210180]     GATHER at ffd60000+0018, 29 words
[ 2061.210182] 90000002 
[ 2061.210183] a0004008 
[ 2061.210184] 00007f1f 
[ 2061.210185] 00000aff 
[ 2061.210185] 9000000c 
[ 2061.210186] a000400c 
[ 2061.210187] 00000fc0 
[ 2061.210188] 00000be0 
[ 2061.210189] 00000001 
[ 2061.210190] 00000001 
[ 2061.210191] 00000000 
[ 2061.210192] 00000000 
[ 2061.210193] 00000000 
[ 2061.210194] 00000000 
[ 2061.210194] 00000000 
[ 2061.210195] 00000000 
[ 2061.210196] 00000fc0 
[ 2061.210197] 00000be0 
[ 2061.210198] 90000002 
[ 2061.210199] a0004018 
[ 2061.210200] 007f0000 
[ 2061.210201] 00000104 
[ 2061.210202] 90000002 
[ 2061.210203] a0004021 
[ 2061.210204] 00000000 
[ 2061.210205] 00000000 
[ 2061.210205] 90000001 
[ 2061.210206] a0004045 
[ 2061.210207] 0003d5ea 
[ 2061.210207] 
[ 2061.210209]     GATHER at ffd60000+008c, 26 words
[ 2061.210210] 90000001 
[ 2061.210211] a0000401 
[ 2061.210212] 00000000 
[ 2061.210213] 90000001 
[ 2061.210214] a000404d 
[ 2061.210215] 00000001 
[ 2061.210216] 90000001 
[ 2061.210217] a0004020 
[ 2061.210218] 00000000 
[ 2061.210219] 90000003 
[ 2061.210220] a0004042 
[ 2061.210221] ffcbf000 
[ 2061.210221] 0000001f 
[ 2061.210222] 00000001 
[ 2061.210223] 90000001 
[ 2061.210224] a0004048 
[ 2061.210225] 00000000 
[ 2061.210226] 90000001 
[ 2061.210227] a000400a 
[ 2061.210228] 00000000 
[ 2061.210229] 90000001 
[ 2061.210230] a0004001 
[ 2061.210230] 00000001 
[ 2061.210231] 90000001 
[ 2061.210232] a0004007 
[ 2061.210233] 00000003 
[ 2061.210233] 
[ 2061.210235]     GATHER at ffd60000+00f4, 2 words
[ 2061.210236] 20000001 
[ 2061.210237] 00000023 
[ 2061.210237] 
[ 2061.210238] 
[ 2061.210239] 
[ 2061.210239] ---- host general irq ----
[ 2061.210239] 
[ 2061.210241] sync_intc0mask = 0x00000001
[ 2061.210244] sync_intmask = 0x50000003
[ 2061.210244] 
[ 2061.210244] ---- host syncpt irq mask ----
[ 2061.210244] 
[ 2061.210245] 
[ 2061.210245] ---- host syncpt irq status ----
[ 2061.210245] 
[ 2061.210248] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 2061.210250] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 2061.210252] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 2061.210255] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 2061.210257] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 2061.210259] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 2061.210261] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 2061.210263] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 2061.210265] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 2061.210268] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 2061.210270] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 2061.210272] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 2061.210274] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 2061.210276] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 2061.210279] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 2061.210281] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 2061.210283] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 2061.210285] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[ 2062.644490] name=[nvhost_sync:22], current value=14 waiting value=15
[ 2062.644493] ---- mlocks ----
[ 2062.644505] 8: locked by channel 1
[ 2062.644522] 

Dumping the low-level trace messages after the failed gstreamer:

jetson-tx2-nx-nobi-revision-a:/data/jan# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 1207/1207   #P:6
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/0:3-9827  [000] ....  2033.739409: rtos_queue_peek_from_isr_failed: tstamp:63863347025 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2033.899438: rtos_queue_peek_from_isr_failed: tstamp:63868347019 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2034.059430: rtos_queue_peek_from_isr_failed: tstamp:63873347013 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2034.219429: rtos_queue_peek_from_isr_failed: tstamp:63878347007 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2034.379393: rtos_queue_peek_from_isr_failed: tstamp:63883346999 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2034.539440: rtos_queue_peek_from_isr_failed: tstamp:63888346992 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2034.699432: rtos_queue_peek_from_isr_failed: tstamp:63893346984 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2034.859430: rtos_queue_peek_from_isr_failed: tstamp:63898346978 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2035.019442: rtos_queue_peek_from_isr_failed: tstamp:63903346972 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2035.179433: rtos_queue_peek_from_isr_failed: tstamp:63908346966 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2035.339431: rtos_queue_peek_from_isr_failed: tstamp:63913346958 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2035.499434: rtos_queue_peek_from_isr_failed: tstamp:63918346952 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2035.659436: rtos_queue_peek_from_isr_failed: tstamp:63923346947 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2035.819470: rtos_queue_peek_from_isr_failed: tstamp:63928346937 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2035.979411: rtos_queue_peek_from_isr_failed: tstamp:63933346933 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2036.139430: rtos_queue_peek_from_isr_failed: tstamp:63938346925 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2036.299432: rtos_queue_peek_from_isr_failed: tstamp:63943346918 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2036.459432: rtos_queue_peek_from_isr_failed: tstamp:63948346911 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2036.619420: rtos_queue_peek_from_isr_failed: tstamp:63953346903 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2036.779434: rtos_queue_peek_from_isr_failed: tstamp:63958346896 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2036.939430: rtos_queue_peek_from_isr_failed: tstamp:63963346890 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2037.099432: rtos_queue_peek_from_isr_failed: tstamp:63968346883 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2037.259419: rtos_queue_peek_from_isr_failed: tstamp:63973346877 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2037.419430: rtos_queue_peek_from_isr_failed: tstamp:63978346868 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2037.579460: rtos_queue_peek_from_isr_failed: tstamp:63983346862 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2037.742752: rtos_queue_peek_from_isr_failed: tstamp:63988346862 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2037.906128: rtos_queue_peek_from_isr_failed: tstamp:63993346850 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2038.066098: rtos_queue_peek_from_isr_failed: tstamp:63998346842 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2038.226110: rtos_queue_peek_from_isr_failed: tstamp:64003346835 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2038.386073: rtos_queue_peek_from_isr_failed: tstamp:64008346827 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2038.546095: rtos_queue_peek_from_isr_failed: tstamp:64013346832 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2038.706091: rtos_queue_peek_from_isr_failed: tstamp:64018346815 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2038.866094: rtos_queue_peek_from_isr_failed: tstamp:64023346809 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2039.026083: rtos_queue_peek_from_isr_failed: tstamp:64028346801 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2039.186119: rtos_queue_peek_from_isr_failed: tstamp:64033346795 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2039.346092: rtos_queue_peek_from_isr_failed: tstamp:64038346786 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2039.506092: rtos_queue_peek_from_isr_failed: tstamp:64043346780 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2039.666124: rtos_queue_peek_from_isr_failed: tstamp:64048346774 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2039.826101: rtos_queue_peek_from_isr_failed: tstamp:64053346768 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2039.986092: rtos_queue_peek_from_isr_failed: tstamp:64058346761 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2040.146091: rtos_queue_peek_from_isr_failed: tstamp:64063346754 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2040.306091: rtos_queue_peek_from_isr_failed: tstamp:64068346745 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2040.466089: rtos_queue_peek_from_isr_failed: tstamp:64073346739 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2040.626079: rtos_queue_peek_from_isr_failed: tstamp:64078346733 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2040.786087: rtos_queue_peek_from_isr_failed: tstamp:64083346727 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2040.946041: rtos_queue_peek_from_isr_failed: tstamp:64088346720 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2041.106099: rtos_queue_peek_from_isr_failed: tstamp:64093346713 queue:0x0b4b4500
           <...>-37753 [000] ....  2041.250148: tegra_channel_open: vi-output, imx477 2-001a
           <...>-37753 [000] ....  2041.250861: tegra_channel_close: vi-output, imx477 2-001a
     kworker/0:3-9827  [000] ....  2041.266060: rtos_queue_peek_from_isr_failed: tstamp:64098346705 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2041.426040: rtos_queue_peek_from_isr_failed: tstamp:64103346699 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2041.586095: rtos_queue_peek_from_isr_failed: tstamp:64108346692 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2041.746103: rtos_queue_peek_from_isr_failed: tstamp:64113346689 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2041.906100: rtos_queue_peek_from_isr_failed: tstamp:64118346679 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2042.066082: rtos_queue_peek_from_isr_failed: tstamp:64123346670 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2042.226093: rtos_queue_peek_from_isr_failed: tstamp:64128346664 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2042.386091: rtos_queue_peek_from_isr_failed: tstamp:64133346658 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2042.546092: rtos_queue_peek_from_isr_failed: tstamp:64138346652 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2042.706091: rtos_queue_peek_from_isr_failed: tstamp:64143346644 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2042.866128: rtos_queue_peek_from_isr_failed: tstamp:64148346638 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2043.026118: rtos_queue_peek_from_isr_failed: tstamp:64153346629 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2043.186092: rtos_queue_peek_from_isr_failed: tstamp:64158346623 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2043.346089: rtos_queue_peek_from_isr_failed: tstamp:64163346617 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2043.506096: rtos_queue_peek_from_isr_failed: tstamp:64168346611 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2043.666091: rtos_queue_peek_from_isr_failed: tstamp:64173346605 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2043.826105: rtos_queue_peek_from_isr_failed: tstamp:64178346599 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2043.986091: rtos_queue_peek_from_isr_failed: tstamp:64183346588 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2044.146043: rtos_queue_peek_from_isr_failed: tstamp:64188346582 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2044.306105: rtos_queue_peek_from_isr_failed: tstamp:64193346576 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2044.466070: rtos_queue_peek_from_isr_failed: tstamp:64198346570 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2044.626093: rtos_queue_peek_from_isr_failed: tstamp:64203346563 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2044.786090: rtos_queue_peek_from_isr_failed: tstamp:64208346557 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2044.946092: rtos_queue_peek_from_isr_failed: tstamp:64213346548 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2045.106090: rtos_queue_peek_from_isr_failed: tstamp:64218346542 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2045.266135: rtos_queue_peek_from_isr_failed: tstamp:64223346535 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2045.426093: rtos_queue_peek_from_isr_failed: tstamp:64228346528 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2045.586091: rtos_queue_peek_from_isr_failed: tstamp:64233346521 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2045.746082: rtos_queue_peek_from_isr_failed: tstamp:64238346513 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2045.906085: rtos_queue_peek_from_isr_failed: tstamp:64243346506 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2046.066079: rtos_queue_peek_from_isr_failed: tstamp:64248346500 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2046.226091: rtos_queue_peek_from_isr_failed: tstamp:64253346495 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2046.386090: rtos_queue_peek_from_isr_failed: tstamp:64258346487 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2046.546094: rtos_queue_peek_from_isr_failed: tstamp:64263346481 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2046.706096: rtos_queue_peek_from_isr_failed: tstamp:64268346472 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2046.866092: rtos_queue_peek_from_isr_failed: tstamp:64273346466 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2047.026062: rtos_queue_peek_from_isr_failed: tstamp:64278346460 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2047.186091: rtos_queue_peek_from_isr_failed: tstamp:64283346454 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2047.346094: rtos_queue_peek_from_isr_failed: tstamp:64288346447 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2047.509431: rtos_queue_peek_from_isr_failed: tstamp:64293346440 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2047.669431: rtos_queue_peek_from_isr_failed: tstamp:64298346431 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2047.829438: rtos_queue_peek_from_isr_failed: tstamp:64303346426 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2047.989424: rtos_queue_peek_from_isr_failed: tstamp:64308346419 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2048.149415: rtos_queue_peek_from_isr_failed: tstamp:64313346413 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2048.309429: rtos_queue_peek_from_isr_failed: tstamp:64318346405 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2048.469426: rtos_queue_peek_from_isr_failed: tstamp:64323346400 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2048.629424: rtos_queue_peek_from_isr_failed: tstamp:64328346391 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2048.789422: rtos_queue_peek_from_isr_failed: tstamp:64333346396 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2048.949439: rtos_queue_peek_from_isr_failed: tstamp:64338346382 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2049.109433: rtos_queue_peek_from_isr_failed: tstamp:64343346371 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2049.269421: rtos_queue_peek_from_isr_failed: tstamp:64348346362 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2049.429425: rtos_queue_peek_from_isr_failed: tstamp:64353346356 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2049.589463: rtos_queue_peek_from_isr_failed: tstamp:64358346350 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2049.749427: rtos_queue_peek_from_isr_failed: tstamp:64363346344 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2049.909401: rtos_queue_peek_from_isr_failed: tstamp:64368346338 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2050.069418: rtos_queue_peek_from_isr_failed: tstamp:64373346330 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2050.229426: rtos_queue_peek_from_isr_failed: tstamp:64378346324 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2050.389426: rtos_queue_peek_from_isr_failed: tstamp:64383346315 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2050.549415: rtos_queue_peek_from_isr_failed: tstamp:64388346309 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2050.709427: rtos_queue_peek_from_isr_failed: tstamp:64393346303 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2050.869405: rtos_queue_peek_from_isr_failed: tstamp:64398346297 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2051.029408: rtos_queue_peek_from_isr_failed: tstamp:64403346289 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2051.189416: rtos_queue_peek_from_isr_failed: tstamp:64408346283 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2051.349431: rtos_queue_peek_from_isr_failed: tstamp:64413346277 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2051.509426: rtos_queue_peek_from_isr_failed: tstamp:64418346268 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2051.669427: rtos_queue_peek_from_isr_failed: tstamp:64423346261 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2051.829438: rtos_queue_peek_from_isr_failed: tstamp:64428346258 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2051.989426: rtos_queue_peek_from_isr_failed: tstamp:64433346249 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2052.149422: rtos_queue_peek_from_isr_failed: tstamp:64438346240 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2052.309430: rtos_queue_peek_from_isr_failed: tstamp:64443346234 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2052.469451: rtos_queue_peek_from_isr_failed: tstamp:64448346227 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2052.629426: rtos_queue_peek_from_isr_failed: tstamp:64453346221 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2052.789427: rtos_queue_peek_from_isr_failed: tstamp:64458346215 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2052.949448: rtos_queue_peek_from_isr_failed: tstamp:64463346205 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2053.109422: rtos_queue_peek_from_isr_failed: tstamp:64468346199 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2053.269422: rtos_queue_peek_from_isr_failed: tstamp:64473346193 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2053.429425: rtos_queue_peek_from_isr_failed: tstamp:64478346187 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2053.589401: rtos_queue_peek_from_isr_failed: tstamp:64483346181 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2053.749458: rtos_queue_peek_from_isr_failed: tstamp:64488346174 queue:0x0b4b4500
     kworker/0:2-32252 [000] ....  2053.909388: rtos_queue_peek_from_isr_failed: tstamp:64493346168 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2054.069421: rtos_queue_peek_from_isr_failed: tstamp:64498346158 queue:0x0b4b4500
  nvargus-daemon-30916 [004] ....  2054.219898: tegra_channel_close: vi-output, imx477 2-001a
  nvargus-daemon-30916 [004] ....  2054.219909: tegra_channel_set_stream: enable : 0x0
  nvargus-daemon-30916 [004] ....  2054.219910: tegra_channel_set_stream: imx477 2-001a : 0x0
  nvargus-daemon-30916 [004] ....  2054.220121: tegra_channel_set_stream: 150c0000.nvcsi--1 : 0x0
  nvargus-daemon-30916 [004] ....  2054.220126: csi_s_stream: enable : 0x0
  nvargus-daemon-30916 [004] ....  2054.221222: tegra_channel_set_power: imx477 2-001a : 0x0
  nvargus-daemon-30916 [004] ....  2054.221230: camera_common_s_power: status : 0x0
  nvargus-daemon-30916 [004] ....  2054.221313: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x0
  nvargus-daemon-30916 [004] ....  2054.221317: csi_s_power: enable : 0x0
     kworker/0:3-9827  [000] ....  2054.239417: rtos_queue_peek_from_isr_failed: tstamp:64503346152 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2054.239423: rtos_queue_send_from_isr_failed: tstamp:64503508383 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2054.239425: rtos_queue_send_from_isr_failed: tstamp:64503508492 queue:0x0b4aad68
     kworker/0:3-9827  [000] ....  2054.239426: rtos_queue_send_from_isr_failed: tstamp:64503508601 queue:0x0b4ac998
     kworker/0:3-9827  [000] ....  2054.239428: rtos_queue_send_from_isr_failed: tstamp:64503508708 queue:0x0b4af2d8
     kworker/0:3-9827  [000] ....  2054.239429: rtos_queue_send_from_isr_failed: tstamp:64503508812 queue:0x0b4b0098
     kworker/0:3-9827  [000] ....  2054.239430: rtos_queue_send_from_isr_failed: tstamp:64503508917 queue:0x0b4b0e58
     kworker/0:3-9827  [000] ....  2054.239432: rtos_queue_send_from_isr_failed: tstamp:64503509021 queue:0x0b4b1c18
     kworker/0:3-9827  [000] ....  2054.239434: rtos_queue_send_failed: tstamp:64503509665 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2054.239435: rtos_queue_send_from_isr_failed: tstamp:64503512526 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2054.239437: rtos_queue_send_from_isr_failed: tstamp:64503512649 queue:0x0b4aad68
     kworker/0:3-9827  [000] ....  2054.239438: rtos_queue_send_from_isr_failed: tstamp:64503512754 queue:0x0b4ac998
     kworker/0:3-9827  [000] ....  2054.239439: rtos_queue_send_from_isr_failed: tstamp:64503512862 queue:0x0b4af2d8
     kworker/0:3-9827  [000] ....  2054.239441: rtos_queue_send_from_isr_failed: tstamp:64503512966 queue:0x0b4b0098
     kworker/0:3-9827  [000] ....  2054.239442: rtos_queue_send_from_isr_failed: tstamp:64503513072 queue:0x0b4b0e58
     kworker/0:3-9827  [000] ....  2054.239443: rtos_queue_send_from_isr_failed: tstamp:64503513175 queue:0x0b4b1c18
     kworker/0:3-9827  [000] ....  2054.239445: rtos_queue_send_failed: tstamp:64503513598 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2054.399379: rtos_queue_peek_from_isr_failed: tstamp:64508346150 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2054.559380: rtos_queue_peek_from_isr_failed: tstamp:64513346140 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2054.719371: rtos_queue_peek_from_isr_failed: tstamp:64518346132 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2054.879400: rtos_queue_peek_from_isr_failed: tstamp:64523346126 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2055.039431: rtos_queue_peek_from_isr_failed: tstamp:64528346118 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2055.199425: rtos_queue_peek_from_isr_failed: tstamp:64533346111 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2055.359398: rtos_queue_peek_from_isr_failed: tstamp:64538346105 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2055.519424: rtos_queue_peek_from_isr_failed: tstamp:64543346099 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2055.679608: rtos_queue_peek_from_isr_failed: tstamp:64548346092 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2055.839440: rtos_queue_peek_from_isr_failed: tstamp:64553346086 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2055.999413: rtos_queue_peek_from_isr_failed: tstamp:64558346083 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2056.159426: rtos_queue_peek_from_isr_failed: tstamp:64563346071 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2056.319425: rtos_queue_peek_from_isr_failed: tstamp:64568346064 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2056.479423: rtos_queue_peek_from_isr_failed: tstamp:64573346066 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2056.639421: rtos_queue_peek_from_isr_failed: tstamp:64578346051 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2056.799454: rtos_queue_peek_from_isr_failed: tstamp:64583346042 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2056.959416: rtos_queue_peek_from_isr_failed: tstamp:64588346036 queue:0x0b4b4500
           <...>-37992 [000] ....  2057.030758: tegra_channel_open: vi-output, imx477 2-001a
           <...>-37992 [000] ....  2057.031577: tegra_channel_set_power: imx477 2-001a : 0x1
           <...>-37992 [000] ....  2057.031588: camera_common_s_power: status : 0x1
     kworker/0:3-9827  [000] .n..  2057.126305: rtos_queue_peek_from_isr_failed: tstamp:64593346030 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2057.286104: rtos_queue_peek_from_isr_failed: tstamp:64598346024 queue:0x0b4b4500
           <...>-37992 [000] ....  2057.331805: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x1
           <...>-37992 [000] ....  2057.331812: csi_s_power: enable : 0x1
           <...>-37992 [000] ....  2057.336384: tegra_channel_close: vi-output, imx477 2-001a
           <...>-37992 [000] ....  2057.336395: tegra_channel_set_power: imx477 2-001a : 0x0
           <...>-37992 [000] ....  2057.336418: camera_common_s_power: status : 0x0
           <...>-37992 [000] ....  2057.336551: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x0
           <...>-37992 [000] ....  2057.336556: csi_s_power: enable : 0x0
     kworker/0:3-9827  [000] ....  2057.446074: rtos_queue_peek_from_isr_failed: tstamp:64603346016 queue:0x0b4b4500
           <...>-38047 [000] ....  2057.508257: tegra_channel_open: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2057.508265: tegra_channel_set_power: imx477 2-001a : 0x1
           <...>-38047 [000] ....  2057.508271: camera_common_s_power: status : 0x1
     kworker/0:3-9827  [000] ....  2057.612803: rtos_queue_peek_from_isr_failed: tstamp:64608346012 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2057.772738: rtos_queue_peek_from_isr_failed: tstamp:64613346001 queue:0x0b4b4500
           <...>-38047 [003] ....  2057.808496: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x1
           <...>-38047 [003] ....  2057.808504: csi_s_power: enable : 0x1
           <...>-38047 [003] ....  2057.808910: tegra_channel_close: vi-output, imx477 2-001a
           <...>-38047 [003] ....  2057.808920: tegra_channel_set_power: imx477 2-001a : 0x0
           <...>-38047 [003] ....  2057.808938: camera_common_s_power: status : 0x0
           <...>-38047 [003] ....  2057.809073: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x0
           <...>-38047 [003] ....  2057.809078: csi_s_power: enable : 0x0
           <...>-38047 [000] ....  2057.820029: tegra_channel_open: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2057.820046: tegra_channel_set_power: imx477 2-001a : 0x1
           <...>-38047 [000] ....  2057.820067: camera_common_s_power: status : 0x1
     kworker/0:3-9827  [000] ....  2057.932738: rtos_queue_peek_from_isr_failed: tstamp:64618345995 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2058.092757: rtos_queue_peek_from_isr_failed: tstamp:64623345989 queue:0x0b4b4500
           <...>-38047 [000] ....  2058.120328: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x1
           <...>-38047 [000] ....  2058.120335: csi_s_power: enable : 0x1
           <...>-38047 [000] ....  2058.120680: tegra_channel_close: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2058.120688: tegra_channel_set_power: imx477 2-001a : 0x0
           <...>-38047 [000] ....  2058.120705: camera_common_s_power: status : 0x0
           <...>-38047 [000] ....  2058.120830: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x0
           <...>-38047 [000] ....  2058.120835: csi_s_power: enable : 0x0
           <...>-38047 [000] ....  2058.150524: tegra_channel_open: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2058.150532: tegra_channel_set_power: imx477 2-001a : 0x1
           <...>-38047 [000] ....  2058.150538: camera_common_s_power: status : 0x1
     kworker/0:3-9827  [000] ....  2058.252771: rtos_queue_peek_from_isr_failed: tstamp:64628345983 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2058.412749: rtos_queue_peek_from_isr_failed: tstamp:64633345975 queue:0x0b4b4500
           <...>-38047 [000] ....  2058.450754: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x1
           <...>-38047 [000] ....  2058.450761: csi_s_power: enable : 0x1
           <...>-38047 [000] ....  2058.450831: tegra_channel_close: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2058.450840: tegra_channel_set_power: imx477 2-001a : 0x0
           <...>-38047 [000] ....  2058.450856: camera_common_s_power: status : 0x0
           <...>-38047 [000] ....  2058.450980: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x0
           <...>-38047 [000] ....  2058.450984: csi_s_power: enable : 0x0
           <...>-38047 [000] ....  2058.451324: tegra_channel_open: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2058.451339: tegra_channel_set_power: imx477 2-001a : 0x1
           <...>-38047 [000] ....  2058.451357: camera_common_s_power: status : 0x1
     kworker/0:3-9827  [000] ....  2058.572765: rtos_queue_peek_from_isr_failed: tstamp:64638345969 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2058.732759: rtos_queue_peek_from_isr_failed: tstamp:64643345960 queue:0x0b4b4500
           <...>-38047 [000] ....  2058.751594: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x1
           <...>-38047 [000] ....  2058.751600: csi_s_power: enable : 0x1
           <...>-38047 [000] ....  2058.751930: tegra_channel_close: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2058.751938: tegra_channel_set_power: imx477 2-001a : 0x0
           <...>-38047 [000] ....  2058.751956: camera_common_s_power: status : 0x0
           <...>-38047 [003] ....  2058.752892: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x0
           <...>-38047 [003] ....  2058.752899: csi_s_power: enable : 0x0
           <...>-38047 [000] ....  2058.754653: tegra_channel_open: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2058.754669: tegra_channel_set_power: imx477 2-001a : 0x1
           <...>-38047 [000] ....  2058.754687: camera_common_s_power: status : 0x1
     kworker/0:3-9827  [000] ....  2058.892745: rtos_queue_peek_from_isr_failed: tstamp:64648345954 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2059.052755: rtos_queue_peek_from_isr_failed: tstamp:64653345950 queue:0x0b4b4500
           <...>-38047 [000] ....  2059.054929: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x1
           <...>-38047 [000] ....  2059.054935: csi_s_power: enable : 0x1
           <...>-38047 [000] ....  2059.055220: tegra_channel_close: vi-output, imx477 2-001a
           <...>-38047 [000] ....  2059.055228: tegra_channel_set_power: imx477 2-001a : 0x0
           <...>-38047 [000] ....  2059.055242: camera_common_s_power: status : 0x0
           <...>-38047 [000] ....  2059.055365: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x0
           <...>-38047 [000] ....  2059.055370: csi_s_power: enable : 0x0
     kworker/0:3-9827  [000] ....  2059.159396: rtos_queue_send_from_isr_failed: tstamp:64656286366 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2059.159399: rtos_queue_send_from_isr_failed: tstamp:64656286476 queue:0x0b4aad68
     kworker/0:3-9827  [000] ....  2059.159399: rtos_queue_send_from_isr_failed: tstamp:64656286585 queue:0x0b4ac998
     kworker/0:3-9827  [000] ....  2059.159400: rtos_queue_send_from_isr_failed: tstamp:64656286692 queue:0x0b4af2d8
     kworker/0:3-9827  [000] ....  2059.159402: rtos_queue_send_from_isr_failed: tstamp:64656286796 queue:0x0b4b0098
     kworker/0:3-9827  [000] ....  2059.159403: rtos_queue_send_from_isr_failed: tstamp:64656286901 queue:0x0b4b0e58
     kworker/0:3-9827  [000] ....  2059.159403: rtos_queue_send_from_isr_failed: tstamp:64656287006 queue:0x0b4b1c18
     kworker/0:3-9827  [000] ....  2059.159405: rtos_queue_send_failed: tstamp:64656287460 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2059.212718: rtos_queue_peek_from_isr_failed: tstamp:64658345942 queue:0x0b4b4500
           <...>-38066 [004] ....  2059.269501: tegra_channel_open: vi-output, imx477 2-001a
           <...>-38066 [004] ....  2059.269510: tegra_channel_set_power: imx477 2-001a : 0x1
           <...>-38066 [004] ....  2059.269518: camera_common_s_power: status : 0x1
     kworker/0:3-9827  [000] ....  2059.319372: rtos_queue_send_from_isr_failed: tstamp:64661284347 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2059.319375: rtos_queue_send_from_isr_failed: tstamp:64661284463 queue:0x0b4aad68
     kworker/0:3-9827  [000] ....  2059.319376: rtos_queue_send_from_isr_failed: tstamp:64661284570 queue:0x0b4ac998
     kworker/0:3-9827  [000] ....  2059.319377: rtos_queue_send_from_isr_failed: tstamp:64661284678 queue:0x0b4af2d8
     kworker/0:3-9827  [000] ....  2059.319377: rtos_queue_send_from_isr_failed: tstamp:64661284783 queue:0x0b4b0098
     kworker/0:3-9827  [000] ....  2059.319378: rtos_queue_send_from_isr_failed: tstamp:64661284888 queue:0x0b4b0e58
     kworker/0:3-9827  [000] ....  2059.319379: rtos_queue_send_from_isr_failed: tstamp:64661284994 queue:0x0b4b1c18
     kworker/0:3-9827  [000] ....  2059.319380: rtos_queue_send_failed: tstamp:64661285451 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2059.372713: rtos_queue_peek_from_isr_failed: tstamp:64663345935 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2059.532764: rtos_queue_peek_from_isr_failed: tstamp:64668345928 queue:0x0b4b4500
           <...>-38066 [003] ....  2059.569717: tegra_channel_set_power: 150c0000.nvcsi--1 : 0x1
           <...>-38066 [003] ....  2059.569725: csi_s_power: enable : 0x1
           <...>-38066 [004] ....  2059.575313: tegra_channel_set_stream: enable : 0x1
           <...>-38066 [000] ....  2059.576815: tegra_channel_set_stream: 150c0000.nvcsi--1 : 0x1
           <...>-38066 [000] ....  2059.576823: csi_s_stream: enable : 0x1
           <...>-38066 [000] ....  2059.576826: tegra_channel_set_stream: imx477 2-001a : 0x1
     kworker/0:3-9827  [000] ....  2059.586112: rtos_queue_send_from_isr_failed: tstamp:64670786717 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2059.586118: rtos_queue_send_from_isr_failed: tstamp:64670786826 queue:0x0b4aad68
     kworker/0:3-9827  [000] ....  2059.586121: rtos_queue_send_from_isr_failed: tstamp:64670786933 queue:0x0b4ac998
     kworker/0:3-9827  [000] ....  2059.586123: rtos_queue_send_from_isr_failed: tstamp:64670787040 queue:0x0b4af2d8
     kworker/0:3-9827  [000] ....  2059.586125: rtos_queue_send_from_isr_failed: tstamp:64670787144 queue:0x0b4b0098
     kworker/0:3-9827  [000] ....  2059.586127: rtos_queue_send_from_isr_failed: tstamp:64670787248 queue:0x0b4b0e58
     kworker/0:3-9827  [000] ....  2059.586130: rtos_queue_send_from_isr_failed: tstamp:64670787353 queue:0x0b4b1c18
     kworker/0:3-9827  [000] ....  2059.586133: rtos_queue_send_failed: tstamp:64670787796 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2059.692747: rtos_queue_peek_from_isr_failed: tstamp:64673345922 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2059.746112: rtos_queue_send_from_isr_failed: tstamp:64674935783 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2059.746118: rtos_queue_send_from_isr_failed: tstamp:64674935890 queue:0x0b4aad68
     kworker/0:3-9827  [000] ....  2059.746121: rtos_queue_send_from_isr_failed: tstamp:64674935998 queue:0x0b4ac998
     kworker/0:3-9827  [000] ....  2059.746123: rtos_queue_send_from_isr_failed: tstamp:64674936108 queue:0x0b4af2d8
     kworker/0:3-9827  [000] ....  2059.746125: rtos_queue_send_from_isr_failed: tstamp:64674936214 queue:0x0b4b0098
     kworker/0:3-9827  [000] ....  2059.746127: rtos_queue_send_from_isr_failed: tstamp:64674936319 queue:0x0b4b0e58
     kworker/0:3-9827  [000] ....  2059.746130: rtos_queue_send_from_isr_failed: tstamp:64674936423 queue:0x0b4b1c18
     kworker/0:3-9827  [000] ....  2059.746133: rtos_queue_send_failed: tstamp:64674937318 queue:0x0b4a7258
     kworker/0:3-9827  [000] ....  2059.852771: rtos_queue_peek_from_isr_failed: tstamp:64678345916 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2060.012765: rtos_queue_peek_from_isr_failed: tstamp:64683345912 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2060.172762: rtos_queue_peek_from_isr_failed: tstamp:64688345901 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2060.332788: rtos_queue_peek_from_isr_failed: tstamp:64693345893 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2060.492776: rtos_queue_peek_from_isr_failed: tstamp:64698345885 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2060.652768: rtos_queue_peek_from_isr_failed: tstamp:64703345879 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2060.812733: rtos_queue_peek_from_isr_failed: tstamp:64708345872 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2060.972765: rtos_queue_peek_from_isr_failed: tstamp:64713345866 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2061.132756: rtos_queue_peek_from_isr_failed: tstamp:64718345859 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2061.299365: rtos_queue_peek_from_isr_failed: tstamp:64723345853 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2061.459360: rtos_queue_peek_from_isr_failed: tstamp:64728345844 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2061.619362: rtos_queue_peek_from_isr_failed: tstamp:64733345847 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2061.779362: rtos_queue_peek_from_isr_failed: tstamp:64738345832 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2061.939362: rtos_queue_peek_from_isr_failed: tstamp:64743345826 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2062.099359: rtos_queue_peek_from_isr_failed: tstamp:64748345824 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2062.259362: rtos_queue_peek_from_isr_failed: tstamp:64753345812 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2062.422694: rtos_queue_peek_from_isr_failed: tstamp:64758345803 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2062.582695: rtos_queue_peek_from_isr_failed: tstamp:64763345797 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2062.742719: rtos_queue_peek_from_isr_failed: tstamp:64768345791 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2062.856025: rtos_queue_peek_from_isr_failed: tstamp:64773345788 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2063.016022: rtos_queue_peek_from_isr_failed: tstamp:64778345778 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2063.229354: rtos_queue_peek_from_isr_failed: tstamp:64783345772 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2063.386033: rtos_queue_peek_from_isr_failed: tstamp:64788345763 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2063.546037: rtos_queue_peek_from_isr_failed: tstamp:64793345756 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2063.706043: rtos_queue_peek_from_isr_failed: tstamp:64798345750 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2063.816094: rtos_queue_peek_from_isr_failed: tstamp:64803345744 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2063.976126: rtos_queue_peek_from_isr_failed: tstamp:64808345737 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2064.136095: rtos_queue_peek_from_isr_failed: tstamp:64813345732 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2064.296098: rtos_queue_peek_from_isr_failed: tstamp:64818345722 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2064.456112: rtos_queue_peek_from_isr_failed: tstamp:64823345716 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2064.616104: rtos_queue_peek_from_isr_failed: tstamp:64828345709 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2064.776090: rtos_queue_peek_from_isr_failed: tstamp:64833345702 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2064.936077: rtos_queue_peek_from_isr_failed: tstamp:64838345693 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2065.096097: rtos_queue_peek_from_isr_failed: tstamp:64843345687 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2065.256070: rtos_queue_peek_from_isr_failed: tstamp:64848345680 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2065.416091: rtos_queue_peek_from_isr_failed: tstamp:64853345675 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2065.576092: rtos_queue_peek_from_isr_failed: tstamp:64858345669 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2065.736102: rtos_queue_peek_from_isr_failed: tstamp:64863345661 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2065.899366: rtos_queue_peek_from_isr_failed: tstamp:64868345655 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2066.069361: rtos_queue_peek_from_isr_failed: tstamp:64873345650 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2066.229368: rtos_queue_peek_from_isr_failed: tstamp:64878345640 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2066.389370: rtos_queue_peek_from_isr_failed: tstamp:64883345634 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2066.549369: rtos_queue_peek_from_isr_failed: tstamp:64888345628 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2066.709368: rtos_queue_peek_from_isr_failed: tstamp:64893345620 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2066.869374: rtos_queue_peek_from_isr_failed: tstamp:64898345614 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2067.029430: rtos_queue_peek_from_isr_failed: tstamp:64903345605 queue:0x0b4b4500
     kworker/0:3-9827  [000] ....  2067.189428: rtos_queue_peek_from_isr_failed: tstamp:64908345599 queue:0x0b4b4500
[.... continues with rtos_queue_peek_from_isr_failed ...]

The trace log tell didn’t receive any validate data from sensor.
Maybe need to probe the MIPI signal.
And what CSI is HW connected? What’s the port-index form the device tree dump file?

Hi ShaneCCC

The trace log tell didn’t receive any validate data from sensor.
Maybe need to probe the MIPI signal.

Is the ISP picky on certain data it gets from VI/Sensor?
Because when we use v4l2-ctl, we get images.

What could be wrong with the MIPI signal?

We put a spectrum analyser over the lanes, and saw data being passed (1,5GHz) so the sensor is sending data.

And what CSI is HW connected?

CSI0, 2-lanes.

What’s the port-index form the device tree dump file?

I also made a post before the previous one, discussing the device-tree dump.
Is there anything extra that I can grep/show from the dump?

Here is also the output from media-ctl:

jetson-tx2-nx-nobi-revision-a:/data/jan# media-ctl -p -d /dev/media0
Media controller API version 0.1.0

Media device information
------------------------
driver          tegra-vi4
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  0.0.0

Device topology
- entity 1: 150c0000.nvcsi--1 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		<- "imx477 2-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx477 2-001a":0 [ENABLED]

- entity 4: imx477 2-001a (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev1
	pad0: Source
		[fmt:SRGGB10_1X10/4032x3040 field:none colorspace:srgb]
		-> "150c0000.nvcsi--1":0 [ENABLED]

- entity 6: vi-output, imx477 2-001a (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
	pad0: Sink
		<- "150c0000.nvcsi--1":1 [ENABLED]

OK, please attach the extracted_proc.dts device tree dump file.

OK, here you go:
extracted_proc.dts.gz (40.0 KB)

The device tree looks correct. I don’t have idea why for this case currently.

That’s unfortunate because we can’t find it either.

We’ve put all psu’s external and changed the sequencing too, but no change.

Gstreamer/nvargus-daemon gives following error msg:

Streaming camera 0 at 4032x3040 to file /dev/null ... 
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
Execution ended after 0:00:02.077720047
Setting pipeline to NULL ...
[ 3280.450401] imx477 2-001a: imx477_set_frame_rate: Jan's dummy set framerate function
[ 3280.551435] imx477 2-001a: imx477_set_frame_rate: Jan's dummy set framerate function
[ 3282.066194] fence timeout on [ffffffc0caa7ea80] after 1500ms
[ 3282.066207] fence timeout on [ffffffc0caa7e900] after 1500ms
[ 3282.066225] name=[nvhost_sync:25], current value=0 waiting value=1
[ 3282.066237] ---- mlocks ----
[ 3282.066258] 8: locked by channel 1
[ 3282.066282] 
[ 3282.066288] ---- syncpts ----
[ 3282.066318] id 8 (vblank0) min 196943 max 0 refs 1 (previous client : )
[ 3282.066350] id 21 (15700000.vi_0) min 7 max 7 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[ 3282.066362] id 22 (15600000.isp_nvargus-daemon_0) min 0 max 4 refs 4 (previous client : )
[ 3282.066373] id 23 (15600000.isp_nvargus-daemon_1) min 0 max 2 refs 4 (previous client : )
[ 3282.066383] id 24 (15600000.isp_nvargus-daemon_2) min 0 max 2 refs 4 (previous client : )
[ 3282.066393] id 25 (15600000.isp_nvargus-daemon_3) min 0 max 2 refs 4 (previous client : )
[ 3282.066407] id 28 (15600000.isp_nvargus-daemon_4) min 15 max 26 refs 9 (previous client : )
[ 3282.066417] id 29 (15600000.isp_nvargus-daemon_5) min 0 max 2 refs 4 (previous client : )

So what the following mean?:

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found

Are there no other hw/sw debugging we can do to rectify this issue?

Are there any minimum requirements regarding the line-length of the MIPI signals on a custom PCB that the Nvidia TX2-NX module needs?

Please refer to the MIPI DSI and CSI Design Guidelines table in TX2 NX DG.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.