FPGA CSI TX to TX1 Linux freeze

Hello,
I am passing MIPI csi stream from an image sensor through a FPGA to the TX1:
Imager->FPGA->TX1
Camera I2C Interface is directly connected to TX1.
Running nvgstcapture/argus_camera application freezes linux system.
Here is part of the log-file after running the applications:

[ 60.487560] fence timeout on [ffffffc024bfb800] after 1500ms
[ 60.493224] name=[nvhost_sync:7], current value=0 waiting value=1
[ 60.499374] fence timeout on [ffffffc024bfb300] after 1500ms
[ 60.499598] fence timeout on [ffffffc0225cac00] after 1500ms
[ 60.499602] name=[nvhost_sync:29], current value=1 waiting value=2
[ 60.499606] ---- mlocks ----
[ 60.499616]
[ 60.499618] ---- syncpts ----
[ 60.499623] id 1 (disp1_a) min 159 max 159 refs 1 (previous client : )
[ 60.499626] id 2 (disp1_b) min 1 max 1 refs 1 (previous client : )
[ 60.499629] id 3 (disp1_c) min 1 max 1 refs 1 (previous client : )
[ 60.499632] id 4 (57000000.gpu_507) min 6708 max 6708 refs 1 (previous client : )
[ 60.499635] id 5 (57000000.gpu_506) min 26 max 26 refs 1 (previous client : )
[ 60.499638] id 6 (57000000.gpu_505) min 838 max 838 refs 1 (previous client : 57000000.gpu_505)
[ 60.499641] id 7 (54600000.isp_0) min 0 max 3 refs 4 (previous client : )
[ 60.499644] id 8 (54600000.isp_1) min 0 max 3 refs 4 (previous client : )
[ 60.499646] id 9 (54600000.isp_2) min 7 max 16 refs 11 (previous client : )
[ 60.499650] id 11 (54600000.isp_3) min 1 max 3 refs 4 (previous client : )
[ 60.499655] id 16 (57000000.gpu_504) min 1222 max 1222 refs 1 (previous client : )
[ 60.499658] id 17 (57000000.gpu_503) min 2 max 2 refs 1 (previous client : )
[ 60.499661] id 18 (57000000.gpu_502) min 2 max 2 refs 1 (previous client : )
[ 60.499663] id 19 (57000000.gpu_501) min 2 max 2 refs 1 (previous client : )
[ 60.499666] id 20 (57000000.gpu_500) min 2 max 2 refs 1 (previous client : )
[ 60.499669] id 21 (57000000.gpu_499) min 8 max 8 refs 1 (previous client : )
[ 60.499672] id 22 (54080000.vi_0) min 0 max 3 refs 4 (previous client : 54080000.vi_0)
[ 60.499675] id 24 (54080000.vi_2) min 2 max 7 refs 7 (previous client : 54080000.vi_2)
[ 60.499679] id 27 (vblank1) min 3421 max 0 refs 1 (previous client : )
[ 60.499683] id 29 (54080000.vi_4) min 1 max 3 refs 4 (previous client : 54080000.vi_4)
[ 60.499688] id 34 (57000000.gpu_498) min 1060 max 1060 refs 1 (previous client : )
[ 60.499691] id 35 (57000000.gpu_497) min 2 max 2 refs 1 (previous client : )
[ 60.499694] id 36 (57000000.gpu_496) min 2 max 2 refs 1 (previous client : )
[ 60.499696] id 37 (57000000.gpu_495) min 2 max 2 refs 1 (previous client : )
[ 60.499699] id 38 (57000000.gpu_494) min 2 max 2 refs 1 (previous client : )
[ 60.499702] id 39 (57000000.gpu_493) min 4 max 4 refs 1 (previous client : )
[ 60.499705] id 40 (57000000.gpu_492) min 6 max 6 refs 1 (previous client : )
[ 60.499708] id 41 (57000000.gpu_491) min 8 max 8 refs 1 (previous client : )
[ 60.499710] id 42 (57000000.gpu_490) min 6 max 6 refs 1 (previous client : )
[ 60.499713] id 43 (57000000.gpu_489) min 6 max 6 refs 1 (previous client : )
[ 60.499716] id 44 (57000000.gpu_488) min 6 max 6 refs 1 (previous client : )
[ 60.499718] id 45 (57000000.gpu_487) min 6 max 6 refs 1 (previous client : )
[ 60.499721] id 46 (57000000.gpu_486) min 6 max 6 refs 1 (previous client : )
[ 60.499816]
[ 60.499816] ---- channels ----
[ 60.499822]
[ 60.499822] channel 0 - 54600000.isp
[ 60.499822]
[ 60.499830] 0-54600000.isp (19): active class 01, offset 0050, val 00000007
[ 60.499832] DMAPUT 000001b0, DMAGET 000000e8, DMACTL 00000000
[ 60.499833] CBREAD 00000007, CBSTAT 00010050
[ 60.499837]
[ 60.499837] ffffffc023b2ca00: JOB, syncpt_id=7, syncpt_val=1, first_get=00000090, timeout=10000, num_slots=3, num_handles=1
[ 60.499845] GATHER at 80270000+4514, 8 words
[ 60.499853] 20000001 00000407 20000001 00000508 20000001 0000060b 200c0001 00000005
[ 60.499856] GATHER at 80270000+4534, 2 words
[ 60.499858] 20000001 00000009
[ 60.499859]
[ 60.499862]
[ 60.499862] channel 1 - 54680000.isp
[ 60.499862]
[ 60.499867] 1-54680000.isp (1): inactive
[ 60.499867]
[ 60.499869]
[ 60.499869] channel 2 - 54080000.vi
[ 60.499869]
[ 60.499876] 2-54080000.vi (12): active class 01, offset 0050, val 00000016
[ 60.499878] DMAPUT 00000100, DMAGET 00000100, DMACTL 00000000
[ 60.499879] CBREAD 00000016, CBSTAT 00010050
[ 60.499882]
[ 60.499882] ffffffc0d4eeb000: JOB, syncpt_id=22, syncpt_val=1, first_get=00000018, timeout=10000, num_slots=7, num_handles=1
[ 60.499886] GATHER at 802e0000+0000, 39 words
[ 60.499920] 10620001 0000001f 10630002 00000000 00000000 103a0001 10100010 103c0001 10101010 10420001 00000001 10430006 00000000 001c984c 00000000 00000000 00000000 00000000 10590001 00000003 120e0006 007f0017 080301f0 00000000 00140000 0000f005 00000000 12420001 0000
[ 60.499922] GATHER at 802e0000+009c, 3 words
[ 60.499925] 304e0005 00000007 00000009
[ 60.499927] GATHER at 802e0000+00a8, 21 words
[ 60.499946] 10430006 00200002 001c984c 00000000 087e0f18 000012de 0000002b 124b0003 00010000 00000000 00000007 12580003 00010000 00000000 00000007 20000001 00001c16 20000001 0000051d 20410001 00000001
[ 60.499948] GATHER at 802e0000+00fc, 2 words
[ 60.499950] 20000001 00000018
[ 60.499951]
[ 60.499952]
[ 60.499952] ---- host general irq ----
[ 60.499952]
[ 60.499954] sync_hintmask_ext = 0xc0000000
[ 60.499956] sync_hintmask = 0x80000000
[ 60.499958] sync_intc0mask = 0x00000001
[ 60.499959] sync_intmask = 0x00000011
[ 60.499960]
[ 60.499960] ---- host syncpt irq mask ----
[ 60.499960]
[ 60.499962] syncpt_thresh_int_mask(0) = 0x00454001
[ 60.499964] syncpt_thresh_int_mask(1) = 0x04011000
[ 60.499966] syncpt_thresh_int_mask(2) = 0x00000000
[ 60.499968] syncpt_thresh_int_mask(3) = 0x00000000
[ 60.499970] syncpt_thresh_int_mask(4) = 0x00000000
[ 60.499971] syncpt_thresh_int_mask(5) = 0x00000000
[ 60.499973] syncpt_thresh_int_mask(6) = 0x00000000
[ 60.499975] syncpt_thresh_int_mask(7) = 0x00000000
[ 60.499976] syncpt_thresh_int_mask(8) = 0x00000000
[ 60.499978] syncpt_thresh_int_mask(9) = 0x00000000
[ 60.499980] syncpt_thresh_int_mask(10) = 0x00000000
[ 60.499982] syncpt_thresh_int_mask(11) = 0x00000000
[ 60.499983]
[ 60.499983] ---- host syncpt irq status ----
[ 60.499983]
[ 60.499985] syncpt_thresh_cpu0_int_status(0) = 0x00000800
[ 60.499986] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 60.499988] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 60.499990] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 60.499992] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 60.499993] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 60.499994]
[ 60.499994] ---- host syncpt thresh ----
[ 60.499994]
[ 60.499997] syncpt_int_thresh_thresh_0(0) = 1
[ 60.500002] syncpt_int_thresh_thresh_0(7) = 1
[ 60.500005] syncpt_int_thresh_thresh_0(8) = 1
[ 60.500007] syncpt_int_thresh_thresh_0(9) = 8
[ 60.500010] syncpt_int_thresh_thresh_0(11) = 1
[ 60.500018] syncpt_int_thresh_thresh_0(22) = 1
[ 60.500021] syncpt_int_thresh_thresh_0(24) = 3
[ 60.500025] syncpt_int_thresh_thresh_0(29) = 2
[ 60.505656] ---- mlocks ----
[ 60.505667]
[ 60.505669] ---- syncpts ----
[ 60.505675] id 1 (disp1_a) min 159 max 159 refs 1 (previous client : )
[ 60.505678] id 2 (disp1_b) min 1 max 1 refs 1 (previous client : )
[ 60.505681] id 3 (disp1_c) min 1 max 1 refs 1 (previous client : )
[ 60.505684] id 4 (57000000.gpu_507) min 6708 max 6708 refs 1 (previous client : )
[ 60.505687] id 5 (57000000.gpu_506) min 26 max 26 refs 1 (previous client : )
[ 60.505690] id 6 (57000000.gpu_505) min 838 max 838 refs 1 (previous client : 57000000.gpu_505)
[ 60.505693] id 7 (54600000.isp_0) min 0 max 4 refs 5 (previous client : )
[ 60.505695] id 8 (54600000.isp_1) min 0 max 4 refs 5 (previous client : )
[ 60.505698] id 9 (54600000.isp_2) min 7 max 20 refs 15 (previous client : )
[ 60.505701] id 11 (54600000.isp_3) min 1 max 4 refs 5 (previous client : )
[ 60.505707] id 16 (57000000.gpu_504) min 1222 max 1222 refs 1 (previous client : )
[ 60.505710] id 17 (57000000.gpu_503) min 2 max 2 refs 1 (previous client : )
[ 60.505712] id 18 (57000000.gpu_502) min 2 max 2 refs 1 (previous client : )
[ 60.505715] id 19 (57000000.gpu_501) min 2 max 2 refs 1 (previous client : )
[ 60.505718] id 20 (57000000.gpu_500) min 2 max 2 refs 1 (previous client : )
[ 60.505720] id 21 (57000000.gpu_499) min 8 max 8 refs 1 (previous client : )
[ 60.505723] id 22 (54080000.vi_0) min 0 max 4 refs 5 (previous client : 54080000.vi_0)
[ 60.505727] id 24 (54080000.vi_2) min 2 max 9 refs 9 (previous client : 54080000.vi_2)
[ 60.505732] id 27 (vblank1) min 3421 max 0 refs 1 (previous client : )
[ 60.505735] id 29 (54080000.vi_4) min 1 max 4 refs 5 (previous client : 54080000.vi_4)
[ 60.505741] id 34 (57000000.gpu_498) min 1060 max 1060 refs 1 (previous client : )
[ 60.505743] id 35 (57000000.gpu_497) min 2 max 2 refs 1 (previous client : )
[ 60.505748] id 36 (57000000.gpu_496) min 2 max 2 refs 1 (previous client : )
[ 60.505751] id 37 (57000000.gpu_495) min 2 max 2 refs 1 (previous client : )
[ 60.505753] id 38 (57000000.gpu_494) min 2 max 2 refs 1 (previous client : )
[ 60.505756] id 39 (57000000.gpu_493) min 4 max 4 refs 1 (previous client : )
[ 60.505759] id 40 (57000000.gpu_492) min 6 max 6 refs 1 (previous client : )
[ 60.505761] id 41 (57000000.gpu_491) min 8 max 8 refs 1 (previous client : )
[ 60.505764] id 42 (57000000.gpu_490) min 6 max 6 refs 1 (previous client : )
[ 60.505767] id 43 (57000000.gpu_489) min 6 max 6 refs 1 (previous client : )
[ 60.505770] id 44 (57000000.gpu_488) min 6 max 6 refs 1 (previous client : )
[ 60.505772] id 45 (57000000.gpu_487) min 6 max 6 refs 1 (previous client : )
[ 60.505775] id 46 (57000000.gpu_486) min 6 max 6 refs 1 (previous client : )
[ 60.505869]
[ 60.505870] ---- channels ----
[ 60.505875]
[ 60.505875] channel 0 - 54600000.isp
[ 60.505875]
[ 60.505883] 0-54600000.isp (26): active class 01, offset 0050, val 00000007
[ 60.505885] DMAPUT 00000220, DMAGET 000000e8, DMACTL 00000000
[ 60.505886] CBREAD 00000007, CBSTAT 00010050
[ 60.505891]
[ 60.505891] ffffffc023b2ca00: JOB, syncpt_id=7, syncpt_val=1, first_get=00000090, timeout=10000, num_slots=3, num_handles=1
[ 60.505894] GATHER at 80270000+4514, 8 words
[ 60.505903] 20000001 00000407 20000001 00000508 20000001 0000060b 200c0001 00000005
[ 60.505905] GATHER at 80270000+4534, 2 words
[ 60.505908] 20000001 00000009
[ 60.505909]
[ 60.505911]
[ 60.505911] channel 1 - 54680000.isp
[ 60.505911]
[ 60.505917] 1-54680000.isp (1): inactive
[ 60.505917]
[ 60.505949]
[ 60.505949] channel 2 - 54080000.vi
[ 60.505949]
[ 60.505957] 2-54080000.vi (16): active class 01, offset 0050, val 00000016
[ 60.505958] DMAPUT 00000148, DMAGET 00000110, DMACTL 00000000
[ 60.505960] CBREAD 00000016, CBSTAT 00010050
[ 60.505963]
[ 60.505963] ffffffc0d4eeb000: JOB, syncpt_id=22, syncpt_val=1, first_get=00000018, timeout=10000, num_slots=7, num_handles=1
[ 60.505965] GATHER at 802e0000+0000, 39 words
[ 60.506003] 10620001 0000001f 10630002 00000000 00000000 103a0001 10100010 103c0001 10101010 10420001 00000001 10430006 00000000 001c984c 00000000 00000000 00000000 00000000 10590001 00000003 120e0006 007f0017 080301f0 00000000 00140000 0000f005 00000000 12420001 0000
[ 60.506005] GATHER at 802e0000+009c, 3 words
[ 60.506008] 304e0005 00000007 00000009
[ 60.506010] GATHER at 802e0000+00a8, 21 words
[ 60.506030] 10430006 00200002 001c984c 00000000 087e0f18 000012de 0000002b 124b0003 00010000 00000000 00000007 12580003 00010000 00000000 00000007 20000001 00001c16 20000001 0000051d 20410001 00000001
[ 60.506032] GATHER at 802e0000+00fc, 2 words
[ 60.506035] 20000001 00000018
[ 60.506036]
[ 60.506037]
[ 60.506037] ---- host general irq ----
[ 60.506037]
[ 60.506039] sync_hintmask_ext = 0xc0000000
[ 60.506041] sync_hintmask = 0x80000000
[ 60.506043] sync_intc0mask = 0x00000001
[ 60.506045] sync_intmask = 0x00000011
[ 60.506046]
[ 60.506046] ---- host syncpt irq mask ----
[ 60.506046]
[ 60.506048] syncpt_thresh_int_mask(0) = 0x00454001
[ 60.506050] syncpt_thresh_int_mask(1) = 0x04011000
[ 60.506052] syncpt_thresh_int_mask(2) = 0x00000000
[ 60.506053] syncpt_thresh_int_mask(3) = 0x00000000
[ 60.506055] syncpt_thresh_int_mask(4) = 0x00000000
[ 60.506057] syncpt_thresh_int_mask(5) = 0x00000000
[ 60.506059] syncpt_thresh_int_mask(6) = 0x00000000
[ 60.506061] syncpt_thresh_int_mask(7) = 0x00000000
[ 60.506063] syncpt_thresh_int_mask(8) = 0x00000000
[ 60.506064] syncpt_thresh_int_mask(9) = 0x00000000
[ 60.506066] syncpt_thresh_int_mask(10) = 0x00000000
[ 60.506068] syncpt_thresh_int_mask(11) = 0x00000000
[ 60.506069]
[ 60.506069] ---- host syncpt irq status ----
[ 60.506069]
[ 60.506071] syncpt_thresh_cpu0_int_status(0) = 0x00000800
[ 60.506073] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 60.506075] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 60.506076] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 60.506078] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 60.506080] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 60.506082]
[ 60.506082] ---- host syncpt thresh ----
[ 60.506082]
[ 60.506084] syncpt_int_thresh_thresh_0(0) = 1
[ 60.506090] syncpt_int_thresh_thresh_0(7) = 1
[ 60.506092] syncpt_int_thresh_thresh_0(8) = 1
[ 60.506095] syncpt_int_thresh_thresh_0(9) = 8
[ 60.506098] syncpt_int_thresh_thresh_0(11) = 1
[ 60.506105] syncpt_int_thresh_thresh_0(22) = 1
[ 60.506108] syncpt_int_thresh_thresh_0(24) = 3
[ 60.506113] syncpt_int_thresh_thresh_0(29) = 2
[ 61.709630] tegra-i2c 7000d000.i2c: pio xfer timed out addr: 0x3c
[ 61.711473] name=[nvhost_sync:8], current value=0 waiting value=1
[ 61.711476] ---- mlocks ----

The log is not helpful. How to get more info?

But running v4l2-ctl --stream-mmap=3 frames are being continously recognized @ 60fps.

What’s the pixel format? nvcanerasrc and argus only support bayer sensor now.

Hello @ShaneCCC
It is bayer format. I forgot to mention in my setup the Imager can also be connected directly to the TX1. In this case it is working as expected. The same settings are applied for the FPGA in between but fails.
Obviously the FPGA is causing this. But I need more info about the root cause from the TX1.

Hey,
Seems like I’m currently facing a similar problem; I mentioned it here:

https://devtalk.nvidia.com/default/topic/1031214/jetson-tx2/debugging-csi-on-tx1/post/5248215/#5248215

Also trying to get Information of the CSI Status register for debuging purposes. Using devmem2 wasn’t very helpfull…

@beti
Check the log if any csi status before the syncpt timeout.