Jetson nano auto reboot with fence timeout

I just use my nano normally but it reboots usually when i run ros programs.
I’m not alone in this problem, the persons who do the same race say they also meet this case but we both don’t know what’s wrong with it.
we use darknet, csi camera and move_base(ros), the cpu, gpu and ram is going well, but it reboot in the process.
The only information I know is the logs before it reboots.
Wrong start at [ 564.292170]

[   33.906163] vdd-fan: disabling
[   33.906168] vdd-usb-vbus: disabling
[   33.906171] vdd-usb-vbus2: disabling
[   33.906182] vddio-sdmmc3-ap: disabling
[   33.906287] avdd-io-edp-1v05: disabling
[   33.906290] vdd-usb-hub-en: disabling
[  127.872455] usb 1-2.1: new full-speed USB device number 9 using tegra-xusb
[  127.894338] usb 1-2.1: New USB device found, idVendor=10c4, idProduct=ea60
[  127.894345] usb 1-2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  127.894350] usb 1-2.1: Product: CP2102 USB to UART Bridge Controller
[  127.894354] usb 1-2.1: Manufacturer: Silicon Labs
[  127.894357] usb 1-2.1: SerialNumber: 0001
[  127.949681] usbcore: registered new interface driver cp210x
[  127.949730] usbserial: USB Serial support registered for cp210x
[  127.949830] cp210x 1-2.1:1.0: cp210x converter detected
[  127.956085] usb 1-2.1: cp210x converter now attached to ttyUSB1
[  564.292170] fence timeout on [ffffffc0f518d0c0] after 1500ms
[  564.292235] name=[nvhost_sync:18], current value=8760 waiting value=8761
[  564.292278] ---- mlocks ----

[  564.292353] ---- syncpts ----
[  564.292436] id 8 (gm20b_507) min 13168 max 13168 refs 1 (previous client : )
[  564.292482] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[  564.292530] id 11 (gm20b_505) min 624 max 624 refs 1 (previous client : gm20b_505)
[  564.292576] id 12 (54340000.vic_0) min 26250 max 26250 refs 1 (previous client : 54340000.vic_0)
[  564.292639] id 18 (54680000.isp_0) min 8760 max 8763 refs 4 (previous client : 54680000.isp_0)
[  564.292683] id 19 (54680000.isp_1) min 8760 max 8763 refs 4 (previous client : 54680000.isp_1)
[  564.292726] id 20 (54680000.isp_2) min 29461 max 29469 refs 10 (previous client : 54680000.isp_2)
[  564.292768] id 21 (54680000.isp_3) min 8761 max 8763 refs 4 (previous client : 54680000.isp_3)
[  564.292811] id 22 (gm20b_504) min 704 max 704 refs 1 (previous client : )
[  564.292853] id 23 (gm20b_503) min 52 max 52 refs 1 (previous client : )
[  564.292893] id 24 (gm20b_502) min 2 max 2 refs 1 (previous client : )
[  564.292932] id 25 (gm20b_501) min 2 max 2 refs 1 (previous client : )
[  564.292979] id 28 (gm20b_500) min 6 max 6 refs 1 (previous client : )
[  564.293026] id 30 (54080000.vi_1) min 8760 max 8763 refs 4 (previous client : 54080000.vi_1)
[  564.293069] id 31 (54080000.vi_2) min 17526 max 17531 refs 7 (previous client : 54080000.vi_2)
[  564.293120] id 34 (54080000.vi_4) min 8761 max 8763 refs 4 (previous client : 54080000.vi_4)
[  564.293163] id 35 (54340000.vic_0) min 52690 max 52690 refs 1 (previous client : 54340000.vic_0)
[  564.293205] id 36 (gm20b_496) min 6 max 6 refs 1 (previous client : 54340000.vic_0)
[  564.293249] id 37 (54340000.vic_0) min 24 max 24 refs 1 (previous client : gm20b_493)
[  564.293292] id 38 (gm20b_494) min 30 max 30 refs 1 (previous client : gm20b_492)
[  564.293334] id 39 (gm20b_499) min 32 max 32 refs 1 (previous client : gm20b_491)
[  564.293376] id 40 (gm20b_495) min 30 max 30 refs 1 (previous client : gm20b_490)
[  564.293418] id 41 (gm20b_497) min 30 max 30 refs 1 (previous client : gm20b_489)
[  564.293461] id 42 (gm20b_489) min 30 max 30 refs 1 (previous client : gm20b_498)
[  564.293512] id 45 (gm20b_498) min 1355613 max 1355613 refs 1 (previous client : 54340000.vic_0)
[  564.293557] id 46 (gm20b_491) min 45710 max 45710 refs 1 (previous client : gm20b_494)
[  564.293599] id 47 (gm20b_492) min 6 max 6 refs 1 (previous client : gm20b_499)
[  564.293640] id 48 (gm20b_490) min 6 max 6 refs 1 (previous client : gm20b_495)
[  564.293683] id 49 (gm20b_493) min 46612 max 46612 refs 1 (previous client : gm20b_497)

[  564.294139] ---- channels ----
[  564.294204] 
               channel 0 - 54680000.isp

[  564.294248] 0-54680000.isp (17): 
[  564.294294] active class 01, offset 0050, val 00000012
[  564.294335] DMAPUT 00000ee0, DMAGET 00000e70, DMACTL 00000000
[  564.294367] CBREAD 00000012, CBSTAT 00010050
[  564.294428] 
               ffffffc0d29dac00: JOB, syncpt_id=18, syncpt_val=8761, first_get=00000dd8, timeout=10000, num_slots=3, num_handles=1
[  564.294509]     GATHER at fd130000+3330, 8 words
[  564.294537] 20000001 
[  564.294566] 00000412 
[  564.294592] 20000001 
[  564.294620] 00000513 
[  564.294645] 20000001 
[  564.294672] 00000615 
[  564.294696] 200c0001 
[  564.294721] 00000005 

[  564.294792]     GATHER at fd130000+3350, 2 words
packet_write_wait: Connection to 192.168.123.191 port 22: Broken pipe

Just now it reboots again.

[   33.909905] avdd-io-edp-1v05: disabling
[   33.909940] vdd-usb-hub-en: disabling
[  485.705654] fence timeout on [ffffffc097b25cc0] after 1500ms
[  485.705721] name=[nvhost_sync:19], current value=6666 waiting value=6667
[  485.705765] ---- mlocks ----

[  485.705841] ---- syncpts ----
[  485.705929] id 8 (gm20b_507) min 11752 max 11752 refs 1 (previous client : )
[  485.705976] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[  485.706028] id 11 (gm20b_505) min 522 max 522 refs 1 (previous client : gm20b_505)
[  485.706073] id 12 (54340000.vic_0) min 19992 max 19992 refs 1 (previous client : )
[  485.706116] id 13 (54340000.vic_0) min 19992 max 19992 refs 1 (previous client : )
[  485.706173] id 18 (54680000.isp_0) min 6666 max 6669 refs 4 (previous client : )
[  485.706216] id 19 (54680000.isp_1) min 6666 max 6669 refs 4 (previous client : )
[  485.706261] id 20 (54680000.isp_2) min 22752 max 22761 refs 11 (previous client : )
[  485.706303] id 21 (54680000.isp_3) min 6667 max 6669 refs 4 (previous client : )
[  485.706346] id 22 (gm20b_504) min 704 max 704 refs 1 (previous client : )
[  485.706387] id 23 (gm20b_503) min 58 max 58 refs 1 (previous client : )
[  485.706429] id 24 (gm20b_502) min 2 max 2 refs 1 (previous client : )
[  485.706470] id 25 (gm20b_501) min 2 max 2 refs 1 (previous client : )
[  485.706517] id 28 (gm20b_500) min 6 max 6 refs 1 (previous client : )
[  485.706567] id 30 (54080000.vi_1) min 6666 max 6669 refs 4 (previous client : 54080000.vi_1)
[  485.706613] id 31 (54080000.vi_2) min 13334 max 13339 refs 7 (previous client : 54080000.vi_2)
[  485.706667] id 34 (54080000.vi_4) min 6667 max 6669 refs 4 (previous client : 54080000.vi_4)
[  485.706710] id 35 (54340000.vic_0) min 40130 max 40130 refs 1 (previous client : )
[  485.706755] id 37 (gm20b_499) min 6 max 6 refs 1 (previous client : )
[  485.706796] id 38 (gm20b_498) min 8 max 8 refs 1 (previous client : )
packet_write_wait: Connection to 192.168.123.191 port 22: Broken pipe

Here is journalctl logs, reboots at 14:19:54

8月 18 14:16:13 ai-desktop nm-applet[6655]: gtk_widget_destroy: assertion 'GTK_IS_WIDGET (widget)' failed
8月 18 14:16:13 ai-desktop nm-applet[6655]: gtk_widget_destroy: assertion 'GTK_IS_WIDGET (widget)' failed
8月 18 14:16:13 ai-desktop nm-applet[6655]: Can't set a parent on widget which has a parent
8月 18 14:17:01 ai-desktop sshd[14481]: Accepted password for ai from 192.168.123.221 port 44434 ssh2
8月 18 14:17:01 ai-desktop sshd[14481]: pam_unix(sshd:session): session opened for user ai by (uid=0)
8月 18 14:17:01 ai-desktop systemd-logind[4005]: New session 9 of user ai.
8月 18 14:17:01 ai-desktop systemd[1]: Started Session 9 of user ai.
8月 18 14:17:01 ai-desktop CRON[14611]: pam_unix(cron:session): session opened for user root by (uid=0)
8月 18 14:17:01 ai-desktop CRON[14625]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
8月 18 14:17:01 ai-desktop CRON[14611]: pam_unix(cron:session): session closed for user root
8月 18 14:17:15 ai-desktop sshd[14963]: Accepted password for ai from 192.168.123.221 port 44436 ssh2
8月 18 14:17:15 ai-desktop sshd[14963]: pam_unix(sshd:session): session opened for user ai by (uid=0)
8月 18 14:17:15 ai-desktop systemd-logind[4005]: New session 11 of user ai.
8月 18 14:17:15 ai-desktop systemd[1]: Started Session 11 of user ai.
-- Reboot --
8月 18 14:20:33 ai-desktop kernel: Booting Linux on physical CPU 0x0
8月 18 14:20:33 ai-desktop kernel: Linux version 4.9.201-tegra (buildbrain@mobile-u64-5486-d4000) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro
8月 18 14:20:33 ai-desktop kernel: Boot CPU: AArch64 Processor [411fd071]
8月 18 14:20:33 ai-desktop kernel: OF: fdt:memory scan node memory@80000000, reg size 32,
8月 18 14:20:33 ai-desktop kernel: OF: fdt: - 80000000 ,  7ee00000
8月 18 14:20:33 ai-desktop kernel: OF: fdt: - 100000000 ,  7f200000
8月 18 14:20:33 ai-desktop kernel: earlycon: uart8250 at MMIO32 0x0000000070006000 (options '')
8月 18 14:20:33 ai-desktop kernel: bootconsole [uart8250] enabled

hello himco,

it’s software using syncpt (sync points) to communicate with the sensor hardware to acquire sensor start-of-frame signaling, and programming the memory buffer as camera frames.

may I know which camera sensor you’re working with,
are you able to narrow down the issue by only preview the camera stream to repo the same failure?

hello JerryChang,

The camera sensor is IMX219-160. And for the race, I just remove the csi camera, and replace it with usb camera. Then the problem seems to be solved.

After the race, I’ll try to preview the frame.

Thanks a lot !

hello himco,

what did you mean exactly about “for the race”?

Hi JerryChang,
I am doing some work in the College student smart car competition, and I meet this problem but luckly I use USB camera and it works well.

hello himco,

USB camera go through standard v4l2 interface, but bayer sensors (such as, IMX219) taking CSI interface.
by default, Jetson Nano support Raspberry Pi v2, IMX219.