Hi there,
I have a problem where my Jetson TX2 boots but then some failures cause the board to reboot, only to crash again. I tried re-flashing with JetPack 3.2. It succeeded in flashing the image but fails to copy cuda or other installs because the board crashes. I instead flashed the image with the “flash.sh” command but this was no better. (I also tried the “flash.sh” command with JetPack 3.1 and similar issues arose.) I connected a UART console with directions from https://www.jetsonhacks.com/2017/03/24/serial-console-nvidia-jetson-tx2/ and have copied a typical output below.
Note “Timeout detected @ gr_gk20a_submit_fecs_method_op+0x104/0x274” is an error which is always present. Sometimes the other errors change but the system always freezes then restarts.
[ 110.276967] IPVS: Creating netns size=1424 id=1
[ 110.449054] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 110.465844] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 110.528675] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 110.539792]
[ 110.539792] Dongle Host Driver, version 1.201.82 (r)
[ 110.539792] Compiled in drivers/net/wireless/bcmdhd on Mar 1 2018 at 20:46:20
[ 110.571928] wl_android_wifi_on in
[ 110.580058] wifi_platform_set_power = 1
[ 110.864367] mmc1: queuing unknown CIS tuple 0x80 (5 bytes)
[ 110.953438] sdhci-tegra 3440000.sdhci: Tuning already done, restoring the best tap value : 58
[ 110.963721] F1 signature read @0x18000000=0x17214354
[ 110.979525] F1 signature OK, socitype:0x1 chip:0x4354 rev:0x1 pkg:0x2
[ 110.987972] DHD: dongle ram size is set to 786432(orig 786432) at 0x180000
[ 111.126706] dhdsdio_write_vars: Download, Upload and compare of NVRAM succeeded.
[ 111.177656] dhd_bus_init: enable 0x06, ready 0x06 (waited 0us)
[ 111.184167] Enabling wake69
[ 111.189118] wifi_platform_get_mac_addr
[ 111.195645] Firmware up: op_mode=0x0005, MAC=00:04:4b:8d:03:1c
[ 111.211358] dhd_preinit_ioctls pspretend_threshold for HostAPD failed -23
[ 111.225691] Firmware version = wl0: Dec 12 2017 15:09:35 version 7.35.221.34 (r679642) FWID 01-e35dbe99
[ 111.240371] dhd_interworking_enable: failed to set WNM info, ret=-23
[ 111.247189] tegra_sysfs_on
[ 111.509211] CFGP2P-ERROR) wl_cfgp2p_add_p2p_disc_if : P2P interface registered
[ 111.554890] WLC_E_IF: NO_IF set, event Ignored
[ 113.850812] gk20a 17000000.gp10b: Timeout detected @ gr_gk20a_submit_fecs_method_op+0x104/0x274
[ 113.860179] gk20a 17000000.gp10b: gr_gk20a_ctx_wait_ucode: timeout waiting on ucode response
[ 113.868737] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_os_r : 0
[ 113.876248] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_cpuctl_r : 0x40
[ 113.884432] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_idlestate_r : 0x1
[ 113.892717] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox0_r : 0x0
[ 113.900906] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox1_r : 0x0
[ 113.909073] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqstat_r : 0x0
[ 113.917172] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmode_r : 0x4
[ 113.925254] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmask_r : 0x8704
[ 113.933614] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqdest_r : 0x0
[ 113.942154] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_debug1_r : 0x40
[ 113.950406] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_debuginfo_r : 0x0
[ 113.958787] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(0) : 0x10
[ 113.967920] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(1) : 0x0
[ 113.976922] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(2) : 0x41009
[ 113.986241] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(3) : 0x20
[ 113.995268] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(4) : 0x3ffd20
[ 114.004686] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(5) : 0x0
[ 114.013629] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(6) : 0x0
[ 114.022675] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(7) : 0x0
[ 114.031720] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_engctl_r : 0x0
[ 114.039816] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_curctx_r : 0x0
[ 114.047910] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_nxtctx_r : 0x0
[ 114.055998] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_IMB : 0xbadfbadf
[ 114.064997] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_DMB : 0xbadfbadf
[ 114.073999] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CSW : 0xbadfbadf
[ 114.082988] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CTX : 0xbadfbadf
[ 114.091994] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_EXCI : 0xbadfbadf
[ 114.101077] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 114.109958] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 114.118806] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 114.127738] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 114.136635] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 114.145531] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 114.154408] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 114.163303] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 114.172179] NV_PGRAPH_STATUS: 0xa1
[ 114.175749] NV_PGRAPH_STATUS1: 0x0
[ 114.179376] NV_PGRAPH_STATUS2: 0x0
[ 114.182962] NV_PGRAPH_ENGINE_STATUS: 0x1
[ 114.187055] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 114.191251] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 114.195888] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 114.201024] NV_PGRAPH_EXCEPTION : 0x0
[ 114.204931] NV_PGRAPH_FECS_INTR : 0x0
[ 114.208862] NV_PFIFO_ENGINE_STATUS(GR) : 0x80000000
[ 114.213907] NV_PGRAPH_ACTIVITY0: 0x0
[ 114.217663] NV_PGRAPH_ACTIVITY1: 0x600
[ 114.221570] NV_PGRAPH_ACTIVITY2: 0x0
[ 114.225319] NV_PGRAPH_ACTIVITY4: 0x0
[ 114.229051] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 114.233547] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 114.238961] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 114.244432] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 114.249823] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 114.255216] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 114.261036] NV_PGRAPH_PRI_GPC0_TPC1_TPCCS_TPC_ACTIVITY0: 0x0
[ 114.266860] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 114.272677] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 114.278117] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 114.283566] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 114.288997] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 114.294389] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 114.300217] NV_PGRAPH_PRI_GPCS_TPC1_TPCCS_TPC_ACTIVITY0: 0x0
[ 114.306036] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 114.311864] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 114.316998] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 114.322129] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 114.327251] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 114.331858] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x800
[ 114.337040] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x23000700
[ 114.342374] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 114.348783] NV_PGRAPH_PRI_CWD_FS: 0x201
[ 114.352788] NV_PGRAPH_PRI_FE_TPC_FS: 0x3
[ 114.356874] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x100
[ 114.361840] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x100
[ 114.366356] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x2000
[ 114.371818] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x140
[ 114.376960] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 114.382776] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 114.388486] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xe
[ 114.393618] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 114.399300] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x80261f51
[ 114.404697] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x80261f51
[ 114.409739] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x5f00000
[ 114.415053] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x5f00000
[ 114.420360] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 114.425062] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 114.429841] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 114.434545] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
[ 114.439391] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 114.444530] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 114.449940] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 114.455340] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 114.461333] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 114.467172] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 114.473263] gk20a 17000000.gp10b: gr_gk20a_fecs_ctx_image_save: save context image failed
[ 114.485726] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_buffer: Invalid FECS local header: magic value
[ 114.485726]
[ 114.499829] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_pm_buffer: Lookup failed for address 0x500400
[ 114.518500] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_buffer: Invalid FECS local header: magic value
[ 114.518500]
[ 114.530231] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_pm_buffer: Lookup failed for address 0x500400
[ 114.548037] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_buffer: Invalid FECS local header: magic value
[ 114.548037]
[ 114.559721] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_pm_buffer: Lookup failed for address 0x500400
[ 114.574983] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_buffer: Invalid FECS local header: magic value
[ 114.574983]
[ 114.586559] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_pm_buffer: Lookup failed for address 0x500400
[ 114.622170] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_buffer: Invalid FECS local header: magic value
[ 114.622170]
[ 114.633839] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_pm_buffer: Lookup failed for address 0x500400
Ubuntu 16.04 LTS jetson4 ttyS0
jetson4 login: nvidia (automatic login)
Last login: Tue May 22 18:56:55 UTC 2018 on ttyS0
Welcome to Ubuntu 16.04 LTS (GNU/Linux 4.4.38-tegra aarch64)
* Documentation: https://help.ubuntu.com/
324 packages can be updated.
0 updates are security updates.
[ 117.960337] gk20a 17000000.gp10b: gk20a_set_error_notifier_locked: error notifier set to 8 for ch 507
[ 117.969711] gk20a 17000000.gp10b: gk20a_fifo_handle_sched_error: fifo sched ctxsw timeout error: engine=0, tsg=0, ms=3100
[ 117.969904] tegradc 15210000.nvdisplay: unblank
[ 117.969956] ---- mlocks ----
[ 117.969993]
[ 117.969998] ---- syncpts ----
[ 117.970042] id 19 (17000000.gp10b_507) min 2 max 4 refs 1 (previous client : )
[ 117.970949]
[ 117.970954] ---- channels ----
[ 117.970994]
[ 117.970994] channel 1 - 15820000.se
[ 117.970994]
[ 117.971000] NvHost basic channel registers:
[ 117.971008] CMDFIFO_STAT_0: 00002040
[ 117.971015] CMDFIFO_RDATA_0: 00824082
[ 117.971025] CMDP_OFFSET_0: 00000000
[ 117.971031] CMDP_CLASS_0: 00000000
[ 117.971037] CHANNELSTAT_0: 00000000
[ 117.971042] The CDMA sync queue is empty.
[ 117.971045]
[ 117.971056]
[ 117.971056] channel 2 - 15830000.se
[ 117.971056]
[ 117.971060] NvHost basic channel registers:
[ 117.971072] CMDFIFO_STAT_0: 00002040
[ 117.971080] CMDFIFO_RDATA_0: a1942170
[ 117.971098] CMDP_OFFSET_0: 00000000
[ 117.971106] CMDP_CLASS_0: 00000000
[ 117.971114] CHANNELSTAT_0: 00000000
[ 117.971122] The CDMA sync queue is empty.
[ 117.971129]
[ 117.971143]
[ 117.971143] channel 3 - 15840000.se
[ 117.971143]
[ 117.971151] NvHost basic channel registers:
[ 117.971158] CMDFIFO_STAT_0: 00002040
[ 117.971176] CMDFIFO_RDATA_0: 100d1142
[ 117.971190] CMDP_OFFSET_0: 00000000
[ 117.971197] CMDP_CLASS_0: 00000000
[ 117.971204] CHANNELSTAT_0: 00000000
[ 117.971212] The CDMA sync queue is empty.
[ 117.971219]
[ 117.971247]
[ 117.971247] ---- host general irq ----
[ 117.971247]
[ 117.971255] sync_intc0mask = 0x00000001
[ 117.971263] sync_intmask = 0x50000003
[ 117.971271]
[ 117.971271] ---- host syncpt irq mask ----
[ 117.971271]
[ 117.971280]
[ 117.971280] ---- host syncpt irq status ----
[ 117.971280]
[ 117.971293] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 117.971302] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 117.971310] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 117.971327] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 117.971335] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 117.971343] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 117.971360] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 117.971377] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 117.971385] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 117.971393] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 117.971401] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 117.971409] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 117.971417] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 117.971425] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 117.971432] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 117.971440] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 117.971448] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 117.971455] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[ 117.971486] 17000000.gp10b pbdma 0:
[ 117.971488] id: 0 (tsg), next_id: 0 (tsg) chan status: valid
[ 117.971514] PUT: 0000001e00008254 GET: 0000001e00000044 FETCH: 00000006 HEADER: 800302ec
[ 117.971522]
[ 117.971548] 17000000.gp10b eng 0:
[ 117.971554] id: 0 (channel), next_id: 0 (tsg), ctx status: load
[ 117.971558] busy
[ 117.971559]
[ 117.971576] 17000000.gp10b eng 1:
[ 117.971581] id: 0 (tsg), next_id: 0 (tsg), ctx status: valid
[ 117.971581]
[ 117.971585]
[ 117.971908] 507-17000000.gp10b, pid 734, refs: 4:
[ 117.971909] channel status: in use on_pbdma_and_eng busy
[ 117.971932] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 117.971932] HEADER: 20400000 COUNT: 00000000
[ 117.971932] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 117.971936]
[ 117.971957] 508-17000000.gp10b, pid 734, refs: 2:
[ 117.971958] channel status: in use idle not busy
[ 117.971977] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 117.971977] HEADER: 60400000 COUNT: 00000000
[ 117.971977] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 117.971981]
[ 117.971998] 509-17000000.gp10b, pid 734, refs: 2:
[ 117.971999] channel status: in use idle not busy
[ 117.972018] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 117.972018] HEADER: 60400000 COUNT: 00000000
[ 117.972018] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 117.972022]
[ 117.972039] 510-17000000.gp10b, pid 734, refs: 2:
[ 117.972040] channel status: in use idle not busy
[ 117.972058] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 117.972058] HEADER: 60400000 COUNT: 00000000
[ 117.972058] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 117.972062]
[ 117.972079] 511-17000000.gp10b, pid 734, refs: 2:
[ 117.972080] channel status: in use idle not busy
[ 117.972098] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 117.972098] HEADER: 60400000 COUNT: 00000000
[ 117.972098] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 117.972101]
[ 117.972720] gk20a 17000000.gp10b: gk20a_fifo_handle_mmu_fault: fake mmu fault on engine 0, engine subid 0 (gpc), client 12 (rast), addr 0x0000ebf2:0xd0400000, type 8 (pitch mask), info 0x01122c08,inst_ptr 0x1e903a0000
[ 117.972720]
[ 117.972741] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_os_r : 0
[ 117.972760] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_cpuctl_r : 0x40
[ 117.972792] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_idlestate_r : 0x1
[ 117.972804] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox0_r : 0x0
[ 117.972816] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox1_r : 0x0
[ 117.972829] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqstat_r : 0x0
[ 117.972841] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmode_r : 0x4
[ 117.972853] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmask_r : 0x8704
[ 117.972864] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqdest_r : 0x0
[ 117.972877] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_debug1_r : 0x40
[ 117.972898] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_debuginfo_r : 0x0
[ 117.972911] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(0) : 0x10
[ 117.972924] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(1) : 0x0
[ 117.972937] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(2) : 0x41009
[ 117.972949] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(3) : 0x20
[ 117.972965] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(4) : 0x3ffd20
[ 117.972977] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(5) : 0x0
[ 117.973007] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(6) : 0x0
[ 117.973024] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(7) : 0x0
[ 117.973053] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_engctl_r : 0x0
[ 117.973070] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_curctx_r : 0x0
[ 117.973098] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_nxtctx_r : 0x0
[ 117.973115] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_IMB : 0xbadfbadf
[ 117.973146] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_DMB : 0xbadfbadf
[ 117.973170] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CSW : 0xbadfbadf
[ 117.973189] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CTX : 0xbadfbadf
[ 117.973208] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_EXCI : 0xbadfbadf
[ 117.973226] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 117.973245] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 117.973263] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 117.973282] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 117.973301] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 117.973319] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 117.973337] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 117.973355] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 117.973374] gk20a 17000000.gp10b: gk20a_fifo_handle_mmu_fault: gr_status_r : 0xa1
[ 117.991845] gk20a 17000000.gp10b: gk20a_fifo_set_ctx_mmu_error_tsg: TSG 0 generated a mmu fault
[ 117.991945] gk20a 17000000.gp10b: fifo_error_isr: channel reset initiated from fifo_error_isr; intr=0x00000100
nvidia@jetson4:~$ [ 122.728686] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_buffer: Invalid FECS local header: magic value
[ 122.728686]
[ 122.740334] gk20a 17000000.gp10b: gr_gk20a_find_priv_offset_in_pm_buffer: Lookup failed for address 0x500400
[ 126.167446] gk20a 17000000.gp10b: gk20a_set_error_notifier_locked: error notifier set to 8 for ch 507
[ 126.176828] gk20a 17000000.gp10b: gk20a_fifo_handle_sched_error: fifo sched ctxsw timeout error: engine=0, tsg=0, ms=3100
[ 126.176972] tegradc 15210000.nvdisplay: unblank
[ 126.177053] ---- mlocks ----
[ 126.177089]
[ 126.177094] ---- syncpts ----
[ 126.177138] id 19 (17000000.gp10b_507) min 6 max 8 refs 1 (previous client : 17000000.gp10b_507)
[ 126.177937]
[ 126.177942] ---- channels ----
[ 126.177972]
[ 126.177972] channel 1 - 15820000.se
[ 126.177972]
[ 126.177977] NvHost basic channel registers:
[ 126.177985] CMDFIFO_STAT_0: 00002040
[ 126.177991] CMDFIFO_RDATA_0: 00824082
[ 126.178000] CMDP_OFFSET_0: 00000000
[ 126.178007] CMDP_CLASS_0: 00000000
[ 126.178012] CHANNELSTAT_0: 00000000
[ 126.178018] The CDMA sync queue is empty.
[ 126.178021]
[ 126.178031]
[ 126.178031] channel 2 - 15830000.se
[ 126.178031]
[ 126.178035] NvHost basic channel registers:
[ 126.178042] CMDFIFO_STAT_0: 00002040
[ 126.178048] CMDFIFO_RDATA_0: a1942170
[ 126.178055] CMDP_OFFSET_0: 00000000
[ 126.178061] CMDP_CLASS_0: 00000000
[ 126.178066] CHANNELSTAT_0: 00000000
[ 126.178072] The CDMA sync queue is empty.
[ 126.178075]
[ 126.178085]
[ 126.178085] channel 3 - 15840000.se
[ 126.178085]
[ 126.178089] NvHost basic channel registers:
[ 126.178096] CMDFIFO_STAT_0: 00002040
[ 126.178102] CMDFIFO_RDATA_0: 100d1142
[ 126.178109] CMDP_OFFSET_0: 00000000
[ 126.178115] CMDP_CLASS_0: 00000000
[ 126.178120] CHANNELSTAT_0: 00000000
[ 126.178125] The CDMA sync queue is empty.
[ 126.178128]
[ 126.178143]
[ 126.178143] ---- host general irq ----
[ 126.178143]
[ 126.178150] sync_intc0mask = 0x00000001
[ 126.178156] sync_intmask = 0x50000003
[ 126.178160]
[ 126.178160] ---- host syncpt irq mask ----
[ 126.178160]
[ 126.178165]
[ 126.178165] ---- host syncpt irq status ----
[ 126.178165]
[ 126.178173] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 126.178181] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 126.178187] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 126.178194] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 126.178200] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 126.178207] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 126.178213] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 126.178220] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 126.178226] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 126.178233] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 126.178240] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 126.178247] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 126.178253] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 126.178260] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 126.178267] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 126.178273] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 126.178280] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 126.178287] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[ 126.178311] 17000000.gp10b pbdma 0:
[ 126.178312] id: 0 (tsg), next_id: 0 (tsg) chan status: valid
[ 126.178335] PUT: 0000001e00008254 GET: 0000001e00000044 FETCH: 00000006 HEADER: 800302ec
[ 126.178339]
[ 126.178358] 17000000.gp10b eng 0:
[ 126.178363] id: 0 (tsg), next_id: 0 (tsg), ctx status: load
[ 126.178368] busy
[ 126.178369]
[ 126.178385] 17000000.gp10b eng 1:
[ 126.178389] id: 0 (tsg), next_id: 0 (tsg), ctx status: invalid
[ 126.178390]
[ 126.178394]
[ 126.178756] 507-17000000.gp10b, pid 859, refs: 4:
[ 126.178757] channel status: in use on_pbdma_and_eng busy
[ 126.178803] RAMFC : TOP: 8000001e00000044 PUT: 0000001e00008254 GET: 0000001e00000044 FETCH: 00dba81e00001480
[ 126.178803] HEADER: 800302ec COUNT: 01110001
[ 126.178803] SYNCPOINT 00000000 00001301 SEMAPHORE 0000000d fc002000 00000000 01100002
[ 126.178811]
[ 126.178845] 508-17000000.gp10b, pid 734, refs: 2:
[ 126.178846] channel status: in use idle not busy
[ 126.178868] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 126.178868] HEADER: 60400000 COUNT: 00000000
[ 126.178868] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 126.178876]
[ 126.178898] 509-17000000.gp10b, pid 734, refs: 2:
[ 126.178899] channel status: in use idle not busy
[ 126.178919] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 126.178919] HEADER: 60400000 COUNT: 00000000
[ 126.178919] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 126.178927]
[ 126.178949] 510-17000000.gp10b, pid 734, refs: 2:
[ 126.178950] channel status: in use idle not busy
[ 126.178971] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 126.178971] HEADER: 60400000 COUNT: 00000000
[ 126.178971] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 126.178978]
[ 126.179005] 511-17000000.gp10b, pid 734, refs: 2:
[ 126.179006] channel status: in use idle not busy
[ 126.179026] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
[ 126.179026] HEADER: 60400000 COUNT: 00000000
[ 126.179026] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 126.179034]
[ 126.179539] gk20a 17000000.gp10b: gk20a_fifo_handle_mmu_fault: fake mmu fault on engine 0, engine subid 0 (gpc), client 12 (rast), addr 0x0000ebf2:0xd0400000, type 8 (pitch mask), info 0x01122c08,inst_ptr 0x1e903a0000
[ 126.179539]
[ 126.179555] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_os_r : 0
[ 126.179569] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_cpuctl_r : 0x40
[ 126.179583] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_idlestate_r : 0x1
[ 126.179595] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox0_r : 0x0
[ 126.179607] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox1_r : 0x0
[ 126.179619] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqstat_r : 0x0
[ 126.179631] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmode_r : 0x4
[ 126.179643] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmask_r : 0x8704
[ 126.179655] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqdest_r : 0x0
[ 126.179667] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_debug1_r : 0x40
[ 126.179679] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_debuginfo_r : 0x0
[ 126.179692] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(0) : 0x1
[ 126.179704] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(1) : 0x0
[ 126.179717] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(2) : 0x90009
[ 126.179729] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(3) : 0x0
[ 126.179741] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(4) : 0x3ffd20
[ 126.179753] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(5) : 0x1
[ 126.179765] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(6) : 0x1
[ 126.179776] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(7) : 0x0
[ 126.179789] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_engctl_r : 0x0
[ 126.179801] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_curctx_r : 0x0
[ 126.179812] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_nxtctx_r : 0x0
[ 126.179825] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_IMB : 0xbadfbadf
[ 126.179852] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_DMB : 0xbadfbadf
[ 126.179866] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CSW : 0xbadfbadf
[ 126.179879] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CTX : 0xbadfbadf
[ 126.179892] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_EXCI : 0xbadfbadf
[ 126.179905] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 126.179918] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 126.179932] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 126.179945] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 126.179957] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 126.179970] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 126.179982] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 126.179995] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 126.180008] gk20a 17000000.gp10b: gk20a_fifo_handle_mmu_fault: gr_status_r : 0x200081
[ 129.182750] gk20a 17000000.gp10b: Timeout detected @ gr_gk20a_submit_fecs_method_op+0x104/0x274
[ 129.208510] gk20a 17000000.gp10b: gr_gk20a_ctx_wait_ucode: timeout waiting on ucode response
[ 129.233908] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_os_r : 0
[ 129.258196] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_cpuctl_r : 0x40
[ 129.283088] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_idlestate_r : 0x1
[ 129.308140] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox0_r : 0x0
[ 129.333081] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox1_r : 0x0
[ 129.358023] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqstat_r : 0x0
[ 129.382840] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmode_r : 0x4
[ 129.407671] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmask_r : 0x8704
[ 129.432744] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_irqdest_r : 0x0
[ 129.457546] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_debug1_r : 0x40
[ 129.482398] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_debuginfo_r : 0x0
[ 129.507380] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(0) : 0x0
[ 129.532967] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(1) : 0xffffffff
[ 129.559232] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(2) : 0x94019
[ 129.586801] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(3) : 0x0
[ 129.612477] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(4) : 0x3ffd20
[ 129.643352] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(5) : 0x1
[ 129.669035] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(6) : 0x1
[ 129.694706] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(7) : 0x0
[ 129.720404] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_engctl_r : 0x0
[ 129.745209] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_curctx_r : 0x0
[ 129.770023] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: gr_fecs_nxtctx_r : 0x0
[ 129.794817] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_IMB : 0xbadfbadf
[ 129.820548] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_DMB : 0xbadfbadf
[ 129.846272] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CSW : 0xbadfbadf
[ 129.871995] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CTX : 0xbadfbadf
[ 129.897722] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_EXCI : 0xbadfbadf
[ 129.923550] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 129.949179] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 129.974934] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 130.000612] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 130.026267] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 130.051906] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 130.077594] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadfbadf
[ 130.103277] gk20a 17000000.gp10b: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadfbadf
[ 130.128951] NV_PGRAPH_STATUS: 0x200089
[ 130.141073] NV_PGRAPH_STATUS1: 0x0
[ 130.152623] NV_PGRAPH_STATUS2: 0x0
[ 130.163890] NV_PGRAPH_ENGINE_STATUS: 0x1
[ 130.175505] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 130.186996] NV_PGRAPH_GRFIFO_CONTROL : 0x0
[ 130.198381] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x1
[ 130.210485] NV_PGRAPH_EXCEPTION : 0x0
[ 130.221172] NV_PGRAPH_FECS_INTR : 0x0
[ 130.231624] NV_PFIFO_ENGINE_STATUS(GR) : 0xd000b000
[ 130.243122] NV_PGRAPH_ACTIVITY0: 0x0
[ 130.253108] NV_PGRAPH_ACTIVITY1: 0x7000
[ 130.263130] NV_PGRAPH_ACTIVITY2: 0x0
[ 130.272627] NV_PGRAPH_ACTIVITY4: 0x0
[ 130.281872] NV_PGRAPH_PRI_SKED_ACTIVITY: 0xe01c0000
[ 130.292321] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0xffffff
[ 130.303441] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0xffffff
[ 130.314293] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x7fcf
[ 130.324744] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x3f
[ 130.334817] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0xfff
[ 130.345220] NV_PGRAPH_PRI_GPC0_TPC1_TPCCS_TPC_ACTIVITY0: 0xfff
[ 130.355362] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0xfff
[ 130.365258] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0xffffff
[ 130.374814] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0xffffff
[ 130.384109] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x7fcf
[ 130.393007] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x3f
[ 130.401478] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0xfff
[ 130.410362] NV_PGRAPH_PRI_GPCS_TPC1_TPCCS_TPC_ACTIVITY0: 0xfff
[ 130.418966] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0xfff
[ 130.427287] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x7fff
[ 130.434972] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x7fff
[ 130.442635] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x7fff
[ 130.450345] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 130.457241] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x0
[ 130.464509] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x0
[ 130.471519] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x3f77
[ 130.480555] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 130.486750] NV_PGRAPH_PRI_FE_TPC_FS: 0x2
[ 130.493219] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 130.500373] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 130.507023] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x3
[ 130.514658] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x4080
[ 130.522312] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 130.530708] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x1300
[ 130.538940] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xe
[ 130.546536] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xe
[ 130.554760] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x80265596
[ 130.562815] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x80265596
[ 130.570619] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x5f00000
[ 130.578655] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x5f00000
[ 130.586664] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0xe00000
[ 130.594557] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x7
[ 130.602021] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0xe00000
[ 130.609877] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x7
[ 130.617428] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 130.625365] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 130.633513] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 130.641721] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 130.650601] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 130.659291] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 130.668261] gk20a 17000000.gp10b: gk20a_fifo_reset_engine: failed to HALT gr pipe
[ 130.686905] gk20a 17000000.gp10b: gr_gk20a_load_falcon_bind_instblk: arbiter complete timeout
[ 130.702759] gk20a 17000000.gp10b: gr_gk20a_load_falcon_bind_instblk: arbiter complete timeout
[ 156.266726] Watchdog detected hard LOCKUP on cpu 3
[ 156.271699] ------------[ cut here ]------------
[ 156.281693] WARNING: at ffffffc00013edf8 [verbose debug info unavailable]
[ 156.291206] Modules linked in: bcmdhd pci_tegra bluedroid_pm
[ 156.299833]
[ 156.304114] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.4.38-tegra #1
[ 156.313476] Hardware name: quill (DT)
[ 156.320098] task: ffffffc1ece81900 ti: ffffffc1ece94000 task.ti: ffffffc1ece94000
[ 156.333707] PC is at watchdog_timer_fn+0x230/0x33c
[ 156.341684] LR is at watchdog_timer_fn+0x230/0x33c
[ 156.349601] pc : [<ffffffc00013edf8>] lr : [<ffffffc00013edf8>] pstate: 600001c5
[ 156.363401] sp : ffffffc1ece97ae0
[ 156.370059] x29: ffffffc1ece97ae0 x28: 0000000000000003
[ 156.378863] x27: ffffffc001281b30 x26: ffffffc1f5fbe278
[ 156.392463] x25: ffffffc0012502d8 x24: ffffffc1ece97dc0
[ 156.401297] x23: 0000000000000000 x22: 0000000000000000
[ 156.410157] x21: ffffffc001281000 x20: ffffffc001250000
[ 156.419046] x19: ffffffc001250260 x18: ffffffc000bfc248
[ 156.427933] x17: 000000000000000e x16: ffffffc000b88a60
[ 156.436844] x15: ffffffc000b88a60 x14: 0000000000000008
[ 156.445783] x13: ffffffc1eaa686c0 x12: 0000000000000001
[ 156.454756] x11: 00000000ffffffff x10: 0000000000aaaaaa
[ 156.463664] x9 : 0000000000000522 x8 : 0000000000000000
[ 156.472447] x7 : 0000000000000001 x6 : ffffffc001299738
[ 156.481125] x5 : 0000000000000000 x4 : 0000000000000000
[ 156.489696] x3 : 0000000000000000 x2 : 0000000000010001
[ 156.501663] x1 : ffffffc1ece94000 x0 : 0000000000000026
[ 156.513660]
[ 156.522384] ---[ end trace b92561905f843e52 ]---
[ 156.531099] Call trace:
[ 156.536739] [<ffffffc00013edf8>] watchdog_timer_fn+0x230/0x33c
[ 156.545808] [<ffffffc000107d64>] __hrtimer_run_queues+0x140/0x350
[ 156.555094] [<ffffffc0001087c4>] hrtimer_interrupt+0x9c/0x1e0
[ 156.563982] [<ffffffc000936874>] tegra186_timer_isr+0x24/0x30
[ 156.572820] [<ffffffc0000f5650>] handle_irq_event_percpu+0x84/0x290
[ 156.582122] [<ffffffc0000f58a0>] handle_irq_event+0x44/0x74
[ 156.590582] [<ffffffc0000f8ba8>] handle_fasteoi_irq+0xb4/0x188
[ 156.599258] [<ffffffc0000f4c70>] generic_handle_irq+0x24/0x38
[ 156.607896] [<ffffffc0000f4f78>] __handle_domain_irq+0x60/0xb4
[ 156.616549] [<ffffffc000081774>] gic_handle_irq+0x5c/0xb4
[ 156.624692] [<ffffffc000084740>] el1_irq+0x80/0xf8
[ 156.632197] [<ffffffc000820d20>] cpuidle_enter+0x18/0x20
[ 156.640258] [<ffffffc0000e8354>] call_cpuidle+0x28/0x50
[ 156.648224] [<ffffffc0000e84f8>] cpu_startup_entry+0x17c/0x340
[ 156.656847] [<ffffffc00008ee44>] secondary_start_kernel+0x12c/0x164
[ 156.665904] [<0000000080081acc>] 0x80081acc
If you have any ideas for things to try let me know. Thanks!