GPU error and xdisplay not work

We have use our customize carrierboard and match Jetson nano B01 model . We use jetpack 4.4 burning, after a while, GPU error is reported, and xdisplay will not be displayed. Through the terminal, you can see that xorg process cpu usage up to 100% , we donn’t know why, can someone help us

test@test-desktop:~$ top
top - 00:44:49 up 46 min, 2 users, load average: 1.00, 1.00, 1.00
Tasks: 289 total, 2 running, 287 sleeping, 0 stopped, 0 zombie
%Cpu(s): 8.0 us, 14.7 sy, 0.0 ni, 77.0 id, 0.1 wa, 0.1 hi, 0.1 si, 0.0 st
KiB Mem : 4051084 total, 2226876 free, 1047876 used, 776332 buff/cache
KiB Swap: 2025536 total, 2025536 free, 0 used. 3059476 avail Mem

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
      6397 root      20   0 6431256  36852  20516 R 100.0  0.9  35:44.91 Xorg
      8909 test      20   0   10872   3864   3128 R  16.7  0.1   0:00.05 top
      1 root      20   0  161296   8340   5732 S   0.0  0.2   0:02.51 systemd
      2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd
      3 root      20   0       0      0      0 S   0.0  0.0   0:00.03 ksoftirqd/0
      5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+
      7 root      20   0       0      0      0 S   0.0  0.0   0:00.17 rcu_preempt

1229.464843]
---- host syncpt thresh ----

[ 1229.464846] syncpt_int_thresh_thresh_0(0) = 1
[ 1229.464855] syncpt_int_thresh_thresh_0(15) = 9354
[ 1229.464941] gm20b pbdma 0:
[ 1229.464945] id: 10 (tsg), next_id: 10 (tsg) chan status: valid
[ 1229.464959] PBDMA_PUT: 0000001e0438350c PBDMA_GET: 0000001e043833cc GP_PUT: 0000034f GP_GET: 00000319 FETCH: 0000034f HEADER: 20020180
HDR: 20024060 SHADOW0: 04382e5c SHADOW1: 0006b01e

[ 1229.464966] gm20b eng 0:
[ 1229.464969] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1229.464974] gm20b eng 1:
[ 1229.464977] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1229.465165] 501-gm20b, pid 7012, refs 11:
[ 1229.465169] channel status: not in use pending_acquire busy
[ 1229.465175] RAMFC : TOP: 8000001e043833cc PUT: 0000001e0438350c GET: 0000001e043833cc FETCH: 0000421e021c77d0
HEADER: 20020180 COUNT: 00110002
SYNCPOINT 00000000 00000f01 SEMAPHORE 0000001e 02380010 00000001 00081001

[ 1229.465182] 502-gm20b, pid 6532, refs 2:
[ 1229.465185] channel status: in use idle not busy
[ 1229.465191] RAMFC : TOP: 8000001f0000c0a8 PUT: 0000001f0000c0a8 GET: 0000001f0000c0a8 FETCH: 0000001f0000c0a8
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000e01 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1229.465196] 503-gm20b, pid 6532, refs 2:
[ 1229.465199] channel status: not in use idle not busy
[ 1229.465204] RAMFC : TOP: 8000001f00004d00 PUT: 0000001f00004d00 GET: 0000001f00004d00 FETCH: 0000001f00004d00
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000d01 SEMAPHORE 0000000d fc001000 019c2c3f 01100002

[ 1229.465210] 504-gm20b, pid 6155, refs 2:
[ 1229.465212] channel status: in use idle not busy
[ 1229.465219] RAMFC : TOP: 8000001f00068060 PUT: 0000001f00068060 GET: 0000001f00068060 FETCH: 0000001f00068060
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000c01 SEMAPHORE 0000001e 00dd0aa0 00000000 00000002

[ 1229.465224] 505-gm20b, pid 5607, refs 2:
[ 1229.465227] channel status: in use idle not busy
[ 1229.465232] RAMFC : TOP: 8000001f00002670 PUT: 0000001f00002670 GET: 0000001f00002670 FETCH: 0000001f00002670
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000b01 SEMAPHORE 0000001e 00890000 00000002 00000002

[ 1229.465237] 506-gm20b, pid 5185, refs 2:
[ 1229.465239] channel status: in use idle not busy
[ 1229.465244] RAMFC : TOP: 8000001f0000c150 PUT: 0000001f0000c150 GET: 0000001f0000c150 FETCH: 0000001f0000c150
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000901 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1229.465249] 507-gm20b, pid 5185, refs 2:
[ 1229.465252] channel status: in use idle not busy
[ 1229.465257] RAMFC : TOP: 8000001f00005fc0 PUT: 0000001f00005fc0 GET: 0000001f00005fc0 FETCH: 0000001f00005fc0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000801 SEMAPHORE 0000000d fc001000 03f4219e 01100002

[ 1229.465262] 508-gm20b, pid 4687, refs 2:
[ 1229.465264] channel status: in use idle not busy
[ 1229.465269] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1229.465275] 509-gm20b, pid 4687, refs 2:
[ 1229.465277] channel status: in use idle not busy
[ 1229.465282] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1229.465287] 510-gm20b, pid 4687, refs 2:
[ 1229.465289] channel status: in use idle not busy
[ 1229.465294] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1229.465302] 511-gm20b, pid 4687, refs 2:
[ 1229.465304] channel status: in use idle not busy
[ 1229.465309] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1234.710759] nvgpu: 57000000.gpu gk20a_channel_timeout_handler:1570 [ERR] Job on channel 501 timed out
[ 1234.721129] NV_PGRAPH_STATUS: 0x400000
[ 1234.721134] NV_PGRAPH_STATUS1: 0x0
[ 1234.721137] NV_PGRAPH_STATUS2: 0x0
[ 1234.721141] NV_PGRAPH_ENGINE_STATUS: 0x0
[ 1234.721145] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 1234.721149] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 1234.721152] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 1234.721156] NV_PGRAPH_EXCEPTION : 0x0
[ 1234.721159] NV_PGRAPH_FECS_INTR : 0x0
[ 1234.721162] NV_PFIFO_ENGINE_STATUS(GR) : 0x100a100a
[ 1234.721166] NV_PGRAPH_ACTIVITY0: 0x0
[ 1234.721169] NV_PGRAPH_ACTIVITY1: 0xc0000
[ 1234.721173] NV_PGRAPH_ACTIVITY2: 0x0
[ 1234.721176] NV_PGRAPH_ACTIVITY4: 0x0
[ 1234.721180] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 1234.721184] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 1234.721188] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 1234.721192] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 1234.721196] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 1234.721201] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1234.721205] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1234.721209] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 1234.721213] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 1234.721217] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 1234.721221] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 1234.721225] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1234.721230] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1234.721235] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 1234.721239] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 1234.721243] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 1234.721248] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 1234.721251] NV_PGRAPH_PRI_FE_GO_IDLE_ON_STATUS: 0x2e
[ 1234.721255] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x400
[ 1234.721259] NV_PGRAPH_PRI_FE_GO_IDLE_CHECK : 0x200
[ 1234.721263] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x33000700
[ 1234.721268] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 1234.721273] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 1234.721276] NV_PGRAPH_PRI_FE_TPC_FS: 0x0
[ 1234.721280] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 1234.721284] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 1234.721287] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x0
[ 1234.721290] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x300
[ 1234.721295] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 1234.721299] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 1234.721303] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xf
[ 1234.721308] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 1234.721313] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x7fac3e
[ 1234.721316] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x7fac3e
[ 1234.721321] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x0
[ 1234.721325] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x0
[ 1234.721330] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 1234.721334] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 1234.721338] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 1234.721343] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
[ 1234.721347] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 1234.721351] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 1234.721356] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 1234.721360] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 1234.721365] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 1234.721369] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 1234.721388] nvgpu: 57000000.gpu nvgpu_set_error_notifier_locked:137 [ERR] error notifier set to 8 for ch 501
[ 1234.732398] ---- mlocks ----

[ 1234.732410] ---- syncpts ----
[ 1234.732417] id 1 (disp0_a) min 640 max 640 refs 1 (previous client : )
[ 1234.732421] id 2 (disp0_b) min 4 max 4 refs 1 (previous client : )
[ 1234.732424] id 3 (disp0_c) min 4 max 4 refs 1 (previous client : )
[ 1234.732430] id 8 (gm20b_507) min 508 max 508 refs 1 (previous client : )
[ 1234.732434] id 9 (gm20b_506) min 28 max 28 refs 1 (previous client : )
[ 1234.732439] id 11 (gm20b_505) min 576 max 576 refs 1 (previous client : gm20b_505)
[ 1234.732442] id 12 (gm20b_504) min 6 max 6 refs 1 (previous client : )
[ 1234.732446] id 13 (gm20b_503) min 17620 max 17620 refs 1 (previous client : )
[ 1234.732449] id 14 (gm20b_502) min 14 max 14 refs 1 (previous client : )
[ 1234.732453] id 15 (gm20b_501) min 9380 max 9388 refs 1 (previous client : gm20b_501)
[ 1234.732463] id 26 (vblank0) min -6 max -6 refs 1 (previous client : )

[ 1234.732558] ---- channels ----
[ 1234.732563]
channel 0 - 54340000.vic

[ 1234.732569] 0-54340000.vic (0):
[ 1234.732571] inactive

[ 1234.732577]
---- host general irq ----

[ 1234.732579] sync_hintmask_ext = 0xc0000000
[ 1234.732582] sync_hintmask = 0x80000000
[ 1234.732584] sync_intc0mask = 0x00000001
[ 1234.732587] sync_intmask = 0x00000011
[ 1234.732589]
---- host syncpt irq mask ----

[ 1234.732592] syncpt_thresh_int_mask(0) = 0x40000001
[ 1234.732594] syncpt_thresh_int_mask(1) = 0x00000000
[ 1234.732597] syncpt_thresh_int_mask(2) = 0x00000000
[ 1234.732599] syncpt_thresh_int_mask(3) = 0x00000000
[ 1234.732602] syncpt_thresh_int_mask(4) = 0x00000000
[ 1234.732604] syncpt_thresh_int_mask(5) = 0x00000000
[ 1234.732607] syncpt_thresh_int_mask(6) = 0x00000000
[ 1234.732609] syncpt_thresh_int_mask(7) = 0x00000000
[ 1234.732612] syncpt_thresh_int_mask(8) = 0x00000000
[ 1234.732614] syncpt_thresh_int_mask(9) = 0x00000000
[ 1234.732617] syncpt_thresh_int_mask(10) = 0x00000000
[ 1234.732619] syncpt_thresh_int_mask(11) = 0x00000000
[ 1234.732621]
---- host syncpt irq status ----

[ 1234.732624] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1234.732628] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1234.732631] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1234.732633] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1234.732636] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1234.732638] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1234.732640]
---- host syncpt thresh ----

[ 1234.732643] syncpt_int_thresh_thresh_0(0) = 1
[ 1234.732653] syncpt_int_thresh_thresh_0(15) = 9382
[ 1234.732738] gm20b pbdma 0:
[ 1234.732742] id: 10 (tsg), next_id: 10 (tsg) chan status: valid
[ 1234.732756] PBDMA_PUT: 0000001e0432350c PBDMA_GET: 0000001e043233cc GP_PUT: 0000034f GP_GET: 00000319 FETCH: 0000034f HEADER: 20020180
HDR: 20024060 SHADOW0: 04322e5c SHADOW1: 0006b01e

[ 1234.732761] gm20b eng 0:
[ 1234.732765] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1234.732769] gm20b eng 1:
[ 1234.732772] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1234.732961] 501-gm20b, pid 7012, refs 11:
[ 1234.732964] channel status: not in use pending_acquire busy
[ 1234.732971] RAMFC : TOP: 8000001e043233cc PUT: 0000001e0432350c GET: 0000001e043233cc FETCH: 0000421e023077d0
HEADER: 20020180 COUNT: 00110002
SYNCPOINT 00000000 00000f01 SEMAPHORE 0000001e 02380030 00000001 00081001

[ 1234.732978] 502-gm20b, pid 6532, refs 2:
[ 1234.732981] channel status: in use idle not busy
[ 1234.732986] RAMFC : TOP: 8000001f0000c0a8 PUT: 0000001f0000c0a8 GET: 0000001f0000c0a8 FETCH: 0000001f0000c0a8
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000e01 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1234.732992] 503-gm20b, pid 6532, refs 2:
[ 1234.732994] channel status: not in use idle not busy
[ 1234.733000] RAMFC : TOP: 8000001f00004d00 PUT: 0000001f00004d00 GET: 0000001f00004d00 FETCH: 0000001f00004d00
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000d01 SEMAPHORE 0000000d fc001000 019c2c3f 01100002

[ 1234.733006] 504-gm20b, pid 6155, refs 2:
[ 1234.733008] channel status: in use idle not busy
[ 1234.733013] RAMFC : TOP: 8000001f00068060 PUT: 0000001f00068060 GET: 0000001f00068060 FETCH: 0000001f00068060
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000c01 SEMAPHORE 0000001e 00dd0aa0 00000000 00000002

[ 1234.733018] 505-gm20b, pid 5607, refs 2:
[ 1234.733021] channel status: in use idle not busy
[ 1234.733026] RAMFC : TOP: 8000001f00002670 PUT: 0000001f00002670 GET: 0000001f00002670 FETCH: 0000001f00002670
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000b01 SEMAPHORE 0000001e 00890000 00000002 00000002

[ 1234.733031] 506-gm20b, pid 5185, refs 2:
[ 1234.733034] channel status: in use idle not busy
[ 1234.733039] RAMFC : TOP: 8000001f0000c150 PUT: 0000001f0000c150 GET: 0000001f0000c150 FETCH: 0000001f0000c150
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000901 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1234.733044] 507-gm20b, pid 5185, refs 2:
[ 1234.733047] channel status: in use idle not busy
[ 1234.733053] RAMFC : TOP: 8000001f00005fc0 PUT: 0000001f00005fc0 GET: 0000001f00005fc0 FETCH: 0000001f00005fc0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000801 SEMAPHORE 0000000d fc001000 03f4219e 01100002

[ 1234.733059] 508-gm20b, pid 4687, refs 2:
[ 1234.733061] channel status: in use idle not busy
[ 1234.733066] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1234.733071] 509-gm20b, pid 4687, refs 2:
[ 1234.733074] channel status: in use idle not busy
[ 1234.733079] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1234.733084] 510-gm20b, pid 4687, refs 2:
[ 1234.733086] channel status: in use idle not busy
[ 1234.733091] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1234.733097] 511-gm20b, pid 4687, refs 2:
[ 1234.733099] channel status: in use idle not busy
[ 1234.733104] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1239.982756] nvgpu: 57000000.gpu gk20a_channel_timeout_handler:1570 [ERR] Job on channel 501 timed out
[ 1239.993125] NV_PGRAPH_STATUS: 0x400000
[ 1239.993130] NV_PGRAPH_STATUS1: 0x0
[ 1239.993134] NV_PGRAPH_STATUS2: 0x0
[ 1239.993138] NV_PGRAPH_ENGINE_STATUS: 0x0
[ 1239.993141] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 1239.993145] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 1239.993149] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 1239.993153] NV_PGRAPH_EXCEPTION : 0x0
[ 1239.993156] NV_PGRAPH_FECS_INTR : 0x0
[ 1239.993160] NV_PFIFO_ENGINE_STATUS(GR) : 0x100a100a
[ 1239.993163] NV_PGRAPH_ACTIVITY0: 0x0
[ 1239.993167] NV_PGRAPH_ACTIVITY1: 0xc0000
[ 1239.993171] NV_PGRAPH_ACTIVITY2: 0x0
[ 1239.993175] NV_PGRAPH_ACTIVITY4: 0x0
[ 1239.993178] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 1239.993183] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 1239.993187] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 1239.993191] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 1239.993195] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 1239.993200] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1239.993204] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1239.993208] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 1239.993212] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 1239.993216] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 1239.993220] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 1239.993224] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1239.993229] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1239.993234] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 1239.993238] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 1239.993242] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 1239.993246] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 1239.993250] NV_PGRAPH_PRI_FE_GO_IDLE_ON_STATUS: 0x2e
[ 1239.993254] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x400
[ 1239.993257] NV_PGRAPH_PRI_FE_GO_IDLE_CHECK : 0x200
[ 1239.993261] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x33000700
[ 1239.993266] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 1239.993270] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 1239.993274] NV_PGRAPH_PRI_FE_TPC_FS: 0x0
[ 1239.993277] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 1239.993281] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 1239.993284] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x0
[ 1239.993287] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x300
[ 1239.993292] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 1239.993296] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 1239.993300] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xf
[ 1239.993305] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 1239.993309] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x7fac1f
[ 1239.993313] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x7fac1f
[ 1239.993317] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x0
[ 1239.993322] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x0
[ 1239.993326] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 1239.993330] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 1239.993335] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 1239.993339] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
[ 1239.993343] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 1239.993347] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 1239.993352] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 1239.993356] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 1239.993361] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 1239.993365] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 1239.993384] nvgpu: 57000000.gpu nvgpu_set_error_notifier_locked:137 [ERR] error notifier set to 8 for ch 501
[ 1240.004476] ---- mlocks ----

[ 1240.004488] ---- syncpts ----
[ 1240.004495] id 1 (disp0_a) min 640 max 640 refs 1 (previous client : )
[ 1240.004499] id 2 (disp0_b) min 4 max 4 refs 1 (previous client : )
[ 1240.004502] id 3 (disp0_c) min 4 max 4 refs 1 (previous client : )
[ 1240.004508] id 8 (gm20b_507) min 508 max 508 refs 1 (previous client : )
[ 1240.004512] id 9 (gm20b_506) min 28 max 28 refs 1 (previous client : )
[ 1240.004516] id 11 (gm20b_505) min 576 max 576 refs 1 (previous client : gm20b_505)
[ 1240.004520] id 12 (gm20b_504) min 6 max 6 refs 1 (previous client : )
[ 1240.004523] id 13 (gm20b_503) min 17620 max 17620 refs 1 (previous client : )
[ 1240.004527] id 14 (gm20b_502) min 14 max 14 refs 1 (previous client : )
[ 1240.004531] id 15 (gm20b_501) min 9408 max 9418 refs 1 (previous client : gm20b_501)
[ 1240.004540] id 26 (vblank0) min -6 max -6 refs 1 (previous client : )

[ 1240.004637] ---- channels ----
[ 1240.004643]
channel 0 - 54340000.vic

[ 1240.004649] 0-54340000.vic (0):
[ 1240.004651] inactive

[ 1240.004656]
---- host general irq ----

[ 1240.004659] sync_hintmask_ext = 0xc0000000
[ 1240.004662] sync_hintmask = 0x80000000
[ 1240.004664] sync_intc0mask = 0x00000001
[ 1240.004667] sync_intmask = 0x00000011
[ 1240.004669]
---- host syncpt irq mask ----

[ 1240.004672] syncpt_thresh_int_mask(0) = 0x40000001
[ 1240.004675] syncpt_thresh_int_mask(1) = 0x00000000
[ 1240.004677] syncpt_thresh_int_mask(2) = 0x00000000
[ 1240.004680] syncpt_thresh_int_mask(3) = 0x00000000
[ 1240.004682] syncpt_thresh_int_mask(4) = 0x00000000
[ 1240.004685] syncpt_thresh_int_mask(5) = 0x00000000
[ 1240.004688] syncpt_thresh_int_mask(6) = 0x00000000
[ 1240.004690] syncpt_thresh_int_mask(7) = 0x00000000
[ 1240.004693] syncpt_thresh_int_mask(8) = 0x00000000
[ 1240.004695] syncpt_thresh_int_mask(9) = 0x00000000
[ 1240.004698] syncpt_thresh_int_mask(10) = 0x00000000
[ 1240.004700] syncpt_thresh_int_mask(11) = 0x00000000
[ 1240.004702]
---- host syncpt irq status ----

[ 1240.004705] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1240.004709] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1240.004712] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1240.004714] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1240.004717] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1240.004719] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1240.004721]
---- host syncpt thresh ----

[ 1240.004724] syncpt_int_thresh_thresh_0(0) = 1
[ 1240.004734] syncpt_int_thresh_thresh_0(15) = 9410
[ 1240.004819] gm20b pbdma 0:
[ 1240.004823] id: 10 (tsg), next_id: 10 (tsg) chan status: valid
[ 1240.004837] PBDMA_PUT: 0000001e042c350c PBDMA_GET: 0000001e042c33cc GP_PUT: 00000353 GP_GET: 00000319 FETCH: 00000353 HEADER: 20020180
HDR: 20024060 SHADOW0: 042c2e5c SHADOW1: 0006b01e

[ 1240.004843] gm20b eng 0:
[ 1240.004846] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1240.004851] gm20b eng 1:
[ 1240.004854] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1240.005045] 501-gm20b, pid 7012, refs 13:
[ 1240.005049] channel status: not in use pending_acquire busy
[ 1240.005055] RAMFC : TOP: 8000001e042c33cc PUT: 0000001e042c350c GET: 0000001e042c33cc FETCH: 0000421e032c77d0
HEADER: 20020180 COUNT: 00110002
SYNCPOINT 00000000 00000f01 SEMAPHORE 0000001e 02380050 00000001 00081001

[ 1240.005065] 502-gm20b, pid 6532, refs 2:
[ 1240.005068] channel status: in use idle not busy
[ 1240.005074] RAMFC : TOP: 8000001f0000c0a8 PUT: 0000001f0000c0a8 GET: 0000001f0000c0a8 FETCH: 0000001f0000c0a8
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000e01 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1240.005079] 503-gm20b, pid 6532, refs 2:
[ 1240.005082] channel status: not in use idle not busy
[ 1240.005088] RAMFC : TOP: 8000001f00004d00 PUT: 0000001f00004d00 GET: 0000001f00004d00 FETCH: 0000001f00004d00
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000d01 SEMAPHORE 0000000d fc001000 019c2c3f 01100002

[ 1240.005093] 504-gm20b, pid 6155, refs 2:
[ 1240.005095] channel status: in use idle not busy
[ 1240.005101] RAMFC : TOP: 8000001f00068060 PUT: 0000001f00068060 GET: 0000001f00068060 FETCH: 0000001f00068060
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000c01 SEMAPHORE 0000001e 00dd0aa0 00000000 00000002

[ 1240.005106] 505-gm20b, pid 5607, refs 2:
[ 1240.005109] channel status: in use idle not busy
[ 1240.005114] RAMFC : TOP: 8000001f00002670 PUT: 0000001f00002670 GET: 0000001f00002670 FETCH: 0000001f00002670
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000b01 SEMAPHORE 0000001e 00890000 00000002 00000002

[ 1240.005120] 506-gm20b, pid 5185, refs 2:
[ 1240.005122] channel status: in use idle not busy
[ 1240.005128] RAMFC : TOP: 8000001f0000c150 PUT: 0000001f0000c150 GET: 0000001f0000c150 FETCH: 0000001f0000c150
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000901 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1240.005133] 507-gm20b, pid 5185, refs 2:
[ 1240.005136] channel status: in use idle not busy
[ 1240.005142] RAMFC : TOP: 8000001f00005fc0 PUT: 0000001f00005fc0 GET: 0000001f00005fc0 FETCH: 0000001f00005fc0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000801 SEMAPHORE 0000000d fc001000 03f4219e 01100002

[ 1240.005148] 508-gm20b, pid 4687, refs 2:
[ 1240.005150] channel status: in use idle not busy
[ 1240.005155] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1240.005161] 509-gm20b, pid 4687, refs 2:
[ 1240.005163] channel status: in use idle not busy
[ 1240.005168] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1240.005174] 510-gm20b, pid 4687, refs 2:
[ 1240.005176] channel status: in use idle not busy
[ 1240.005181] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1240.005187] 511-gm20b, pid 4687, refs 2:
[ 1240.005189] channel status: in use idle not busy
[ 1240.005194] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1245.250802] nvgpu: 57000000.gpu gk20a_channel_timeout_handler:1570 [ERR] Job on channel 501 timed out
[ 1245.261186] NV_PGRAPH_STATUS: 0x400000
[ 1245.261190] NV_PGRAPH_STATUS1: 0x0
[ 1245.261195] NV_PGRAPH_STATUS2: 0x0
[ 1245.261198] NV_PGRAPH_ENGINE_STATUS: 0x0
[ 1245.261202] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 1245.261206] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 1245.261209] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 1245.261213] NV_PGRAPH_EXCEPTION : 0x0
[ 1245.261216] NV_PGRAPH_FECS_INTR : 0x0
[ 1245.261219] NV_PFIFO_ENGINE_STATUS(GR) : 0x100a100a
[ 1245.261223] NV_PGRAPH_ACTIVITY0: 0x0
[ 1245.261227] NV_PGRAPH_ACTIVITY1: 0xc0000
[ 1245.261230] NV_PGRAPH_ACTIVITY2: 0x0
[ 1245.261234] NV_PGRAPH_ACTIVITY4: 0x0
[ 1245.261237] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 1245.261242] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 1245.261246] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 1245.261250] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 1245.261254] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 1245.261258] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1245.261263] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1245.261267] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 1245.261271] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 1245.261275] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 1245.261278] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 1245.261283] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1245.261287] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1245.261293] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 1245.261297] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 1245.261301] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 1245.261306] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 1245.261309] NV_PGRAPH_PRI_FE_GO_IDLE_ON_STATUS: 0x2e
[ 1245.261313] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x400
[ 1245.261317] NV_PGRAPH_PRI_FE_GO_IDLE_CHECK : 0x200
[ 1245.261320] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x33000700
[ 1245.261326] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 1245.261331] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 1245.261334] NV_PGRAPH_PRI_FE_TPC_FS: 0x0
[ 1245.261338] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 1245.261341] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 1245.261345] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x0
[ 1245.261348] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x300
[ 1245.261353] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 1245.261357] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 1245.261361] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xf
[ 1245.261366] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 1245.261371] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x7fac5f
[ 1245.261374] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x7fac5f
[ 1245.261379] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x0
[ 1245.261383] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x0
[ 1245.261388] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 1245.261392] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 1245.261396] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 1245.261401] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
[ 1245.261405] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 1245.261409] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 1245.261414] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 1245.261418] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 1245.261422] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 1245.261427] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 1245.261446] nvgpu: 57000000.gpu nvgpu_set_error_notifier_locked:137 [ERR] error notifier set to 8 for ch 501
[ 1245.272791] ---- mlocks ----

[ 1245.272811] ---- syncpts ----
[ 1245.272823] id 1 (disp0_a) min 640 max 640 refs 1 (previous client : )
[ 1245.272827] id 2 (disp0_b) min 4 max 4 refs 1 (previous client : )
[ 1245.272832] id 3 (disp0_c) min 4 max 4 refs 1 (previous client : )
[ 1245.272839] id 8 (gm20b_507) min 508 max 508 refs 1 (previous client : )
[ 1245.272843] id 9 (gm20b_506) min 28 max 28 refs 1 (previous client : )
[ 1245.272849] id 11 (gm20b_505) min 576 max 576 refs 1 (previous client : gm20b_505)
[ 1245.272854] id 12 (gm20b_504) min 6 max 6 refs 1 (previous client : )
[ 1245.272859] id 13 (gm20b_503) min 17620 max 17620 refs 1 (previous client : )
[ 1245.272863] id 14 (gm20b_502) min 14 max 14 refs 1 (previous client : )
[ 1245.272869] id 15 (gm20b_501) min 9438 max 9446 refs 1 (previous client : gm20b_501)
[ 1245.272924] id 26 (vblank0) min -6 max -6 refs 1 (previous client : )

[ 1245.273036] ---- channels ----
[ 1245.273042]
channel 0 - 54340000.vic

[ 1245.273047] 0-54340000.vic (0):
[ 1245.273050] inactive

[ 1245.273055]
---- host general irq ----

[ 1245.273058] sync_hintmask_ext = 0xc0000000
[ 1245.273061] sync_hintmask = 0x80000000
[ 1245.273064] sync_intc0mask = 0x00000001
[ 1245.273066] sync_intmask = 0x00000011
[ 1245.273068]
---- host syncpt irq mask ----

[ 1245.273071] syncpt_thresh_int_mask(0) = 0x40000001
[ 1245.273074] syncpt_thresh_int_mask(1) = 0x00000000
[ 1245.273077] syncpt_thresh_int_mask(2) = 0x00000000
[ 1245.273079] syncpt_thresh_int_mask(3) = 0x00000000
[ 1245.273082] syncpt_thresh_int_mask(4) = 0x00000000
[ 1245.273084] syncpt_thresh_int_mask(5) = 0x00000000
[ 1245.273087] syncpt_thresh_int_mask(6) = 0x00000000
[ 1245.273090] syncpt_thresh_int_mask(7) = 0x00000000
[ 1245.273092] syncpt_thresh_int_mask(8) = 0x00000000
[ 1245.273095] syncpt_thresh_int_mask(9) = 0x00000000
[ 1245.273097] syncpt_thresh_int_mask(10) = 0x00000000
[ 1245.273100] syncpt_thresh_int_mask(11) = 0x00000000
[ 1245.273102]
---- host syncpt irq status ----

[ 1245.273105] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1245.273107] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1245.273110] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1245.273112] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1245.273115] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1245.273118] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1245.273120]
---- host syncpt thresh ----

[ 1245.273123] syncpt_int_thresh_thresh_0(0) = 1
[ 1245.273133] syncpt_int_thresh_thresh_0(15) = 9440
[ 1245.273219] gm20b pbdma 0:
[ 1245.273223] id: 10 (tsg), next_id: 10 (tsg) chan status: invalid
[ 1245.273241] PBDMA_PUT: 0000001e0438350c PBDMA_GET: 0000001e043833cc GP_PUT: 0000034f GP_GET: 00000319 FETCH: 0000034f HEADER: 20020180
HDR: 20024060 SHADOW0: 04382e5c SHADOW1: 0006b01e

[ 1245.273247] gm20b eng 0:
[ 1245.273251] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid
[ 1245.273252] busy

[ 1245.273257] gm20b eng 1:
[ 1245.273260] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1245.273460] 501-gm20b, pid 7012, refs 11:
[ 1245.273464] channel status: not in use pending_acquire busy
[ 1245.273471] RAMFC : TOP: 8000001e043833cc PUT: 0000001e0438350c GET: 0000001e043833cc FETCH: 0000421e021c77d0
HEADER: 20020180 COUNT: 00110002
SYNCPOINT 00000000 00000f01 SEMAPHORE 0000001e 02380070 00000001 00081001

[ 1245.273477] 502-gm20b, pid 6532, refs 2:
[ 1245.273480] channel status: in use idle not busy
[ 1245.273486] RAMFC : TOP: 8000001f0000c0a8 PUT: 0000001f0000c0a8 GET: 0000001f0000c0a8 FETCH: 0000001f0000c0a8
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000e01 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1245.273492] 503-gm20b, pid 6532, refs 2:
[ 1245.273494] channel status: not in use idle not busy
[ 1245.273500] RAMFC : TOP: 8000001f00004d00 PUT: 0000001f00004d00 GET: 0000001f00004d00 FETCH: 0000001f00004d00
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000d01 SEMAPHORE 0000000d fc001000 019c2c3f 01100002

[ 1245.273505] 504-gm20b, pid 6155, refs 2:
[ 1245.273508] channel status: in use idle not busy
[ 1245.273513] RAMFC : TOP: 8000001f00068060 PUT: 0000001f00068060 GET: 0000001f00068060 FETCH: 0000001f00068060
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000c01 SEMAPHORE 0000001e 00dd0aa0 00000000 00000002

[ 1245.273519] 505-gm20b, pid 5607, refs 2:
[ 1245.273522] channel status: in use idle not busy
[ 1245.273527] RAMFC : TOP: 8000001f00002670 PUT: 0000001f00002670 GET: 0000001f00002670 FETCH: 0000001f00002670
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000b01 SEMAPHORE 0000001e 00890000 00000002 00000002

[ 1245.273533] 506-gm20b, pid 5185, refs 2:
[ 1245.273535] channel status: in use idle not busy
[ 1245.273541] RAMFC : TOP: 8000001f0000c150 PUT: 0000001f0000c150 GET: 0000001f0000c150 FETCH: 0000001f0000c150
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000901 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1245.273546] 507-gm20b, pid 5185, refs 2:
[ 1245.273549] channel status: in use idle not busy
[ 1245.273554] RAMFC : TOP: 8000001f00005fc0 PUT: 0000001f00005fc0 GET: 0000001f00005fc0 FETCH: 0000001f00005fc0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000801 SEMAPHORE 0000000d fc001000 03f4219e 01100002

[ 1245.273560] 508-gm20b, pid 4687, refs 2:
[ 1245.273563] channel status: in use idle not busy
[ 1245.273568] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1245.273576] 509-gm20b, pid 4687, refs 2:
[ 1245.273579] channel status: in use idle not busy
[ 1245.273584] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1245.273589] 510-gm20b, pid 4687, refs 2:
[ 1245.273592] channel status: in use idle not busy
[ 1245.273596] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1245.273602] 511-gm20b, pid 4687, refs 2:
[ 1245.273604] channel status: in use idle not busy
[ 1245.273609] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1250.518754] nvgpu: 57000000.gpu gk20a_channel_timeout_handler:1570 [ERR] Job on channel 501 timed out
[ 1250.529141] NV_PGRAPH_STATUS: 0x400000
[ 1250.529145] NV_PGRAPH_STATUS1: 0x0
[ 1250.529148] NV_PGRAPH_STATUS2: 0x0
[ 1250.529152] NV_PGRAPH_ENGINE_STATUS: 0x0
[ 1250.529155] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 1250.529158] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 1250.529162] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 1250.529165] NV_PGRAPH_EXCEPTION : 0x0
[ 1250.529168] NV_PGRAPH_FECS_INTR : 0x0
[ 1250.529171] NV_PFIFO_ENGINE_STATUS(GR) : 0x100a100a
[ 1250.529174] NV_PGRAPH_ACTIVITY0: 0x0
[ 1250.529177] NV_PGRAPH_ACTIVITY1: 0xc0000
[ 1250.529180] NV_PGRAPH_ACTIVITY2: 0x0
[ 1250.529184] NV_PGRAPH_ACTIVITY4: 0x0
[ 1250.529187] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 1250.529190] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 1250.529194] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 1250.529197] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 1250.529201] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 1250.529204] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1250.529208] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1250.529211] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 1250.529215] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 1250.529218] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 1250.529221] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 1250.529225] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1250.529228] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1250.529232] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 1250.529236] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 1250.529239] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 1250.529243] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 1250.529246] NV_PGRAPH_PRI_FE_GO_IDLE_ON_STATUS: 0x2e
[ 1250.529249] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x400
[ 1250.529253] NV_PGRAPH_PRI_FE_GO_IDLE_CHECK : 0x200
[ 1250.529256] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x33000700
[ 1250.529260] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 1250.529264] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 1250.529267] NV_PGRAPH_PRI_FE_TPC_FS: 0x0
[ 1250.529270] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 1250.529273] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 1250.529276] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x0
[ 1250.529279] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x300
[ 1250.529283] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 1250.529286] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 1250.529290] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xf
[ 1250.529294] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 1250.529298] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x7fac1f
[ 1250.529301] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x7fac1f
[ 1250.529304] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x0
[ 1250.529308] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x0
[ 1250.529311] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 1250.529315] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 1250.529318] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 1250.529322] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
[ 1250.529325] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 1250.529328] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 1250.529332] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 1250.529336] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 1250.529340] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 1250.529343] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 1250.529360] nvgpu: 57000000.gpu nvgpu_set_error_notifier_locked:137 [ERR] error notifier set to 8 for ch 501
[ 1250.540123] ---- mlocks ----

[ 1250.540139] ---- syncpts ----
[ 1250.540146] id 1 (disp0_a) min 640 max 640 refs 1 (previous client : )
[ 1250.540149] id 2 (disp0_b) min 4 max 4 refs 1 (previous client : )
[ 1250.540152] id 3 (disp0_c) min 4 max 4 refs 1 (previous client : )
[ 1250.540158] id 8 (gm20b_507) min 508 max 508 refs 1 (previous client : )
[ 1250.540161] id 9 (gm20b_506) min 28 max 28 refs 1 (previous client : )
[ 1250.540166] id 11 (gm20b_505) min 576 max 576 refs 1 (previous client : gm20b_505)
[ 1250.540169] id 12 (gm20b_504) min 6 max 6 refs 1 (previous client : )
[ 1250.540173] id 13 (gm20b_503) min 17620 max 17620 refs 1 (previous client : )
[ 1250.540176] id 14 (gm20b_502) min 14 max 14 refs 1 (previous client : )
[ 1250.540180] id 15 (gm20b_501) min 9466 max 9474 refs 1 (previous client : gm20b_501)
[ 1250.540190] id 26 (vblank0) min -6 max -6 refs 1 (previous client : )

[ 1250.540285] ---- channels ----
[ 1250.540290]
channel 0 - 54340000.vic

[ 1250.540296] 0-54340000.vic (0):
[ 1250.540298] inactive

[ 1250.540303]
---- host general irq ----

[ 1250.540308] sync_hintmask_ext = 0xc0000000
[ 1250.540310] sync_hintmask = 0x80000000
[ 1250.540313] sync_intc0mask = 0x00000001
[ 1250.540315] sync_intmask = 0x00000011
[ 1250.540317]
---- host syncpt irq mask ----

[ 1250.540320] syncpt_thresh_int_mask(0) = 0x40000001
[ 1250.540323] syncpt_thresh_int_mask(1) = 0x00000000
[ 1250.540325] syncpt_thresh_int_mask(2) = 0x00000000
[ 1250.540328] syncpt_thresh_int_mask(3) = 0x00000000
[ 1250.540330] syncpt_thresh_int_mask(4) = 0x00000000
[ 1250.540333] syncpt_thresh_int_mask(5) = 0x00000000
[ 1250.540335] syncpt_thresh_int_mask(6) = 0x00000000
[ 1250.540337] syncpt_thresh_int_mask(7) = 0x00000000
[ 1250.540340] syncpt_thresh_int_mask(8) = 0x00000000
[ 1250.540342] syncpt_thresh_int_mask(9) = 0x00000000
[ 1250.540345] syncpt_thresh_int_mask(10) = 0x00000000
[ 1250.540347] syncpt_thresh_int_mask(11) = 0x00000000
[ 1250.540349]
---- host syncpt irq status ----

[ 1250.540352] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1250.540355] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1250.540357] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1250.540360] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1250.540362] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1250.540365] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1250.540366]
---- host syncpt thresh ----

[ 1250.540369] syncpt_int_thresh_thresh_0(0) = 1
[ 1250.540379] syncpt_int_thresh_thresh_0(15) = 9468
[ 1250.540465] gm20b pbdma 0:
[ 1250.540469] id: 10 (tsg), next_id: 10 (tsg) chan status: valid
[ 1250.540482] PBDMA_PUT: 0000001e0432350c PBDMA_GET: 0000001e043233d0 GP_PUT: 0000034f GP_GET: 00000319 FETCH: 0000034f HEADER: 20020180
HDR: 20024060 SHADOW0: 04322e5c SHADOW1: 0006b01e

[ 1250.540488] gm20b eng 0:
[ 1250.540491] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1250.540495] gm20b eng 1:
[ 1250.540498] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1250.540687] 501-gm20b, pid 7012, refs 11:
[ 1250.540691] channel status: not in use pending_acquire busy
[ 1250.540697] RAMFC : TOP: 8000001e043233d0 PUT: 0000001e0432350c GET: 0000001e043233d0 FETCH: 0000421e021877d0
HEADER: 20020180 COUNT: 01110002
SYNCPOINT 00000000 00000f01 SEMAPHORE 0000001e 02380090 00000001 00081001

[ 1250.540704] 502-gm20b, pid 6532, refs 2:
[ 1250.540707] channel status: in use idle not busy
[ 1250.540712] RAMFC : TOP: 8000001f0000c0a8 PUT: 0000001f0000c0a8 GET: 0000001f0000c0a8 FETCH: 0000001f0000c0a8
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000e01 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1250.540718] 503-gm20b, pid 6532, refs 2:
[ 1250.540721] channel status: not in use idle not busy
[ 1250.540726] RAMFC : TOP: 8000001f00004d00 PUT: 0000001f00004d00 GET: 0000001f00004d00 FETCH: 0000001f00004d00
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000d01 SEMAPHORE 0000000d fc001000 019c2c3f 01100002

[ 1250.540731] 504-gm20b, pid 6155, refs 2:
[ 1250.540734] channel status: in use idle not busy
[ 1250.540739] RAMFC : TOP: 8000001f00068060 PUT: 0000001f00068060 GET: 0000001f00068060 FETCH: 0000001f00068060
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000c01 SEMAPHORE 0000001e 00dd0aa0 00000000 00000002

[ 1250.540744] 505-gm20b, pid 5607, refs 2:
[ 1250.540747] channel status: in use idle not busy
[ 1250.540752] RAMFC : TOP: 8000001f00002670 PUT: 0000001f00002670 GET: 0000001f00002670 FETCH: 0000001f00002670
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000b01 SEMAPHORE 0000001e 00890000 00000002 00000002

[ 1250.540757] 506-gm20b, pid 5185, refs 2:
[ 1250.540760] channel status: in use idle not busy
[ 1250.540765] RAMFC : TOP: 8000001f0000c150 PUT: 0000001f0000c150 GET: 0000001f0000c150 FETCH: 0000001f0000c150
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000901 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1250.540772] 507-gm20b, pid 5185, refs 2:
[ 1250.540774] channel status: in use idle not busy
[ 1250.540779] RAMFC : TOP: 8000001f00005fc0 PUT: 0000001f00005fc0 GET: 0000001f00005fc0 FETCH: 0000001f00005fc0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000801 SEMAPHORE 0000000d fc001000 03f4219e 01100002

[ 1250.540784] 508-gm20b, pid 4687, refs 2:
[ 1250.540787] channel status: in use idle not busy
[ 1250.540792] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1250.540797] 509-gm20b, pid 4687, refs 2:
[ 1250.540800] channel status: in use idle not busy
[ 1250.540804] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1250.540810] 510-gm20b, pid 4687, refs 2:
[ 1250.540812] channel status: in use idle not busy
[ 1250.540817] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1250.540822] 511-gm20b, pid 4687, refs 2:
[ 1250.540824] channel status: in use idle not busy
[ 1250.540829] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1255.770771] nvgpu: 57000000.gpu gk20a_channel_timeout_handler:1570 [ERR] Job on channel 501 timed out
[ 1255.780795] NV_PGRAPH_STATUS: 0x400000
[ 1255.780798] NV_PGRAPH_STATUS1: 0x0
[ 1255.780802] NV_PGRAPH_STATUS2: 0x0
[ 1255.780805] NV_PGRAPH_ENGINE_STATUS: 0x0
[ 1255.780809] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 1255.780812] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 1255.780816] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 1255.780819] NV_PGRAPH_EXCEPTION : 0x0
[ 1255.780822] NV_PGRAPH_FECS_INTR : 0x0
[ 1255.780825] NV_PFIFO_ENGINE_STATUS(GR) : 0x100a100a
[ 1255.780828] NV_PGRAPH_ACTIVITY0: 0x0
[ 1255.780831] NV_PGRAPH_ACTIVITY1: 0xc0000
[ 1255.780834] NV_PGRAPH_ACTIVITY2: 0x0
[ 1255.780837] NV_PGRAPH_ACTIVITY4: 0x0
[ 1255.780840] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 1255.780844] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 1255.780847] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 1255.780851] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 1255.780854] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 1255.780858] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1255.780862] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1255.780865] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 1255.780868] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 1255.780872] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 1255.780875] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 1255.780878] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1255.780882] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1255.780886] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 1255.780889] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 1255.780893] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 1255.780897] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 1255.780900] NV_PGRAPH_PRI_FE_GO_IDLE_ON_STATUS: 0x2e
[ 1255.780903] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x400
[ 1255.780906] NV_PGRAPH_PRI_FE_GO_IDLE_CHECK : 0x200
[ 1255.780910] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x33000700
[ 1255.780914] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 1255.780917] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 1255.780921] NV_PGRAPH_PRI_FE_TPC_FS: 0x0
[ 1255.780924] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 1255.780927] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 1255.780930] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x0
[ 1255.780933] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x300
[ 1255.780937] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 1255.780940] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 1255.780944] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xf
[ 1255.780947] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 1255.780951] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x7f7f2f
[ 1255.780954] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x7f7f2f
[ 1255.780958] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x0
[ 1255.780962] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x0
[ 1255.780965] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 1255.780969] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 1255.780972] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 1255.780976] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
[ 1255.780979] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 1255.780982] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 1255.780986] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 1255.780990] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 1255.780993] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 1255.780997] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 1255.781014] nvgpu: 57000000.gpu nvgpu_set_error_notifier_locked:137 [ERR] error notifier set to 8 for ch 501
[ 1255.791912] ---- mlocks ----

[ 1255.791925] ---- syncpts ----
[ 1255.791932] id 1 (disp0_a) min 640 max 640 refs 1 (previous client : )
[ 1255.791936] id 2 (disp0_b) min 4 max 4 refs 1 (previous client : )
[ 1255.791939] id 3 (disp0_c) min 4 max 4 refs 1 (previous client : )
[ 1255.791945] id 8 (gm20b_507) min 508 max 508 refs 1 (previous client : )
[ 1255.791949] id 9 (gm20b_506) min 28 max 28 refs 1 (previous client : )
[ 1255.791953] id 11 (gm20b_505) min 576 max 576 refs 1 (previous client : gm20b_505)
[ 1255.791957] id 12 (gm20b_504) min 6 max 6 refs 1 (previous client : )
[ 1255.791960] id 13 (gm20b_503) min 17620 max 17620 refs 1 (previous client : )
[ 1255.791964] id 14 (gm20b_502) min 14 max 14 refs 1 (previous client : )
[ 1255.791967] id 15 (gm20b_501) min 9494 max 9504 refs 1 (previous client : gm20b_501)
[ 1255.791977] id 26 (vblank0) min -6 max -6 refs 1 (previous client : )

[ 1255.792077] ---- channels ----
[ 1255.792083]
channel 0 - 54340000.vic

[ 1255.792088] 0-54340000.vic (0):
[ 1255.792091] inactive

[ 1255.792096]
---- host general irq ----

[ 1255.792099] sync_hintmask_ext = 0xc0000000
[ 1255.792101] sync_hintmask = 0x80000000
[ 1255.792104] sync_intc0mask = 0x00000001
[ 1255.792106] sync_intmask = 0x00000011
[ 1255.792108]
---- host syncpt irq mask ----

[ 1255.792111] syncpt_thresh_int_mask(0) = 0x40000001
[ 1255.792114] syncpt_thresh_int_mask(1) = 0x00000000
[ 1255.792116] syncpt_thresh_int_mask(2) = 0x00000000
[ 1255.792119] syncpt_thresh_int_mask(3) = 0x00000000
[ 1255.792121] syncpt_thresh_int_mask(4) = 0x00000000
[ 1255.792124] syncpt_thresh_int_mask(5) = 0x00000000
[ 1255.792126] syncpt_thresh_int_mask(6) = 0x00000000
[ 1255.792129] syncpt_thresh_int_mask(7) = 0x00000000
[ 1255.792131] syncpt_thresh_int_mask(8) = 0x00000000
[ 1255.792134] syncpt_thresh_int_mask(9) = 0x00000000
[ 1255.792136] syncpt_thresh_int_mask(10) = 0x00000000
[ 1255.792139] syncpt_thresh_int_mask(11) = 0x00000000
[ 1255.792141]
---- host syncpt irq status ----

[ 1255.792144] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1255.792146] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1255.792149] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1255.792151] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1255.792154] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1255.792156] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1255.792160]
---- host syncpt thresh ----

[ 1255.792163] syncpt_int_thresh_thresh_0(0) = 1
[ 1255.792173] syncpt_int_thresh_thresh_0(15) = 9496
[ 1255.792258] gm20b pbdma 0:
[ 1255.792262] id: 10 (tsg), next_id: 10 (tsg) chan status: valid
[ 1255.792275] PBDMA_PUT: 0000001e042c350c PBDMA_GET: 0000001e042c33cc GP_PUT: 0000038d GP_GET: 00000319 FETCH: 00000370 HEADER: 20020180
HDR: 20024060 SHADOW0: 042c2e5c SHADOW1: 0006b01e

[ 1255.792281] gm20b eng 0:
[ 1255.792284] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1255.792289] gm20b eng 1:
[ 1255.792292] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1255.792481] 501-gm20b, pid 7012, refs 13:
[ 1255.792484] channel status: not in use pending_acquire busy
[ 1255.792491] RAMFC : TOP: 8000001e042c33cc PUT: 0000001e042c350c GET: 0000001e042c33cc FETCH: 0000421e032c77d0
HEADER: 20020180 COUNT: 00110002
SYNCPOINT 00000000 00000f01 SEMAPHORE 0000001e 023800b0 00000001 00081001

[ 1255.792497] 502-gm20b, pid 6532, refs 2:
[ 1255.792501] channel status: in use idle not busy
[ 1255.792507] RAMFC : TOP: 8000001f0000c0a8 PUT: 0000001f0000c0a8 GET: 0000001f0000c0a8 FETCH: 0000001f0000c0a8
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000e01 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1255.792512] 503-gm20b, pid 6532, refs 2:
[ 1255.792515] channel status: not in use idle not busy
[ 1255.792520] RAMFC : TOP: 8000001f00004d00 PUT: 0000001f00004d00 GET: 0000001f00004d00 FETCH: 0000001f00004d00
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000d01 SEMAPHORE 0000000d fc001000 019c2c3f 01100002

[ 1255.792525] 504-gm20b, pid 6155, refs 2:
[ 1255.792528] channel status: in use idle not busy
[ 1255.792533] RAMFC : TOP: 8000001f00068060 PUT: 0000001f00068060 GET: 0000001f00068060 FETCH: 0000001f00068060
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000c01 SEMAPHORE 0000001e 00dd0aa0 00000000 00000002

[ 1255.792538] 505-gm20b, pid 5607, refs 2:
[ 1255.792541] channel status: in use idle not busy
[ 1255.792546] RAMFC : TOP: 8000001f00002670 PUT: 0000001f00002670 GET: 0000001f00002670 FETCH: 0000001f00002670
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000b01 SEMAPHORE 0000001e 00890000 00000002 00000002

[ 1255.792551] 506-gm20b, pid 5185, refs 2:
[ 1255.792553] channel status: in use idle not busy
[ 1255.792558] RAMFC : TOP: 8000001f0000c150 PUT: 0000001f0000c150 GET: 0000001f0000c150 FETCH: 0000001f0000c150
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000901 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1255.792563] 507-gm20b, pid 5185, refs 2:
[ 1255.792566] channel status: in use idle not busy
[ 1255.792571] RAMFC : TOP: 8000001f00005fc0 PUT: 0000001f00005fc0 GET: 0000001f00005fc0 FETCH: 0000001f00005fc0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000801 SEMAPHORE 0000000d fc001000 03f4219e 01100002

[ 1255.792576] 508-gm20b, pid 4687, refs 2:
[ 1255.792579] channel status: in use idle not busy
[ 1255.792585] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1255.792590] 509-gm20b, pid 4687, refs 2:
[ 1255.792592] channel status: in use idle not busy
[ 1255.792597] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1255.792602] 510-gm20b, pid 4687, refs 2:
[ 1255.792604] channel status: in use idle not busy
[ 1255.792609] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1255.792614] 511-gm20b, pid 4687, refs 2:
[ 1255.792616] channel status: in use idle not busy
[ 1255.792621] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1261.022797] nvgpu: 57000000.gpu gk20a_channel_timeout_handler:1570 [ERR] Job on channel 501 timed out
[ 1261.032822] NV_PGRAPH_STATUS: 0x400000
[ 1261.032825] NV_PGRAPH_STATUS1: 0x0
[ 1261.032829] NV_PGRAPH_STATUS2: 0x0
[ 1261.032832] NV_PGRAPH_ENGINE_STATUS: 0x0
[ 1261.032835] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 1261.032839] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 1261.032842] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 1261.032845] NV_PGRAPH_EXCEPTION : 0x0
[ 1261.032849] NV_PGRAPH_FECS_INTR : 0x0
[ 1261.032852] NV_PFIFO_ENGINE_STATUS(GR) : 0x100a100a
[ 1261.032855] NV_PGRAPH_ACTIVITY0: 0x0
[ 1261.032859] NV_PGRAPH_ACTIVITY1: 0xc0000
[ 1261.032862] NV_PGRAPH_ACTIVITY2: 0x0
[ 1261.032865] NV_PGRAPH_ACTIVITY4: 0x0
[ 1261.032868] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 1261.032872] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 1261.032875] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 1261.032878] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 1261.032882] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 1261.032885] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1261.032889] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1261.032892] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 1261.032895] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 1261.032899] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 1261.032902] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 1261.032906] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1261.032909] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1261.032913] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 1261.032917] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 1261.032920] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 1261.032924] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 1261.032927] NV_PGRAPH_PRI_FE_GO_IDLE_ON_STATUS: 0x2e
[ 1261.032930] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x400
[ 1261.032933] NV_PGRAPH_PRI_FE_GO_IDLE_CHECK : 0x200
[ 1261.032937] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x33000700
[ 1261.032940] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 1261.032944] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 1261.032947] NV_PGRAPH_PRI_FE_TPC_FS: 0x0
[ 1261.032950] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 1261.032954] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 1261.032957] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x0
[ 1261.032960] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x300
[ 1261.032963] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 1261.032967] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 1261.032970] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xf
[ 1261.032974] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 1261.032978] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x7fac5f
[ 1261.032981] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x7fac5f
[ 1261.032985] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x0
[ 1261.032988] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x0
[ 1261.032992] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 1261.032995] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 1261.032999] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 1261.033002] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
[ 1261.033005] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 1261.033009] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 1261.033013] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 1261.033016] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 1261.033020] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 1261.033023] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 1261.033040] nvgpu: 57000000.gpu nvgpu_set_error_notifier_locked:137 [ERR] error notifier set to 8 for ch 501
[ 1261.043797] ---- mlocks ----

[ 1261.043810] ---- syncpts ----
[ 1261.043817] id 1 (disp0_a) min 640 max 640 refs 1 (previous client : )
[ 1261.043821] id 2 (disp0_b) min 4 max 4 refs 1 (previous client : )
[ 1261.043824] id 3 (disp0_c) min 4 max 4 refs 1 (previous client : )
[ 1261.043831] id 8 (gm20b_507) min 508 max 508 refs 1 (previous client : )
[ 1261.043834] id 9 (gm20b_506) min 28 max 28 refs 1 (previous client : )
[ 1261.043842] id 11 (gm20b_505) min 576 max 576 refs 1 (previous client : gm20b_505)
[ 1261.043848] id 12 (gm20b_504) min 6 max 6 refs 1 (previous client : )
[ 1261.043853] id 13 (gm20b_503) min 17620 max 17620 refs 1 (previous client : )
[ 1261.043857] id 14 (gm20b_502) min 14 max 14 refs 1 (previous client : )
[ 1261.043862] id 15 (gm20b_501) min 9524 max 9532 refs 1 (previous client : gm20b_501)
[ 1261.043873] id 26 (vblank0) min -6 max -6 refs 1 (previous client : )

[ 1261.043973] ---- channels ----
[ 1261.043979]
channel 0 - 54340000.vic

[ 1261.043985] 0-54340000.vic (0):
[ 1261.043987] inactive

[ 1261.043992]
---- host general irq ----

[ 1261.043995] sync_hintmask_ext = 0xc0000000
[ 1261.043997] sync_hintmask = 0x80000000
[ 1261.044000] sync_intc0mask = 0x00000001
[ 1261.044002] sync_intmask = 0x00000011
[ 1261.044004]
---- host syncpt irq mask ----

[ 1261.044007] syncpt_thresh_int_mask(0) = 0x40000001
[ 1261.044009] syncpt_thresh_int_mask(1) = 0x00000000
[ 1261.044012] syncpt_thresh_int_mask(2) = 0x00000000
[ 1261.044014] syncpt_thresh_int_mask(3) = 0x00000000
[ 1261.044017] syncpt_thresh_int_mask(4) = 0x00000000
[ 1261.044019] syncpt_thresh_int_mask(5) = 0x00000000
[ 1261.044022] syncpt_thresh_int_mask(6) = 0x00000000
[ 1261.044024] syncpt_thresh_int_mask(7) = 0x00000000
[ 1261.044027] syncpt_thresh_int_mask(8) = 0x00000000
[ 1261.044029] syncpt_thresh_int_mask(9) = 0x00000000
[ 1261.044032] syncpt_thresh_int_mask(10) = 0x00000000
[ 1261.044034] syncpt_thresh_int_mask(11) = 0x00000000
[ 1261.044036]
---- host syncpt irq status ----

[ 1261.044039] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1261.044041] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1261.044044] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1261.044046] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1261.044049] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1261.044051] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1261.044053]
---- host syncpt thresh ----

[ 1261.044056] syncpt_int_thresh_thresh_0(0) = 1
[ 1261.044067] syncpt_int_thresh_thresh_0(15) = 9526
[ 1261.044153] gm20b pbdma 0:
[ 1261.044157] id: 10 (tsg), next_id: 10 (tsg) chan status: valid
[ 1261.044170] PBDMA_PUT: 0000001e0424350c PBDMA_GET: 0000001e042433cc GP_PUT: 0000034f GP_GET: 00000319 FETCH: 0000034f HEADER: 20020180
HDR: 20024060 SHADOW0: 04242e5c SHADOW1: 0006b01e

[ 1261.044176] gm20b eng 0:
[ 1261.044179] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1261.044183] gm20b eng 1:
[ 1261.044186] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1261.044374] 501-gm20b, pid 7012, refs 11:
[ 1261.044377] channel status: not in use pending_acquire busy
[ 1261.044384] RAMFC : TOP: 8000001e042433cc PUT: 0000001e0424350c GET: 0000001e042433cc FETCH: 0000421e021c77d0
HEADER: 20020180 COUNT: 00110002
SYNCPOINT 00000000 00000f01 SEMAPHORE 0000001e 023800d0 00000001 00081001

[ 1261.044390] 502-gm20b, pid 6532, refs 2:
[ 1261.044393] channel status: in use idle not busy
[ 1261.044399] RAMFC : TOP: 8000001f0000c0a8 PUT: 0000001f0000c0a8 GET: 0000001f0000c0a8 FETCH: 0000001f0000c0a8
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000e01 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1261.044405] 503-gm20b, pid 6532, refs 2:
[ 1261.044407] channel status: not in use idle not busy
[ 1261.044413] RAMFC : TOP: 8000001f00004d00 PUT: 0000001f00004d00 GET: 0000001f00004d00 FETCH: 0000001f00004d00
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000d01 SEMAPHORE 0000000d fc001000 019c2c3f 01100002

[ 1261.044418] 504-gm20b, pid 6155, refs 2:
[ 1261.044421] channel status: in use idle not busy
[ 1261.044426] RAMFC : TOP: 8000001f00068060 PUT: 0000001f00068060 GET: 0000001f00068060 FETCH: 0000001f00068060
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000c01 SEMAPHORE 0000001e 00dd0aa0 00000000 00000002

[ 1261.044432] 505-gm20b, pid 5607, refs 2:
[ 1261.044434] channel status: in use idle not busy
[ 1261.044439] RAMFC : TOP: 8000001f00002670 PUT: 0000001f00002670 GET: 0000001f00002670 FETCH: 0000001f00002670
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000b01 SEMAPHORE 0000001e 00890000 00000002 00000002

[ 1261.044445] 506-gm20b, pid 5185, refs 2:
[ 1261.044447] channel status: in use idle not busy
[ 1261.044452] RAMFC : TOP: 8000001f0000c150 PUT: 0000001f0000c150 GET: 0000001f0000c150 FETCH: 0000001f0000c150
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000901 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1261.044458] 507-gm20b, pid 5185, refs 2:
[ 1261.044460] channel status: in use idle not busy
[ 1261.044465] RAMFC : TOP: 8000001f00005fc0 PUT: 0000001f00005fc0 GET: 0000001f00005fc0 FETCH: 0000001f00005fc0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000801 SEMAPHORE 0000000d fc001000 03f4219e 01100002

[ 1261.044471] 508-gm20b, pid 4687, refs 2:
[ 1261.044473] channel status: in use idle not busy
[ 1261.044478] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1261.044484] 509-gm20b, pid 4687, refs 2:
[ 1261.044486] channel status: in use idle not busy
[ 1261.044491] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1261.044498] 510-gm20b, pid 4687, refs 2:
[ 1261.044500] channel status: in use idle not busy
[ 1261.044505] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1261.044510] 511-gm20b, pid 4687, refs 2:
[ 1261.044512] channel status: in use idle not busy
[ 1261.044517] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1266.690754] nvgpu: 57000000.gpu gk20a_channel_timeout_handler:1570 [ERR] Job on channel 501 timed out
[ 1266.700800] NV_PGRAPH_STATUS: 0x400000
[ 1266.700805] NV_PGRAPH_STATUS1: 0x0
[ 1266.700808] NV_PGRAPH_STATUS2: 0x0
[ 1266.700812] NV_PGRAPH_ENGINE_STATUS: 0x0
[ 1266.700815] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 1266.700819] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 1266.700822] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 1266.700825] NV_PGRAPH_EXCEPTION : 0x0
[ 1266.700828] NV_PGRAPH_FECS_INTR : 0x0
[ 1266.700832] NV_PFIFO_ENGINE_STATUS(GR) : 0x100a100a
[ 1266.700835] NV_PGRAPH_ACTIVITY0: 0x0
[ 1266.700838] NV_PGRAPH_ACTIVITY1: 0xc0000
[ 1266.700841] NV_PGRAPH_ACTIVITY2: 0x0
[ 1266.700844] NV_PGRAPH_ACTIVITY4: 0x0
[ 1266.700847] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 1266.700851] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 1266.700854] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 1266.700858] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 1266.700861] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 1266.700865] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1266.700868] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1266.700871] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 1266.700875] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 1266.700878] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 1266.700881] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 1266.700885] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1266.700888] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1266.700892] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 1266.700896] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 1266.700899] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 1266.700903] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 1266.700906] NV_PGRAPH_PRI_FE_GO_IDLE_ON_STATUS: 0x2e
[ 1266.700909] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x400
[ 1266.700912] NV_PGRAPH_PRI_FE_GO_IDLE_CHECK : 0x200
[ 1266.700916] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x33000700
[ 1266.700920] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 1266.700923] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 1266.700927] NV_PGRAPH_PRI_FE_TPC_FS: 0x0
[ 1266.700930] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 1266.700933] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 1266.700936] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x0
[ 1266.700939] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x300
[ 1266.700943] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 1266.700946] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 1266.700950] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xf
[ 1266.700953] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 1266.700957] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x7f7f17
[ 1266.700960] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x7f7f17
[ 1266.700964] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x0
[ 1266.700967] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x0
[ 1266.700971] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 1266.700975] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 1266.700978] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 1266.700982] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
00000000 00000000

[ 1278.053206] 509-gm20b, pid 4687, refs 2:
[ 1278.053209] channel status: in use idle not busy
[ 1278.053214] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1278.053219] 510-gm20b, pid 4687, refs 2:
[ 1278.053222] channel status: in use idle not busy
[ 1278.053227] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1278.053233] 511-gm20b, pid 4687, refs 2:
[ 1278.053236] channel status: in use idle not busy
[ 1278.053241] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1283.698757] nvgpu: 57000000.gpu gk20a_channel_timeout_handler:1570 [ERR] Job on channel 501 timed out
[ 1283.709126] NV_PGRAPH_STATUS: 0x400000
[ 1283.709131] NV_PGRAPH_STATUS1: 0x0
[ 1283.709135] NV_PGRAPH_STATUS2: 0x0
[ 1283.709138] NV_PGRAPH_ENGINE_STATUS: 0x0
[ 1283.709142] NV_PGRAPH_GRFIFO_STATUS : 0x1
[ 1283.709146] NV_PGRAPH_GRFIFO_CONTROL : 0x10001
[ 1283.709149] NV_PGRAPH_PRI_FECS_HOST_INT_STATUS : 0x0
[ 1283.709153] NV_PGRAPH_EXCEPTION : 0x0
[ 1283.709157] NV_PGRAPH_FECS_INTR : 0x0
[ 1283.709160] NV_PFIFO_ENGINE_STATUS(GR) : 0x100a100a
[ 1283.709163] NV_PGRAPH_ACTIVITY0: 0x0
[ 1283.709167] NV_PGRAPH_ACTIVITY1: 0xc0000
[ 1283.709171] NV_PGRAPH_ACTIVITY2: 0x0
[ 1283.709174] NV_PGRAPH_ACTIVITY4: 0x0
[ 1283.709178] NV_PGRAPH_PRI_SKED_ACTIVITY: 0x0
[ 1283.709182] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY0: 0x0
[ 1283.709187] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY1: 0x0
[ 1283.709190] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY2: 0x0
[ 1283.709194] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_ACTIVITY3: 0x0
[ 1283.709199] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1283.709203] NV_PGRAPH_PRI_GPC0_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1283.709207] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY0: 0x0
[ 1283.709211] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY1: 0x0
[ 1283.709215] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY2: 0x0
[ 1283.709219] NV_PGRAPH_PRI_GPCS_GPCCS_GPC_ACTIVITY3: 0x0
[ 1283.709223] NV_PGRAPH_PRI_GPCS_TPC0_TPCCS_TPC_ACTIVITY0: 0x0
[ 1283.709228] NV_PGRAPH_PRI_GPCS_TPCS_TPCCS_TPC_ACTIVITY0: 0x0
[ 1283.709233] NV_PGRAPH_PRI_BE0_BECS_BE_ACTIVITY0: 0x0
[ 1283.709237] NV_PGRAPH_PRI_BE1_BECS_BE_ACTIVITY0: 0x0
[ 1283.709241] NV_PGRAPH_PRI_BES_BECS_BE_ACTIVITY0: 0x0
[ 1283.709246] NV_PGRAPH_PRI_DS_MPIPE_STATUS: 0x0
[ 1283.709250] NV_PGRAPH_PRI_FE_GO_IDLE_ON_STATUS: 0x2e
[ 1283.709253] NV_PGRAPH_PRI_FE_GO_IDLE_TIMEOUT : 0x400
[ 1283.709257] NV_PGRAPH_PRI_FE_GO_IDLE_CHECK : 0x200
[ 1283.709261] NV_PGRAPH_PRI_FE_GO_IDLE_INFO : 0x33000700
[ 1283.709266] NV_PGRAPH_PRI_GPC0_TPC0_TEX_M_TEX_SUBUNITS_STATUS: 0x0
[ 1283.709271] NV_PGRAPH_PRI_CWD_FS: 0x0
[ 1283.709274] NV_PGRAPH_PRI_FE_TPC_FS: 0x0
[ 1283.709278] NV_PGRAPH_PRI_CWD_GPC_TPC_ID(0): 0x0
[ 1283.709282] NV_PGRAPH_PRI_CWD_SM_ID(0): 0x0
[ 1283.709286] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_FE_0: 0x0
[ 1283.709289] NV_PGRAPH_PRI_FECS_CTXSW_STATUS_1: 0x300
[ 1283.709294] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_GPC_0: 0x0
[ 1283.709297] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_STATUS_1: 0x300
[ 1283.709302] NV_PGRAPH_PRI_FECS_CTXSW_IDLESTATE : 0xf
[ 1283.709306] NV_PGRAPH_PRI_GPC0_GPCCS_CTXSW_IDLESTATE : 0xf
[ 1283.709311] NV_PGRAPH_PRI_FECS_CURRENT_CTX : 0x7f791f
[ 1283.709314] NV_PGRAPH_PRI_FECS_NEW_CTX : 0x7f791f
[ 1283.709319] NV_PGRAPH_PRI_BE0_CROP_STATUS1 : 0x0
[ 1283.709324] NV_PGRAPH_PRI_BES_CROP_STATUS1 : 0x0
[ 1283.709328] NV_PGRAPH_PRI_BE0_ZROP_STATUS : 0x0
[ 1283.709333] NV_PGRAPH_PRI_BE0_ZROP_STATUS2 : 0x0
[ 1283.709337] NV_PGRAPH_PRI_BES_ZROP_STATUS : 0x0
[ 1283.709341] NV_PGRAPH_PRI_BES_ZROP_STATUS2 : 0x0
[ 1283.709345] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION: 0x0
[ 1283.709349] NV_PGRAPH_PRI_BE0_BECS_BE_EXCEPTION_EN: 0x0
[ 1283.709354] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION: 0x0
[ 1283.709359] NV_PGRAPH_PRI_GPC0_GPCCS_GPC_EXCEPTION_EN: 0x30000
[ 1283.709363] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION: 0x0
[ 1283.709367] NV_PGRAPH_PRI_GPC0_TPC0_TPCCS_TPC_EXCEPTION_EN: 0x3
[ 1283.709391] nvgpu: 57000000.gpu nvgpu_set_error_notifier_locked:137 [ERR] error notifier set to 8 for ch 501
[ 1283.720664] ---- mlocks ----

[ 1283.720677] ---- syncpts ----
[ 1283.720684] id 1 (disp0_a) min 640 max 640 refs 1 (previous client : )
[ 1283.720688] id 2 (disp0_b) min 4 max 4 refs 1 (previous client : )
[ 1283.720691] id 3 (disp0_c) min 4 max 4 refs 1 (previous client : )
[ 1283.720697] id 8 (gm20b_507) min 508 max 508 refs 1 (previous client : )
[ 1283.720701] id 9 (gm20b_506) min 28 max 28 refs 1 (previous client : )
[ 1283.720705] id 11 (gm20b_505) min 576 max 576 refs 1 (previous client : gm20b_505)
[ 1283.720709] id 12 (gm20b_504) min 6 max 6 refs 1 (previous client : )
[ 1283.720713] id 13 (gm20b_503) min 17620 max 17620 refs 1 (previous client : )
[ 1283.720716] id 14 (gm20b_502) min 14 max 14 refs 1 (previous client : )
[ 1283.720720] id 15 (gm20b_501) min 9638 max 9646 refs 1 (previous client : gm20b_501)
[ 1283.720730] id 26 (vblank0) min -6 max -6 refs 1 (previous client : )

[ 1283.720826] ---- channels ----
[ 1283.720831]
channel 0 - 54340000.vic

[ 1283.720837] 0-54340000.vic (0):
[ 1283.720839] inactive

[ 1283.720844]
---- host general irq ----

[ 1283.720847] sync_hintmask_ext = 0xc0000000
[ 1283.720849] sync_hintmask = 0x80000000
[ 1283.720852] sync_intc0mask = 0x00000001
[ 1283.720854] sync_intmask = 0x00000011
[ 1283.720856]
---- host syncpt irq mask ----

[ 1283.720859] syncpt_thresh_int_mask(0) = 0x40000001
[ 1283.720862] syncpt_thresh_int_mask(1) = 0x00000000
[ 1283.720864] syncpt_thresh_int_mask(2) = 0x00000000
[ 1283.720867] syncpt_thresh_int_mask(3) = 0x00000000
[ 1283.720870] syncpt_thresh_int_mask(4) = 0x00000000
[ 1283.720872] syncpt_thresh_int_mask(5) = 0x00000000
[ 1283.720874] syncpt_thresh_int_mask(6) = 0x00000000
[ 1283.720877] syncpt_thresh_int_mask(7) = 0x00000000
[ 1283.720879] syncpt_thresh_int_mask(8) = 0x00000000
[ 1283.720882] syncpt_thresh_int_mask(9) = 0x00000000
[ 1283.720884] syncpt_thresh_int_mask(10) = 0x00000000
[ 1283.720887] syncpt_thresh_int_mask(11) = 0x00000000
[ 1283.720889]
---- host syncpt irq status ----

[ 1283.720892] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1283.720894] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1283.720897] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1283.720899] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1283.720902] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1283.720905] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1283.720907]
---- host syncpt thresh ----

[ 1283.720910] syncpt_int_thresh_thresh_0(0) = 1
[ 1283.720920] syncpt_int_thresh_thresh_0(15) = 9640
[ 1283.721005] gm20b pbdma 0:
[ 1283.721008] id: 10 (tsg), next_id: 10 (tsg) chan status: valid
[ 1283.721023] PBDMA_PUT: 0000001e0432350c PBDMA_GET: 0000001e043233cc GP_PUT: 0000034f GP_GET: 00000319 FETCH: 0000034f HEADER: 20020180
HDR: 20024060 SHADOW0: 04322e5c SHADOW1: 0006b01e

[ 1283.721028] gm20b eng 0:
[ 1283.721031] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1283.721036] gm20b eng 1:
[ 1283.721039] id: 10 (tsg), next_id: 10 (tsg), ctx status: invalid

[ 1283.721225] 501-gm20b, pid 7012, refs 11:
[ 1283.721228] channel status: not in use pending_acquire busy
[ 1283.721235] RAMFC : TOP: 8000001e043233cc PUT: 0000001e0432350c GET: 0000001e043233cc FETCH: 0000421e021877d0
HEADER: 20020180 COUNT: 00110002
SYNCPOINT 00000000 00000f01 SEMAPHORE 0000001e 02380050 00000001 00081001

Please apply this patch to your driver and update your nvgpu.ko.
76c72cc.diff.zip (1.1 KB)

1 Like

thanks, we test it

Unfortunately, it will still wrong
[ 22.180849] nvgpu: 57000000.gpu gk20a_pmu_isr:724 [ERR] pmu exterr intr not implemented. Clearing interrupt.
[ 22.192658] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:560 [ERR] <<< FALCON id-0 DEBUG INFORMATION - START >>>
[ 22.192662] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:504 [ERR] FALCON IMEM BLK MAPPING (PA->VA) (96 TOTAL):
[ 22.192679] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x00: 0x01000000 0x01000100 0x01000200 0x01000300
[ 22.192683] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x04: 0x01000400 0x01000500 0x01000600 0x01000700
[ 22.192698] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x08: 0x01000800 0x01000900 0x01000a00 0x01000b00
[ 22.192701] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x0c: 0x01000c00 0x01000d00 0x01000e00 0x01000f00
[ 22.192716] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x10: 0x01001000 0x01001100 0x01001200 0x01001300
[ 22.192719] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x14: 0x01001400 0x01001500 0x01001600 0x01002d00
[ 22.192734] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x18: 0x01002e00 0x01002f00 0x01003000 0x01003100
[ 22.192737] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x1c: 0x01001c00 0x01001d00 0x01001e00 0x01001f00
[ 22.192752] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x20: 0x01002000 0x01002100 0x01002200 0x01004b00
[ 22.192755] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x24: 0x01004c00 0x01004d00 0x01004e00 0x01004f00
[ 22.192770] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x28: 0x01005000 0x01005100 0x01002300 0x01002400
[ 22.192773] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x2c: 0x01002500 0x01003200 0x01003300 0x01003400
[ 22.192788] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x30: 0x01003500 0x01003600 0x01003700 0x01003800
[ 22.192791] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x34: 0x01003900 0x01003a00 0x01003b00 0x01003c00
[ 22.192806] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x38: 0x01003d00 0x01003e00 0x01003f00 0x01004000
[ 22.192809] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x3c: 0x01004100 0x01004200 0x01004300 0x01004400
[ 22.192824] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x40: 0x01004500 0x01004600 0x01004700 0x01004800
[ 22.192827] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x44: 0x01004900 0x01004a00 0x01005200 0x01005300
[ 22.192842] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x48: 0x01005400 0x01005500 0x01005600 0x01005700
[ 22.192845] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x4c: 0x01005800 0x01005900 0x01005a00 0x01005b00
[ 22.192860] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x50: 0x01005c00 0x01005d00 0x01005e00 0x01005f00
[ 22.192863] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x54: 0x01006000 0x01006100 0x01006200 0x01006300
[ 22.192878] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:518 [ERR] 0x58: 0x01006400 0x01006500 0x01006600 0x01002600
[ 22.192881] nvgpu: 57000000.gpu gk20a_falcon_dump_imblk:520 [ERR] 0x5c: 0x01002700 0x01002800 0x01002900 0x01002c00
[ 22.192886] nvgpu: 57000000.gpu gk20a_falcon_dump_pc_trace:541 [ERR] PC TRACE (TOTAL 7 ENTRIES. entry 0 is the most recent branch):
[ 22.192891] nvgpu: 57000000.gpu gk20a_falcon_dump_pc_trace:549 [ERR] FALCON_TRACEPC(0) : 0x0000054a
[ 22.192894] nvgpu: 57000000.gpu gk20a_falcon_dump_pc_trace:549 [ERR] FALCON_TRACEPC(1) : 0x00000545
[ 22.192898] nvgpu: 57000000.gpu gk20a_falcon_dump_pc_trace:549 [ERR] FALCON_TRACEPC(2) : 0x0000054e
[ 22.192902] nvgpu: 57000000.gpu gk20a_falcon_dump_pc_trace:549 [ERR] FALCON_TRACEPC(3) : 0x00001041
[ 22.192906] nvgpu: 57000000.gpu gk20a_falcon_dump_pc_trace:549 [ERR] FALCON_TRACEPC(4) : 0x000015e2
[ 22.192910] nvgpu: 57000000.gpu gk20a_falcon_dump_pc_trace:549 [ERR] FALCON_TRACEPC(5) : 0x0000160a
[ 22.192913] nvgpu: 57000000.gpu gk20a_falcon_dump_pc_trace:549 [ERR] FALCON_TRACEPC(6) : 0x00000a59
[ 22.192915] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:567 [ERR] FALCON ICD REGISTERS DUMP
[ 22.192919] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:575 [ERR] FALCON_REG_PC : 0x1041
[ 22.192923] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:582 [ERR] FALCON_REG_SP : 0x43c4
[ 22.192926] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:575 [ERR] FALCON_REG_PC : 0x1041
[ 22.192930] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:582 [ERR] FALCON_REG_SP : 0x43c4
[ 22.192933] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:575 [ERR] FALCON_REG_PC : 0x1041
[ 22.192937] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:582 [ERR] FALCON_REG_SP : 0x43c4
[ 22.192940] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:575 [ERR] FALCON_REG_PC : 0x28e
[ 22.192944] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:582 [ERR] FALCON_REG_SP : 0x43c4
[ 22.192948] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:589 [ERR] FALCON_REG_IMB : 0xff5071
[ 22.192951] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:595 [ERR] FALCON_REG_DMB : 0xff50d8
[ 22.192955] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:601 [ERR] FALCON_REG_CSW : 0x330801
[ 22.192958] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:607 [ERR] FALCON_REG_CTX : 0x0
[ 22.192962] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:613 [ERR] FALCON_REG_EXCI : 0x693
[ 22.192966] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:622 [ERR] FALCON_REG_RSTAT[0] : 0xa5a5a5a5
[ 22.192970] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:622 [ERR] FALCON_REG_RSTAT[1] : 0xd84
[ 22.192974] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:622 [ERR] FALCON_REG_RSTAT[2] : 0xa5a5a5a5
[ 22.192977] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:622 [ERR] FALCON_REG_RSTAT[3] : 0xa5a5a5a5
[ 22.192981] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:622 [ERR] FALCON_REG_RSTAT[4] : 0xa5a5a5a5
[ 22.192985] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:622 [ERR] FALCON_REG_RSTAT[5] : 0xa5a5a5a5
[ 22.192987] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:625 [ERR] FALCON REGISTERS DUMP
[ 22.192990] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:627 [ERR] falcon_falcon_os_r : 0
[ 22.192993] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:629 [ERR] falcon_falcon_cpuctl_r : 0x60
[ 22.192997] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:631 [ERR] falcon_falcon_idlestate_r : 0x0
[ 22.193000] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:633 [ERR] falcon_falcon_mailbox0_r : 0x0
[ 22.193004] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:635 [ERR] falcon_falcon_mailbox1_r : 0x0
[ 22.193007] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:637 [ERR] falcon_falcon_irqstat_r : 0x60
[ 22.193011] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:639 [ERR] falcon_falcon_irqmode_r : 0xfc24
[ 22.193014] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:641 [ERR] falcon_falcon_irqmask_r : 0x7879
[ 22.193017] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:643 [ERR] falcon_falcon_irqdest_r : 0x90372
[ 22.193021] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:645 [ERR] falcon_falcon_debug1_r : 0x10000
[ 22.193024] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:647 [ERR] falcon_falcon_debuginfo_r : 0x0
[ 22.193027] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:649 [ERR] falcon_falcon_bootvec_r : 0x3200
[ 22.193031] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:651 [ERR] falcon_falcon_hwcfg_r : 0x400cc060
[ 22.193034] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:653 [ERR] falcon_falcon_engctl_r : 0x0
[ 22.193038] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:655 [ERR] falcon_falcon_curctx_r : 0x707fffe7
[ 22.193041] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:657 [ERR] falcon_falcon_nxtctx_r : 0x707fffe7
[ 22.193044] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:659 [ERR] falcon_falcon_exterrstat_r : 0x85002efb
[ 22.193048] nvgpu: 57000000.gpu gk20a_falcon_dump_stats:661 [ERR] falcon_falcon_exterraddr_r : 0x1840581c
[ 22.193052] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(0) : 0x0
[ 22.193056] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(1) : 0x0
[ 22.193059] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(2) : 0x0
[ 22.193062] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(3) : 0x0
[ 22.193066] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(4) : 0x0
[ 22.193069] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(5) : 0x0
[ 22.193073] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(6) : 0x0
[ 22.193076] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(7) : 0x0
[ 22.193080] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(8) : 0x0
[ 22.193083] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(9) : 0x0
[ 22.193087] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(10) : 0x0
[ 22.193090] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:630 [ERR] pwr_pmu_mailbox_r(11) : 0x0
[ 22.193094] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:635 [ERR] pwr_pmu_debug_r(0) : 0x0
[ 22.193097] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:635 [ERR] pwr_pmu_debug_r(1) : 0x0
[ 22.193100] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:635 [ERR] pwr_pmu_debug_r(2) : 0x2b
[ 22.193104] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:635 [ERR] pwr_pmu_debug_r(3) : 0x0
[ 22.193107] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:639 [ERR] pwr_pmu_bar0_error_status_r : 0x0
[ 22.193111] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:652 [ERR] pwr_pmu_bar0_fecs_error_r : 0xbadf1201
[ 22.193115] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:655 [ERR] pwr_falcon_exterrstat_r : 0x85002efb
[ 22.193118] nvgpu: 57000000.gpu gk20a_pmu_dump_falcon_stats:659 [ERR] pwr_falcon_exterraddr_r : 0x1840581c
[ 22.193151] nvgpu: 57000000.gpu print_pmu_trace:88 [ERR] dump PMU trace buffer
[ 22.193156] nvgpu: 57000000.gpu nvgpu_pmu_dump_falcon_stats:51 [ERR] pmu state: 7
[ 22.193159] nvgpu: 57000000.gpu nvgpu_pmu_dump_falcon_stats:52 [ERR] elpg state: 0
[ 22.193163] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:129 [ERR] gr_fecs_os_r : 0
[ 22.193166] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:131 [ERR] gr_fecs_cpuctl_r : 0x60
[ 22.193170] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:133 [ERR] gr_fecs_idlestate_r : 0x0
[ 22.193173] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:135 [ERR] gr_fecs_mailbox0_r : 0x0
[ 22.193176] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:137 [ERR] gr_fecs_mailbox1_r : 0x0
[ 22.193179] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:139 [ERR] gr_fecs_irqstat_r : 0x0
[ 22.193183] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:141 [ERR] gr_fecs_irqmode_r : 0x4
[ 22.193186] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:143 [ERR] gr_fecs_irqmask_r : 0x8704
[ 22.193189] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:145 [ERR] gr_fecs_irqdest_r : 0x0
[ 22.193192] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:147 [ERR] gr_fecs_debug1_r : 0x40
[ 22.193196] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:149 [ERR] gr_fecs_debuginfo_r : 0x0
[ 22.193199] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:151 [ERR] gr_fecs_ctxsw_status_1_r : 0x382
[ 22.193202] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(0) : 0x1
[ 22.193205] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(1) : 0x0
[ 22.193209] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(2) : 0x1
[ 22.193212] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(3) : 0x0
[ 22.193215] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(4) : 0x1ffda0
[ 22.193218] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(5) : 0x0
[ 22.193221] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(6) : 0x0
[ 22.193224] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(7) : 0x0
[ 22.193227] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(8) : 0x0
[ 22.193230] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(9) : 0x0
[ 22.193233] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(10) : 0x0
[ 22.193236] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(11) : 0x0
[ 22.193239] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(12) : 0x0
[ 22.193243] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(13) : 0x0
[ 22.193246] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(14) : 0x0
[ 22.193249] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:155 [ERR] gr_fecs_ctxsw_mailbox_r(15) : 0x0
[ 22.193252] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:159 [ERR] gr_fecs_engctl_r : 0x0
[ 22.193255] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:161 [ERR] gr_fecs_curctx_r : 0x0
[ 22.193258] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:163 [ERR] gr_fecs_nxtctx_r : 0x0
[ 22.193262] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:169 [ERR] FECS_FALCON_REG_IMB : 0xbadfbadf
[ 22.193266] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:175 [ERR] FECS_FALCON_REG_DMB : 0xbadfbadf
[ 22.193269] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:181 [ERR] FECS_FALCON_REG_CSW : 0xbadfbadf
[ 22.193273] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:187 [ERR] FECS_FALCON_REG_CTX : 0xbadfbadf
[ 22.193277] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:193 [ERR] FECS_FALCON_REG_EXCI : 0xbadfbadf
[ 22.193280] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:200 [ERR] FECS_FALCON_REG_PC : 0xbadfbadf
[ 22.193284] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:206 [ERR] FECS_FALCON_REG_SP : 0xbadfbadf
[ 22.193288] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:200 [ERR] FECS_FALCON_REG_PC : 0xbadfbadf
[ 22.193291] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:206 [ERR] FECS_FALCON_REG_SP : 0xbadfbadf
[ 22.193295] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:200 [ERR] FECS_FALCON_REG_PC : 0xbadfbadf
[ 22.193299] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:206 [ERR] FECS_FALCON_REG_SP : 0xbadfbadf
[ 22.193303] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:200 [ERR] FECS_FALCON_REG_PC : 0xbadfbadf
[ 22.193306] nvgpu: 57000000.gpu gk20a_fecs_dump_falcon_stats:206 [ERR] FECS_FALCON_REG_SP : 0xbadfbadf
[ 22.193376] nvgpu: 57000000.gpu gk20a_ptimer_isr:50 [ERR] PRI timeout: ADR 0x00122124 READ DATA 0x00000000
[ 22.193379] nvgpu: 57000000.gpu gk20a_ptimer_isr:56 [ERR] FECS_ERRCODE 0xbadf5202

Hi

[ 22.180849] nvgpu: 57000000.gpu gk20a_pmu_isr:724 [ERR] pmu exterr intr not implemented. Clearing interrupt.

The PMU error is a known issue that happens to all of Jetson Nano production module. Not yet resolved.

But this should not affect your usecase. Please try.

Thanks , we keep testing and tracking this case.