GPU driver crashing on boot on TK1 custom board

We are trying to boot a custom board based on TK1. On the display attached to our board, we can see the kernel logs being printed. On serial console, we land on Ubuntu command line, but the display attached to our board does not show login screen or Ubuntu desktop. Below is snippet of boot logs just before login prompt:

* Setting up X socket directories...                                                                                    [ OK ]
 * Stopping System V initialisation compatibility                                                                        [ OK ]
 * Starting System V runlevel compatibility                                                                              [ OK ]
 * Starting Restore Sound Card State                                                                                     [ OK ]
 * Starting save kernel messages                                                                                         [ OK ]
 * Starting OpenSSH server                                                                                               [ OK ]
 * Starting anac(h)ronistic cron                                                                                         [ OK ]
 * Starting crash report submission daemon                                                                               [ OK ]
 * Starting regular background program processing daemon                                                                 [ OK ]
 * Stopping Restore Sound Card State                                                                                     [ OK ]
 * Starting DNS forwarder and DHCP server dnsmasq        * Stopping anac(h)ronistic cron                                 [ OK ]
                                                                                                                         [ OK ]
 * Stopping save kernel messages                                                                                         [ OK ]
/etc/resolvconf/update.d/libc: Warning: /etc/resolv.conf is not a symbolic link to /run/resolvconf/resolv.conf
[   19.649978] gk20a gk20a.0: gk20a_pmu_isr: pmu halt intr not implemented
[   19.656635] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_os_r : 17997577
[   19.663686] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_cpuctl_r : 0x10
[   19.670727] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_idlestate_r : 0x1
[   19.677939] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_mailbox0_r : 0x0
[   19.685075] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_mailbox1_r : 0x0
[   19.692200] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_irqstat_r : 0x18
[   19.699324] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_irqmode_r : 0xfc24
[   19.706634] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_irqmask_r : 0xf3
[   19.713758] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_irqdest_r : 0xff01ff52
[   19.721407] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(0) : 0x0
[   19.728447] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(1) : 0x0
[   19.735484] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(2) : 0x0
[   19.742527] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(3) : 0x0
[   19.749566] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(4) : 0x0
[   19.756612] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(5) : 0x0
[   19.763650] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(6) : 0x0
[   19.770689] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(7) : 0x0
[   19.777726] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(8) : 0x0
[   19.784762] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(9) : 0x0
[   19.791804] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(10) : 0x0
[   19.798928] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_mailbox_r(11) : 0x0
[   19.806051] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_debug_r(0) : 0x0
[   19.812916] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_debug_r(1) : 0x0
[   19.819781] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_debug_r(2) : 0x20
[   19.826734] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_debug_r(3) : 0x0
[   19.833599] gk20a gk20a.0: pmu_dump_falcon_stats: pmu_rstat (0) : 0x40000000
[   19.840637] gk20a gk20a.0: pmu_dump_falcon_stats: pmu_rstat (1) : 0x1230aaf0
[   19.847673] gk20a gk20a.0: pmu_dump_falcon_stats: pmu_rstat (2) : 0x10
[   19.854191] gk20a gk20a.0: pmu_dump_falcon_stats: pmu_rstat (3) : 0xed03d
[   19.860973] gk20a gk20a.0: pmu_dump_falcon_stats: pmu_rstat (4) : 0x0
[   19.867403] gk20a gk20a.0: pmu_dump_falcon_stats: pmu_rstat (5) : 0x3f
[   19.873921] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_bar0_error_status_r : 0x0
[   19.881565] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_pmu_bar0_fecs_error_r : 0x0
[   19.889036] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_exterrstat_r : 0x0
[   19.896338] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_engctl_r : 0x0
[   19.903287] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_curctx_r : 0x0
[   19.910237] gk20a gk20a.0: pmu_dump_falcon_stats: pwr_falcon_nxtctx_r : 0x600f0e4a
[   19.917795] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_IMB : 0x80001
[   19.925005] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_DMB : 0x8006f
[   19.932219] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_CSW : 0x0
[   19.939084] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_CTX : 0x0
[   19.945947] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_EXCI : 0x0
[   19.952904] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_PC : 0x11
[   19.959769] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_SP : 0x2200
[   19.966809] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_PC : 0x11
[   19.973673] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_SP : 0x2200
[   19.980711] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_PC : 0x11
[   19.987574] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_SP : 0x2200
[   19.994612] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_PC : 0x11
[   20.001481] gk20a gk20a.0: pmu_dump_falcon_stats: PMU_FALCON_REG_SP : 0x2200
[   20.008517] gk20a gk20a.0: pmu_dump_falcon_stats: elpg stat: 0
[   20.008517]
[   20.015817] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_os_r : -1159785984
[   20.023462] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_cpuctl_r : 0xbadf1200
[   20.031365] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_idlestate_r : 0xbadf1200
[   20.039535] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox0_r : 0xbadf1200
[   20.047613] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox1_r : 0xbadf1200
[   20.055691] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_irqstat_r : 0xbadf1200
[   20.063680] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmode_r : 0xbadf1200
[   20.071674] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmask_r : 0xbadf1200
[   20.079664] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_irqdest_r : 0xbadf1200
[   20.087654] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_debug1_r : 0xbadf1200
[   20.095575] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_debuginfo_r : 0xbadf1200
[   20.103743] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(0) : 0xbadf1200
[   20.112513] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(1) : 0xbadf1200
[   20.121283] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(2) : 0xbadf1200
[   20.130054] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(3) : 0xbadf1200
[   20.138828] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(4) : 0xbadf1200
[   20.147599] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(5) : 0xbadf1200
[   20.156368] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(6) : 0xbadf1200
[   20.165137] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(7) : 0xbadf1200
[   20.173911] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_engctl_r : 0xbadf1200
[   20.181814] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_curctx_r : 0xbadf1200
[   20.189717] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_nxtctx_r : 0xbadf1200
[   20.197622] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_IMB : 0xbadf1200
[   20.205786] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_DMB : 0xbadf1200
[   20.213953] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CSW : 0xbadf1200
[   20.222118] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CTX : 0xbadf1200
[   20.230281] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_EXCI : 0xbadf1200
[   20.238531] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadf1200
[   20.246613] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadf1200
[   20.254690] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadf1200
[   20.262766] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadf1200
[   20.270844] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadf1200
[   20.278921] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadf1200
[   20.287001] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0xbadf1200
[   20.295080] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xbadf1200
[   20.303264] gk20a gk20a.0: gk20a_pbus_isr: pmc_enable : 0xe031212c
[   20.309437] gk20a gk20a.0: gk20a_pbus_isr: NV_PTIMER_PRI_TIMEOUT_SAVE_0: 0x80409081
[   20.309437]
[   20.318563] gk20a gk20a.0: gk20a_pbus_isr: NV_PTIMER_PRI_TIMEOUT_SAVE_1: 0x0
[   20.318563]
[   20.327074] gk20a gk20a.0: gk20a_pbus_isr: NV_PTIMER_PRI_TIMEOUT_FECS_ERRCODE: 0xbadf1200
[   20.327074]
[   20.336711] gk20a gk20a.0: gk20a_pbus_isr: Unhandled pending pbus interrupt
[   20.336711]

Ubuntu 14.04.1 LTS tegra-ubuntu ttyS0

tegra-ubuntu login: ubuntu (automatic login)

Last login: Sat Jan  1 00:02:15 UTC 2000 on ttyS0
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.10.40 armv7l)

 * Documentation:  https://help.ubuntu.com/
run-parts: /etc/update-motd.d/98-fsck-at-reboot exited with return code 1
ubuntu@tegra-ubuntu:~$

After few seconds, we get following dump on serial console:

[   30.117206] gk20a gk20a.0: gr_gk20a_ctx_wait_ucode: timeout waiting on ucode response
[   30.126315] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_os_r : 0
[   30.134001] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_cpuctl_r : 0x0
[   30.141440] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_idlestate_r : 0x1
[   30.149171] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox0_r : 0x0
[   30.157276] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_mailbox1_r : 0x0
[   30.165381] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_irqstat_r : 0x0
[   30.172968] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmode_r : 0x4
[   30.181251] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_irqmask_r : 0x8704
[   30.189175] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_irqdest_r : 0x0
[   30.196622] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_debug1_r : 0x40
[   30.204537] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_debuginfo_r : 0x910a42
[   30.212600] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(0) : 0x0
[   30.221160] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(1) : 0x0
[   30.229432] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(2) : 0x0
[   30.238037] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(3) : 0x0
[   30.246280] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(4) : 0x0
[   30.256413] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(5) : 0x0
[   30.264737] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(6) : 0x0
[   30.273000] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_ctxsw_mailbox_r(7) : 0x0
[   30.281341] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_engctl_r : 0x0
[   30.288711] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_curctx_r : 0x0
[   30.296118] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: gr_fecs_nxtctx_r : 0x0
[   30.303490] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_IMB : 0x80411
[   30.311577] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_DMB : 0x8044d
[   30.319552] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CSW : 0x10804
[   30.327559] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_CTX : 0x0
[   30.335187] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_EXCI : 0x0
[   30.343008] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0x526
[   30.350722] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xfdc
[   30.358467] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0x4ef
[   30.366178] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xfdc
[   30.373999] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0x526
[   30.381713] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xfdc
[   30.389465] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_PC : 0x52d
[   30.397179] gk20a gk20a.0: gk20a_fecs_dump_falcon_stats: FECS_FALCON_REG_SP : 0xfe0
[   30.404996] gk20a gk20a.0: gr_gk20a_load_ctxsw_ucode: falcon ucode init timeout
[   30.412365] gk20a gk20a.0: gk20a_init_gr_reset_enable_hw: fail
[   30.421608] Unable to handle kernel NULL pointer dereference at virtual address 00000010
[   30.429860] pgd = ece18000
[   30.432630] [00000010] *pgd=00000000
[   30.436286] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[   30.441585] Modules linked in: dm_crypt dm_mod rfcomm bnep bluetooth rfkill nvhost_vi
[   30.449469] CPU: 0 PID: 878 Comm: Xorg Not tainted 3.10.40 #42
[   30.455287] task: ec0b25c0 ti: ebea2000 task.ti: ebea2000
[   30.460686] PC is at gk20a_get_sgtable_from_pages+0x58/0xe0
[   30.466249] LR is at sg_init_table+0x1c/0x34
[   30.470507] pc : [<c03f6e44>]    lr : [<c02bd5d4>]    psr: 600f0013
[   30.470507] sp : ebea3d30  ip : 00000fdc  fp : c0ca55e0
[   30.481958] r10: 00017580  r9 : edb12138  r8 : ee3f6c10
[   30.487167] r7 : 00000010  r6 : 00600000  r5 : 00000000  r4 : edb1236c
[   30.493678] r3 : ec2b9580  r2 : 00000002  r1 : ec079201  r0 : 00000000
[   30.500191] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   30.507309] Control: 10c5387d  Table: ace1806a  DAC: 00000015
[   30.513039]
[   30.513039] PC: 0xc03f6dc4:
[   30.517296] 6dc4  e1a03007 eb00bc85 e1a06000 eaffffdc e3021054 e1a00004 e34c10a1 e3e0600b
[   30.525522] 6de4  eb006ce2 eaffffe2 e92d41f0 e52de004 e8bd4000 e3063c64 e34c30d6 e1a04001
[   30.533747] 6e04  e1a08000 e1a07002 e5930018 e3a0200c e30810d0 e59d6018 ebf541b0 e3500000
[   30.541970] 6e24  e5840000 0a000020 e3a020d0 e3a01001 ebfb1b7c e2505000 1a00000f e5943000
[   30.550194] 6e44  e5971000 e5932000 e3110003 e5923000 e2033003 1a00001a e1833001 e8820028
[   30.558417] 6e64  e59d3020 e5823008 e5943000 e5933000 e583600c e1a00005 e8bd81f0 e30f1180
[   30.566640] 6e84  e1a00008 e34c10a2 eb006cb8 e5940000 e3500000 0afffff6 ebf53dc9 e3a03000
[   30.574862] 6ea4  e1a00005 e5843000 e8bd81f0 e3021054 e1a00008 e34c10a1 e3e0500b eb006cab
[   30.583087]
[   30.583087] LR: 0xc02bd554:
[   30.587343] d554  e1a05001 e5991008 e1a06002 e3510000 0a000010 e0857105 e2658001 e1a07107
[   30.595567] d574  e2477014 e1550001 e3a03000 e1a04003 37904007 30883001 31a01005 e5893008
[   30.603788] d594  33c44003 e12fff36 e5991008 e1a00004 e3510000 1afffff2 e3a03000 e5893000
[   30.612010] d5b4  e8bd87f0 e0811101 e92d4070 e1a05000 e1b04101 0a000001 e1a01004 ebffc30a
[   30.620234] d5d4  e2441014 e7953001 e3c33001 e3833002 e7853001 e8bd8070 e92d4ff0 e1a09002
[   30.628457] d5f4  e0898109 e24dd00c e1a05001 e1a08108 e3a0100c e3a07000 e58d3000 e2488014
[   30.636679] d614  e1a0a000 ebffc2f8 e2493001 e58d3004 e1550009 e1a04005 859d3004 e3a06000
[   30.644903] d634  e59d1000 80456003 81a05009 81a04003 e1a00005 e59d3030 e12fff33 e250b000
[   30.653126]
[   30.653126] SP: 0xebea3cb0:
[   30.657383] 3cb0  ee2cea00 00000002 00000000 00000001 00000001 00000000 00000000 c02bd5d4
[   30.665606] 3cd0  c03f6e44 600f0013 ffffffff ebea3d1c ee3f6c10 c000eb18 00000000 ec079201
[   30.673828] 3cf0  00000002 ec2b9580 edb1236c 00000000 00600000 00000010 ee3f6c10 edb12138
[   30.682051] 3d10  00017580 c0ca55e0 00000fdc ebea3d30 c02bd5d4 c03f6e44 600f0013 ffffffff
[   30.690273] 3d30  edb12368 00000000 ee3f6c00 ee3f6c10 00000010 c03e1c74 00600000 00000000
[   30.698497] 3d50  00000000 c0d78390 00000010 00600000 c0d78390 edb12000 ee3f6c00 edb122a8
[   30.706718] 3d70  00001800 c03f2d54 00000000 ebfd7940 00120000 00115000 ed4d5940 ec128400
[   30.714939] 3d90  ebfd77c0 00000002 00000002 00000002 edb13000 00000000 00000000 00132000
[   30.723163]
[   30.723163] FP: 0xc0ca5560:
[   30.727419] 5560  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.735643] 5580  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.743868] 55a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.752091] 55c0  00000000 00000000 00000000 00000000 00000005 c0ca5670 00000000 00000000
[   30.760316] 55e0  00000000 edb12000 00000101 edda7d50 c0c1b0e8 c0c08df0 c0c13870 00000003
[   30.768537] 5600  000001f4 00000032 c0410948 c0410eb0 c0410738 00000000 00000000 00000000
[   30.776762] 5620  00000000 00000000 00000000 00000000 00000000 c04106bc c041070c c04106fc
[   30.784985] 5640  00000001 00000000 c0ca5648 c0ca5648 00000000 00000000 c04106dc c0410d40
[   30.793211]
[   30.793211] R1: 0xec079181:
[   30.797466] 9180  c0c9d79c 00000002 00000000 ec07918c ec07918c ecb2d0b8 00000000 00000000
[   30.805689] 91a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.813915] 91c0  74726f70 62752e73 75746e75 6d6f632e 7562755f 2d75746e 74726f70 69645f73
[   30.822137] 91e0  5f737473 73757274 732d7974 72756365 5f797469 656c6552 2e657361 00677067
[   30.830360] 9200  00000002 00000000 00000000 00000000 00000000 00000000 000c0000 ffffffff
[   30.838583] 9220  7a6e7565 00007069 00000000 00000000 00000000 00000000 00000000 00000000
[   30.846806] 9240  4eae5890 5e7499ee ebcd4508 00000000 00000000 00000000 0009ef17 6c62011b
[   30.855029] 9260  6c6b6361 2d747369 65726172 74656e2d 6b726f77 6e6f632e 00000066 00000000
[   30.863254] 9280  00000100 ec128000 ec0798e0 ec0798c0 ec07928c 00000010 000c0b00 65670707
[   30.871476]
[   30.871476] R3: 0xec2b9500:
[   30.875733] 9500  ec7c6a00 ebcd4e00 00000000 ebc28000 00000000 00000000 000c0000 ffffffff
[   30.883955] 9520  726f7463 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.892176] 9540  742f2e2e 31357974 00000000 00000000 ec2f4000 c08695c4 00000001 00000000
[   30.900400] 9560  ec2b9560 ec2b9560 00000000 00000000 00000000 00000001 00100100 00200200
[   30.908621] 9580  ec079200 00000001 00000001 00000000 00000000 00000000 00000000 00000000
[   30.916845] 95a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.925067] 95c0  742f2e2e 00357974 00000000 00000000 ec2e3000 c08695c4 00000001 00000000
[   30.933289] 95e0  ec2b95e0 ec2b95e0 00000000 00000000 00000000 00000001 00100100 00200200
[   30.941511]
[   30.941511] R4: 0xedb122ec:
[   30.945768] 22ec  ec5a1300 ec2c39c0 00017580 00220000 00000000 00000010 00000000 00000000
[   30.953990] 230c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.962214] 232c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.970437] 234c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 c03e4938
[   30.978660] 236c  ec2b9580 00000010 00000000 00600000 00000000 00000010 00000000 00000000
[   30.986882] 238c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 f03fd000
[   30.995105] 23ac  00000000 0011b000 00000000 00001000 00000000 00001000 00000000 f05f0000
[   31.003326] 23cc  00000000 00132000 00000000 00001000 00000000 00001000 ecf43680 00000020
[   31.011551]
[   31.011551] R8: 0xee3f6b90:
[   31.015808] 6b90  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   31.024030] 6bb0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   31.032255] 6bd0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   31.040476] 6bf0  00000000 00000000 00000000 00000000 ee3f46c0 ffffffff 00000000 00000000
[   31.048700] 6c10  ee3f3c10 ee3f1740 ee3f46c0 ee3f6a1c ee3f6e1c ee3f3c18 ee279540 c0ca5700
[   31.056922] 6c30  ee3f5348 00000005 00000007 00000000 00000000 00000001 00000000 ee3f6c4c
[   31.065145] 6c50  ee3f6c4c 00000000 00000000 c0ca59d8 c0ca49ac 00000000 00000000 00000020
[   31.073368] 6c70  000b000b ee3f6a74 ee3f6e74 7fffffff 00010001 ee3f6c84 ee3f6c84 00000000
[   31.081592]
[   31.081592] R9: 0xedb120b8:
[   31.085848] 20b8  00000200 00010000 ebd40000 00000001 00000000 edb120cc edb120cc 00000000
[   31.094073] 20d8  00000000 c03d226c 00000001 00000001 00000000 edb120ec edb120ec 00000000
[   31.102295] 20f8  00000000 08101eff c06fe000 04800000 00000000 00000000 00000001 00000000
[   31.110519] 2118  edb12118 edb12118 00000000 00000000 ffffffe0 edb1212c edb1212c c03d4e7c
[   31.118742] 2138  edb12000 00000000 00000004 00000000 00000000 00000000 00000000 00000000
[   31.126966] 2158  00000000 00000000 ecde0000 000007df ebc35000 000001c1 ec5b8000 00000ee6
[   31.135188] 2178  ebc35800 000001df ec726000 00000345 ec280000 00000508 ebf74000 00000174
[   31.143413] 2198  ebf27800 000000aa ec439000 000000ac ec12a000 00000052 ec023c00 00000040
[   31.151637] Process Xorg (pid: 878, stack limit = 0xebea2238)
[   31.157368] Stack: (0xebea3d30 to 0xebea4000)
[   31.161712] 3d20:                                     edb12368 00000000 ee3f6c00 ee3f6c10
[   31.169871] 3d40: 00000010 c03e1c74 00600000 00000000 00000000 c0d78390 00000010 00600000
[   31.178029] 3d60: c0d78390 edb12000 ee3f6c00 edb122a8 00001800 c03f2d54 00000000 ebfd7940
[   31.186187] 3d80: 00120000 00115000 ed4d5940 ec128400 ebfd77c0 00000002 00000002 00000002
[   31.194345] 3da0: edb13000 00000000 00000000 00132000 00000001 edb12000 c0d78390 00000000
[   31.202502] 3dc0: ebea2000 ee3f6c10 edb13000 00000000 00000000 c03ceb74 00000000 edb132c8
[   31.210660] 3de0: edb132c8 edb13344 ee3f6c10 edb13380 aae4d8a9 00000004 00000000 c0424660
[   31.218818] 3e00: aae4d8a9 00000004 00000000 00000000 ec0b25c0 c008da64 ebea3e18 ebea3e18
[   31.226977] 3e20: c1c3fea0 ee3f6c10 ee3f6c70 c042443c 00000000 c0ba60c0 c008da64 ebea2000
[   31.235135] 3e40: 00000000 c041f824 ee3f6c10 00000000 ee3f3c10 c041f880 ee3f6c10 00000000
[   31.243295] 3e60: ee3f3c10 c0420200 c0ce9e80 ebd06000 00100100 c01169fc ece1adc0 c01107c8
[   31.251452] 3e80: ee3f6c10 ee3f6c70 00000004 60010013 edb12000 ebea2000 00000000 c0420ec0
[   31.259609] 3ea0: edb12018 ee3f6c10 ee3f6c00 00000010 edb12000 c03d0780 c0104705 ee3f6c00
[   31.267766] 3ec0: beb39888 c03d1f5c ebea3f00 00000038 00000000 b70114d0 00000000 00000000
[   31.275924] 3ee0: ec00f0d0 b6e63000 b6e64000 bf000000 00000000 00000000 00000400 ebd06000
[   31.284082] 3f00: ec5a1fc0 beb39888 edadf268 ec74a380 0000000d c0104705 beb39888 c0160d0c
[   31.292241] 3f20: ec6b7200 00000001 ecfe7e90 b6e63000 ec79c650 ec6b7204 ec00f0d0 c0133ad0
[   31.300398] 3f40: 00000020 c02b5338 ec6b7244 60010013 00100100 ec6b7240 00000000 00000000
[   31.308556] 3f60: ec74a380 b700b2b8 0000000d c0104705 beb39888 00000000 00000000 c0160eec
[   31.316713] 3f80: b6e63000 00000000 00001000 b7011450 beb39888 b700b2b8 00000036 c000f144
[   31.324870] 3fa0: ebea2000 c000efa0 b7011450 beb39888 0000000d c0104705 beb39888 00000020
[   31.333027] 3fc0: b7011450 beb39888 b700b2b8 00000036 b4cd8390 b70114d0 00000000 00000000
[   31.341185] 3fe0: b6bd70a1 beb3987c b4cccf0f b6bd70a6 80010030 0000000d 00000000 00000000
[   31.349356] [<c03f6e44>] (gk20a_get_sgtable_from_pages+0x58/0xe0) from [<c03e1c74>] (gk20a_gr_alloc_ctx_buffer+0xb0/0x134)
[   31.360378] [<c03e1c74>] (gk20a_gr_alloc_ctx_buffer+0xb0/0x134) from [<c03f2d54>] (gk20a_init_gr_support+0xc20/0x1204)
[   31.371057] [<c03f2d54>] (gk20a_init_gr_support+0xc20/0x1204) from [<c03ceb74>] (gk20a_pm_finalize_poweron+0x1f4/0x644)
[   31.381825] [<c03ceb74>] (gk20a_pm_finalize_poweron+0x1f4/0x644) from [<c0424660>] (pm_genpd_runtime_resume+0x224/0x2ec)
[   31.392677] [<c0424660>] (pm_genpd_runtime_resume+0x224/0x2ec) from [<c041f824>] (__rpm_callback+0x34/0x68)
[   31.402398] [<c041f824>] (__rpm_callback+0x34/0x68) from [<c041f880>] (rpm_callback+0x28/0x7c)
[   31.410991] [<c041f880>] (rpm_callback+0x28/0x7c) from [<c0420200>] (rpm_resume+0x468/0x6b0)
[   31.419411] [<c0420200>] (rpm_resume+0x468/0x6b0) from [<c0420ec0>] (__pm_runtime_resume+0x50/0x68)
[   31.428439] [<c0420ec0>] (__pm_runtime_resume+0x50/0x68) from [<c03d0780>] (gk20a_busy+0x38/0xa0)
[   31.437293] [<c03d0780>] (gk20a_busy+0x38/0xa0) from [<c03d1f5c>] (gk20a_ctrl_dev_ioctl+0x1e8/0x4f8)
[   31.446410] [<c03d1f5c>] (gk20a_ctrl_dev_ioctl+0x1e8/0x4f8) from [<c0160d0c>] (do_vfs_ioctl+0x430/0x5b8)
[   31.455872] [<c0160d0c>] (do_vfs_ioctl+0x430/0x5b8) from [<c0160eec>] (SyS_ioctl+0x58/0x168)
[   31.464296] [<c0160eec>] (SyS_ioctl+0x58/0x168) from [<c000efa0>] (ret_fast_syscall+0x0/0x30)
[   31.472805] Code: ebfb1b7c e2505000 1a00000f e5943000 (e5971000)
[   31.484100] ---[ end trace 0b9c39fd3ad6af2d ]---

We have one hardware change related to display i.e. there’s no i2c interface between HDMI interface and HDMI-to-VGA converter. So we have hardcoded the EDID information in the driver.

Can someone suggest why this might be occurring?

P.S: We have tried a debian based distro (with no desktop manager) on our board and we can see the login prompt on the display.

Please try this workaround.

sudo echo 0 > /sys/devices/platform/host1x/gk20a.0/elpg_enable

Thanks for the response. I have tried this but the command does not return and is not acting upon terminate/interrupt signals

root@tegra-ubuntu:~# echo 0 > /sys/devices/platform/host1x/gk20a.0/elpg_enable
^C^C^X^X^Z^Z

Is there any error log when this command does not return?

This command would return on NVIDIA devkit.

There is no log printed. I have to force reset board to get CLI again.

I have tried to debug the crashing driver and noticed that sometimes the statement

desc->pages = dma_alloc_attrs(...);

in function ‘gk20a_gr_alloc_ctx_buffer()’ of drivers/gpu/nvgpu/gk20a/gr_gk20a.c sets desc->pages to be either 0x10 or 0x0. Can someone give a hint why dma_alloc_attrs() is returning such unexpected values?

Hi EmbEng,

Is this still an issue on your device?
Can issue be reproduced on TK1 devkit?

Thanks

Hi kayccc,

Yes issue was resolved recently & I had a plan to update the status here.

There was a routing issue in DDR which resulted in chip-select of two DDR chips to be continuously asserted. This resulted in a number of random issues including GPU driver crashing on boot. After doing a rework on our board, everything went smooth.