Upstream kernel v5.12 crash at DMA for Xavier AGX

Hi,

I’m testing the experimental UEFI firmware with upstream kernel/device tree.

The board is now using systemd-boot to load kernel from NVME drive.

Although the board can boot upstream kernel without obvious problem (aka, NVME drive is working, network is working), but every time when I try to push linux source tree to the board, it crashes doing dma:

[  246.157408] Unable to handle kernel write to read-only memory at virtual address ffff6ca9ec880000
[  246.166537] Mem abort info:
[  246.169405]   ESR = 0x9600014f
[  246.172615]   EC = 0x25: DABT (current EL), IL = 32 bits
[  246.178090]   SET = 0, FnV = 0
[  246.181204]   EA = 0, S1PTW = 0
[  246.184481] Data abort info:
[  246.187613]   ISV = 0, ISS = 0x0000014f
[  246.191676]   CM = 1, WnR = 1
[  246.194834] swapper pgtable: 64k pages, 48-bit VAs, pgdp=000000033d140000
[  246.201820] [ffff6ca9ec880000] pgd=000000087ffd0003, p4d=000000087ffd0003, pud=000000087ffd0003, pmd=000000087ff80003, pte=006000010c880f87
[  246.214848] Internal error: Oops: 9600014f [#1] PREEMPT SMP
[  246.220507] Modules linked in: rfkill tegra210_adma tegra_drm cec drm_kms_helper crct10dif_ce snd_hda_codec_hdmi tegra_aconnect pwm_fan snd_hda_tegra pwm_tegra snd_hda_codec xhci_tegra host1x snd_hda_core lm90 xhci_hcd phy_tegra_xusb at24 tegra_bpmp_thermal fuse drm ip_tables x_tables ipv6
[  246.246861] CPU: 0 PID: 10 Comm: ksoftirqd/0 Not tainted 5.12.0-custom #35
[  246.253826] Hardware name: NVIDIA Jetson AGX Xavier Developer Kit (DT)
[  246.260415] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
[  246.266503] pc : __inval_dcache_area+0x44/0x5c
[  246.271135] lr : arch_sync_dma_for_cpu+0x24/0x30
[  246.275895] sp : ffff80001028fb90
[  246.279270] x29: ffff80001028fb90 x28: 0000000000000000 
[  246.284802] x27: ffff6ca9e2d008c0 x26: 0000000000000000 
[  246.290311] x25: 00000000000001ec x24: 00000000000005e2 
[  246.295804] x23: ffff6ca9e2d008c0 x22: 00000000ffffff8c 
[  246.301312] x21: 0000000000000002 x20: ffff6ca9e0ba0c10 
[  246.306848] x19: 0000000109140000 x18: 0000000000000000 
[  246.312383] x17: 0000000000000000 x16: ffffcc4f8295eee0 
[  246.317882] x15: 0000000000000002 x14: 18ef843139351600 
[  246.323351] x13: 4297450f00000000 x12: 0000c05b670ca103 
[  246.328824] x11: 0e24640b00000000 x10: 0000c05b670ca103 
[  246.334345] x9 : 0000008000000000 x8 : 00000000ffffff8c 
[  246.339853] x7 : 000000010e310000 x6 : 0000008000000000 
[  246.345323] x5 : 0000000109140000 x4 : ffff6ca9e25d9d80 
[  246.350879] x3 : 000000000000003f x2 : 0000000000000040 
[  246.356346] x1 : ffff6caae913ff80 x0 : ffff6ca9ec880000 
[  246.361793] Call trace:
[  246.364307]  __inval_dcache_area+0x44/0x5c
[  246.368514]  dma_sync_single_for_cpu+0xdc/0x124
[  246.373141]  stmmac_napi_poll_rx+0x488/0xb80
[  246.377536]  __napi_poll+0x38/0x17c
[  246.381113]  net_rx_action+0xe8/0x290
[  246.384914]  _stext+0x12c/0x2b4
[  246.388158]  run_ksoftirqd+0x50/0x64
[  246.391845]  smpboot_thread_fn+0x254/0x280
[  246.396039]  kthread+0x158/0x160
[  246.399390]  ret_from_fork+0x10/0x3c
[  246.403122] Code: 8a230000 54000060 d50b7e20 14000002 (d5087620) 
[  246.409344] ---[ end trace cfb27bf1c722eccd ]---
[  246.414029] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[  246.420991] SMP: stopping secondary CPUs
[  246.425096] Kernel Offset: 0x4c4f72860000 from 0xffff800010000000
[  246.431256] PHYS_OFFSET: 0xffff935720000000
[  246.435496] CPU features: 0x20240002,23802830
[  246.439921] Memory Limit: none
[  246.443093] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

Any clue on this?