NVRM: Xid MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_9 faulted @ 0x1_03e00000 Fault is of type FAULT_PTE ACCESS_TYPE_VIRT_READ

got this error from kernel message, search on Nvidia website, it says “The problem is most often a result of a coding defect, in your kernel code, or in kernel code that is launched by your program.”
looking for some debugging suggestion to find the root cause?

[    4.492100] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  470.63.01  Tue Aug  3 20:44:16 UTC 2021
[  147.282063] NVRM: GPU at PCI:0000:86:00: GPU-99c1b149-3c30-6a5f-a0e6-abd0a2e2249d
[  147.282067] NVRM: Xid (PCI:0000:86:00): 31, pid=51691, Ch 00000050, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_9 faulted @ 0x1_03e00000. Fault is of type FAULT_PTE ACCESS_TYPE_VIRT_READ
[  467.573806] NVRM: Xid (PCI:0000:86:00): 31, pid=8202, Ch 00000002, intr 00000000. MMU Fault: ENGINE HOST2 HUBCLIENT_ESC faulted @ 0x1_00c00000. Fault is of type FAULT_PTE ACCESS_TYPE_VIRT_READ
[  523.293449] NVRM: Xid (PCI:0000:86:00): 31, pid=241421, Ch 00000050, intr 00000000. MMU Fault: ENGINE HOST0 HUBCLIENT_ESC faulted @ 0x1_0b220000. Fault is of type FAULT_PTE ACCESS_TYPE_VIRT_READ
[  931.584379] NVRM: Xid (PCI:0000:86:00): 31, pid=300693, Ch 00000030, intr 00000000. MMU Fault: ENGINE GRAPHICS HUBCLIENT_SCC_NB faulted @ 0x1_00c65000. Fault is of type FAULT_PTE ACCESS_TYPE_VIRT_WRITE

run your codes with compute-sanitizer

also see here and here.

this may also be of interest