No display after installing nvidia driver

We recently upgraded our system from ubuntu 16.04 to 20.04. And after reinstalling nvidia-driver, the display stopped working - black screen after grub. I’m still able to ssh into the machine.

I can see the output of nvidia-smi and it does show the driver 515 and cuda 11.7 installed. However, the nvidia-persistenced.service shows it failed. I doubt it’s a permission problem because the ll -h /dev/nvidia* shows the permissions are 660. I tried to change the permission by adding options nvidia NVreg_DeviceFileMode=0666 in cat /etc/modprobe.d/nvidia.conf. But it doesn’t change anything. The permission stays as 660 forever.

Any help is appreciated!

The nvidia-bug-report is attached:
nvidia-bug-report.log.gz (1.0 MB)

NVIDIA(GPU-0): Push buffer DMA allocation failed
Please remove ‘nopat’ kernel parameter.

Thank you for your suggestion. I removed the nopat kernel parameter, did update-grub, reboot, nothing changed. I googled around for this error and also added nvidia-drm.modeset=1. Nothing changed either.

Here’s the new bug report:
nvidia-bug-report.log.gz (1.2 MB)