I wanted to try the 510.47 and 510.54 drivers but I find myself with the same problem.
With nvidia-drm.modeset=1 I start in graphic mode but the tty consoles ( Ctrl + Alt + F1 F6) do not work (black screen).
Without nvidia-drm.modeset=1 kernel loading stops after the line: nvidia-modeset: Loading Nvidia Kernel …
It seems to be a framebuffer problem.
Trying kernel 5.17 rc4, 5.15 or 5.16 didn’t solve anything.
$ dmesg | grep vga
[ 0.289730] pci 0000:01:00.0: vgaarb: setting as boot VGA device
[ 0.289730] pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[ 0.289730] pci 0000:01:00.0: vgaarb: bridge control possible
[ 0.289730] vgaarb: loaded
[ 8.280376] snd_hda_intel 0000:01:00.1: Handle vga_switcheroo audio client
[ 9.006685] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem