Black screen when booting with modeset on Ubuntu

I am trying to get modesetting to work on my 1070 by adding
options nvidia-drm modeset=1
to /etc/modprobe.d/modeset.conf

(this is all in the effort of getting udev messages when display port connected/disconnected)

This causes a black screen after the luks decrypt screen.

I am able to ssh into the system and I ran nvidia-bug-report

nvidia-bug-report.loggz (287.6 KB)

I also tried with 440.64 no luck.

I made the following modifications as well:
EnableWayland=false in gdm3/custom.conf

blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off

in modprobe.d/blacklist-nouveau.conf
and ran sudo update-initramfs -u

the behavior is still the same.