Setting modeset=1 doesn’t actually install a framebuffer console. All it really does is enable the DRIVER_MODESET capability flag in the nvidia-drm devices so that DRM clients can use the various modesetting APIs. In addition to allowing clients that talk to the low-level DRM interface to work, it’s also necessary for some PRIME-related interoperability features.
The downside, if you want to call it that, is that loading nvidia-drm with modeset=1 causes it to configure and initialize all GPUs immediately rather than waiting for a client to open the /dev/nvidia* device files. This means that some options that require a userspace application to configure them before the GPUs are initialized won’t work if they were already configured by nvidia-drm. The big example at the moment is SLI Mosaic, which is enabled by the X driver if /etc/X11/xorg.conf says it should be.