To me, the cause is [BUG] linux driver fails to remove framebuffer device when HDMI cable plugged out - #2 by Aetherall
where the nvidia-drm is either not receiving or handling hotplug events.
It seems that the linux kernel had some changes regarding framebuffers lifecycle with drm, maybe changes required to handle hotplug events and cleanup framebuffers have yet to be implemented / released
EDIT: issue opened at RTD3 dont allow gpu to sleep after a monitor has been plugged and unplugged on prime reverse sync · Issue #759 · NVIDIA/open-gpu-kernel-modules · GitHub