I’m on Manjaro’s stable branch, which recently updated its Nvidia driver to the 530 series. Ever since then my system hangs whenever many (not all) fullscreen games launch. The display stops updating altogether, and input doesn’t affect the screen (though as far as I can tell, input is still being handled - see below). I can’t tell if this issue is caused by the driver or something further up the stack, so I’m starting here.
When these hangs happen, I can press ctrl+alt+F3 to switch to tty3 and login, and see the game process and whatever else still running in htop. I can see recent errors with journalctl -b, and one line appears consistently in red right around the time the hang happens:
kernel: [drm:nv_drm_atomic_commit [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to apply atomic modeset. Error code: -22
Killing the game process from tty3 doesn’t restore display responsiveness. Killing the gnome-shell process closes the Gnome session and returns me to tty1, and then I see the first rendered frame of the gdm login screen but it has also become unresponsive. So I’m stuck switching back to tty3 and rebooting.
The first game I ran into this with was Satisfactory, launched from Steam and running atop their whole proton/dxvk stack. The game renders via DX12 by default, but when I switched it to Vulkan, it launched without hanging. No Man’s Sky, another game that uses Vulkan, runs with no issues.
At first I thought it might be a Steam / Proton issue, but then I tried launching GZDoom, a Linux-native non-Steam game that supports both Vulkan and OpenGL. Launching GZDoom fullscreen caused the same hang. When I set it to launch windowed, it did not hang. And from within the windowed application, I could set it in and out of fullscreen mode without hanging. So the specific combination of starting the game / creating the window + going fullscreen may be a key feature of this issue.
As far as the Wayland side of my config, I followed the standard steps for enabling a Gnome Wayland session on my system, and definitely have the nvidia-drm.modeset=1 kernel paramter set. Firefox is running in Wayland with no issues so far. All the games I tried were running via Xwayland, of course. In general I can’t really think of anything about my config that is customized or non-standard.