"Failed to apply atomic modeset" + display hang on fullscreen game launch with 530.41.03, 1070 GTX, Wayland, Gnome 43.4, Manjaro

I’m on Manjaro’s stable branch, which recently updated its Nvidia driver to the 530 series. Ever since then my system hangs whenever many (not all) fullscreen games launch. The display stops updating altogether, and input doesn’t affect the screen (though as far as I can tell, input is still being handled - see below). I can’t tell if this issue is caused by the driver or something further up the stack, so I’m starting here.

When these hangs happen, I can press ctrl+alt+F3 to switch to tty3 and login, and see the game process and whatever else still running in htop. I can see recent errors with journalctl -b, and one line appears consistently in red right around the time the hang happens:

kernel: [drm:nv_drm_atomic_commit [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to apply atomic modeset. Error code: -22

Killing the game process from tty3 doesn’t restore display responsiveness. Killing the gnome-shell process closes the Gnome session and returns me to tty1, and then I see the first rendered frame of the gdm login screen but it has also become unresponsive. So I’m stuck switching back to tty3 and rebooting.

The first game I ran into this with was Satisfactory, launched from Steam and running atop their whole proton/dxvk stack. The game renders via DX12 by default, but when I switched it to Vulkan, it launched without hanging. No Man’s Sky, another game that uses Vulkan, runs with no issues.

At first I thought it might be a Steam / Proton issue, but then I tried launching GZDoom, a Linux-native non-Steam game that supports both Vulkan and OpenGL. Launching GZDoom fullscreen caused the same hang. When I set it to launch windowed, it did not hang. And from within the windowed application, I could set it in and out of fullscreen mode without hanging. So the specific combination of starting the game / creating the window + going fullscreen may be a key feature of this issue.

As far as the Wayland side of my config, I followed the standard steps for enabling a Gnome Wayland session on my system, and definitely have the nvidia-drm.modeset=1 kernel paramter set. Firefox is running in Wayland with no issues so far. All the games I tried were running via Xwayland, of course. In general I can’t really think of anything about my config that is customized or non-standard.

3 Likes

Wish I could help, but I have the same issue, same error code. Running Arch 6.2.9. Nvidia 530.41.03.

x11 is fine, Wayland freezes on full screen Doom Eternal, but I can hear the music playing and mouse seems to be touching some of the menu options, but only way out is to do a hard reset. Other games like Cultist Simulator do not freeze.

=/

I have the same issue, fullscreen games just freeze the screen, but i still hear sounds. Im on arch, latest driver and using Hyprland.

I’m trying to investigate this a bit more deeply since it seems very much like it could be a driver-or-library-level bug. A couple findings:

  • GZDoom has 3 renderers: Vulkan, OpenGL, and OpenGL ES (meant for older hardware). This hang happens when I launch GZDoom fullscreen with either the Vulkan or OpenGL renderers, but not when I launch with the GLES renderer. Not sure why; I’m not super familiar with how different the code paths for creating the window, draw context etc are for those.
  • I wrote a small Python + SDL2 (since that’s what I’m most familiar with) program for creating different kinds of fullscreen X11 (thus Xwayland) windows to see if I could trigger the hang with the smallest amount of code possible, but wasn’t able to. I’m guessing that’s down to whatever SDL2 is doing in its code.
  • This happens on a wide enough variety of Steam games that it probably doesn’t seem like it’s something wrong tons of shipped games are doing! Hence my conclusion that it’s a bug in the drivers or somewhere further up the stack. I’d be happy to be proven wrong though, so long as this hang is fixed for everyone experiencing it! Nvidia’s road to fully supporting Wayland has been a long and bumpy road already.

Further digging, I found this Xwayland bug report that has the right things in common (Nvidia 530 + Xwayland + fullscreen = hangs requiring full session restart):

I’ll keep an eye on that thread and report back here if it turns out it was a bug in Xwayland.

Some developments from that xwayland thread: it looks like, in Gnome at least, disabling direct scanout avoids the hang:

I still can’t tell if that makes it an Nvidia driver issue or something else. Another commenter there reports it fixed in Gnome 44, so maybe I’ll just use the disable direct scanout workaround for now and wait for the Arch repos to get that.

Downgrading your drivers sounds like more work. Did you try keeping 530 and just disabling direct scanout? That was a viable workaround for me; it avoided the hang with no apparent side effects. For anyone else coming to this thread, I’ll paste the steps given in that xwayland issue thread here:

  1. Type Alt+F2
  2. In the dialog type lg to open Looking Glass
  3. Type Meta.disable_unredirect_for_display(global.display) and press Enter
  4. Press Escape to exit Looking Glass

Note that you’ll have to do this again each time you reboot.