"Failed to apply atomic modeset" + display hang on fullscreen game launch with 530.41.03, 1070 GTX, Wayland, Gnome 43.4, Manjaro

I’m on Manjaro’s stable branch, which recently updated its Nvidia driver to the 530 series. Ever since then my system hangs whenever many (not all) fullscreen games launch. The display stops updating altogether, and input doesn’t affect the screen (though as far as I can tell, input is still being handled - see below). I can’t tell if this issue is caused by the driver or something further up the stack, so I’m starting here.

When these hangs happen, I can press ctrl+alt+F3 to switch to tty3 and login, and see the game process and whatever else still running in htop. I can see recent errors with journalctl -b, and one line appears consistently in red right around the time the hang happens:

kernel: [drm:nv_drm_atomic_commit [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to apply atomic modeset. Error code: -22

Killing the game process from tty3 doesn’t restore display responsiveness. Killing the gnome-shell process closes the Gnome session and returns me to tty1, and then I see the first rendered frame of the gdm login screen but it has also become unresponsive. So I’m stuck switching back to tty3 and rebooting.

The first game I ran into this with was Satisfactory, launched from Steam and running atop their whole proton/dxvk stack. The game renders via DX12 by default, but when I switched it to Vulkan, it launched without hanging. No Man’s Sky, another game that uses Vulkan, runs with no issues.

At first I thought it might be a Steam / Proton issue, but then I tried launching GZDoom, a Linux-native non-Steam game that supports both Vulkan and OpenGL. Launching GZDoom fullscreen caused the same hang. When I set it to launch windowed, it did not hang. And from within the windowed application, I could set it in and out of fullscreen mode without hanging. So the specific combination of starting the game / creating the window + going fullscreen may be a key feature of this issue.

As far as the Wayland side of my config, I followed the standard steps for enabling a Gnome Wayland session on my system, and definitely have the nvidia-drm.modeset=1 kernel paramter set. Firefox is running in Wayland with no issues so far. All the games I tried were running via Xwayland, of course. In general I can’t really think of anything about my config that is customized or non-standard.

3 Likes

Wish I could help, but I have the same issue, same error code. Running Arch 6.2.9. Nvidia 530.41.03.

x11 is fine, Wayland freezes on full screen Doom Eternal, but I can hear the music playing and mouse seems to be touching some of the menu options, but only way out is to do a hard reset. Other games like Cultist Simulator do not freeze.

=/

I have the same issue, fullscreen games just freeze the screen, but i still hear sounds. Im on arch, latest driver and using Hyprland.

I’m trying to investigate this a bit more deeply since it seems very much like it could be a driver-or-library-level bug. A couple findings:

  • GZDoom has 3 renderers: Vulkan, OpenGL, and OpenGL ES (meant for older hardware). This hang happens when I launch GZDoom fullscreen with either the Vulkan or OpenGL renderers, but not when I launch with the GLES renderer. Not sure why; I’m not super familiar with how different the code paths for creating the window, draw context etc are for those.
  • I wrote a small Python + SDL2 (since that’s what I’m most familiar with) program for creating different kinds of fullscreen X11 (thus Xwayland) windows to see if I could trigger the hang with the smallest amount of code possible, but wasn’t able to. I’m guessing that’s down to whatever SDL2 is doing in its code.
  • This happens on a wide enough variety of Steam games that it probably doesn’t seem like it’s something wrong tons of shipped games are doing! Hence my conclusion that it’s a bug in the drivers or somewhere further up the stack. I’d be happy to be proven wrong though, so long as this hang is fixed for everyone experiencing it! Nvidia’s road to fully supporting Wayland has been a long and bumpy road already.

Further digging, I found this Xwayland bug report that has the right things in common (Nvidia 530 + Xwayland + fullscreen = hangs requiring full session restart):

I’ll keep an eye on that thread and report back here if it turns out it was a bug in Xwayland.

Some developments from that xwayland thread: it looks like, in Gnome at least, disabling direct scanout avoids the hang:

I still can’t tell if that makes it an Nvidia driver issue or something else. Another commenter there reports it fixed in Gnome 44, so maybe I’ll just use the disable direct scanout workaround for now and wait for the Arch repos to get that.

Downgrading your drivers sounds like more work. Did you try keeping 530 and just disabling direct scanout? That was a viable workaround for me; it avoided the hang with no apparent side effects. For anyone else coming to this thread, I’ll paste the steps given in that xwayland issue thread here:

  1. Type Alt+F2
  2. In the dialog type lg to open Looking Glass
  3. Type Meta.disable_unredirect_for_display(global.display) and press Enter
  4. Press Escape to exit Looking Glass

Note that you’ll have to do this again each time you reboot.

how is this still an issue nvidia? it’s been 3 months and i still can’t open a fullscreen xwayland app without the whole driver crashing. the workaround above fixes that so at least you’ve got something to work with, fix this.

Hi All,
I tried to reproduce issue locally on couple of test systems but no luck so far.

HP OMEN by HP 25L Gaming Desktop GT15-0xxx + Ubuntu 22.04.1 LTS + kernel 5.19.0-46-generic + NVIDIA GeForce RTX 3080 + Driver 530.41.03 + GBT AORUS FI27Q-P

Dell Precision T7610 + Arch Linux + kernel 6.4.1-arch1-1 + NVIDIA GeForce GTX 1070 + Driver 530.41.03 + BenQ EL2870U

I cross verified default for game Satisfactory and it is using DX12.
I also changed display mode to Full Screen mode and launched game multiple times.

Can someone please check if issue persists with driver [535.54.03] and share nvidia bug report from repro state and also confirm the repro frequency.

nvidia-bug-report.log.gz (266.7 KB)

driver version 535.54.03

arch linux 6.4.3.arch1-2, gtx 1060, iiyama g-master G2530HSU

able to reproduce crash without issue.

Thaks for sharing the logs, can you please confirm exact repro steps.
I will try the same on my setup.

any fullscreen xwayland game or app i open will crash the driver leaving me unable to move my mouse, keyboard or even change tty’s. tested games = half life 1, ftl, system shock enhanced, cruelty squad, kenshi (sometimes?). all of these games work in windowed or if applied the workaround above in the thread. omega strikers for example doesn’t crash even when fullscreen because it’s an unreal engine game and as far as i know they introduced wayland implementation.

after force-shutting down the pc journalctl says: Jul 16 22:38:10 arch kernel: [drm:nv_drm_atomic_commit [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to apply atomic modeset. Error code: -22

seems to be fixed with 535.86.05-2.

spoke too soon, keeps happening under certain circumstances. sometimes it crashes, sometimes it doesn’t.

@hmpfkafka
Is it still happening as per the steps outlined by you earlier ?

yup, except some games work and some don’t. for example dishonored, death to spies, thief gold work with 0 problems, but half life 1 and celeste crash the video driver.

I am also still experiencing this on Arch, driver version 535.86.05, linux 6.4.2 with a GTX 1060. I can consistently reproduce this by opening DaVinci Resolve (which fullscreens).

I have filed a bug internally 4218400 for tracking purpose.
However I am still struggling to achieve local repro.
Will try latest recommended steps and update further on it.

Installed fresh Manjaro Linux 23.0.0 on my test system and can reproduce issue locally where launching game “Satisfactory” in full screen mode completely hang display (black color).
However, when I switched to latest driver 535.86.05, I no longer reproduce issue while launching game “Satisfactory”.
I will try other games with latest driver 535.86.05 and update.

@hmpfkafka
@silvermight
Is it possible for you to test “Satisfactory” game with latest driver and share test results.

I tried another game DOOMEternal and was able to repro issue with driver 530.41.03.
However, when I upgraded it to 535.86.05, I am not seeing any display hang and game runs fine.

@Orion-JM
Can you try with latest driver 535.86.05 and share test results.