Nvidia driver crashes when external HDMI 2.1 Monitor is turned off

I just got an LG 48CX and am using HDMI 2.1 to connect to it at 4k@120hz. It is all working, except for this one issue. Whenever I turn off the monitor, then turn it back on again, the nvidia driver crashes and all signal is lost. I have to reboot my machine to fix it.

Unfortunately nvidia-bug-report.sh is not working for me, so I will try to provide as much info as I can manually. If you need the bug report I’ll spend some time trying to get that working later. But I need to get work done right now :).

kernel logs:

Nov 01 11:37:29 thor kernel: nvidia-modeset: ERROR: GPU:0: HDMI FRL link training failed.
Nov 01 11:37:32 thor kernel: nvidia-modeset: WARNING: GPU:0: Lost display notification (0:0x00000000); continuing.
Nov 01 11:37:56 thor kernel: nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 0x0000c67d:0:0:1098
Nov 01 11:38:08 thor kernel: nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 0x0000c67e:0:0:1114
Nov 01 11:38:28 thor /nix/store/d0gfx95svzaknz635sqkxhpzfb305pxg-gdm-3.34.1/libexec/gdm-x-session[2042]: (WW) NVIDIA: Wait for channel idle timed out.

nvidia-smi:

Sun Nov  1 11:57:18 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.28       Driver Version: 455.28       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3080    Off  | 00000000:09:00.0  On |                  N/A |
|  0%   49C    P8    35W / 340W |   1216MiB / 10014MiB |      6%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

uname -a

Linux thor 5.4.72 #1-NixOS SMP Sat Oct 17 08:11:24 UTC 2020 x86_64 GNU/Linux

Any suggestions for workarounds would be appreciated. I’ve tried a few things without any luck, including trying to toggle to a TTY, using nomodeset as a kernel parameter, and unplugging / plugging the HDMI cable back in. However nothing helps!

I’m using gnome 3 which uses wayland I believe.

More experimentation shows the steps needed for this to happen:

  1. Screen blanks after 5 minutes (dpms off)
  2. TV turns off on its own after another 5 minutes
  3. I wake up the screen (dpms on) by pressing a key on the keyboard <- nvidia driver crashes with the above messages

A workaround is to make sure the TV is turned on first before waking up the signal output. Seems like it crashes because it can’t negotiate the HDMI 2.1 connection when the TV is off after blanking.