[615.71.09] RM rwlock deadlock after MST hotplug: flip retry loop → "Lost display notification" → nvidia-modeset/nvidia-smi/driftwm stuck in D-state,

GPU: NVIDIA GeForce RTX 5070 Ti Laptop GPU
Driver: 615.71.09 (open kernel modules)
Kernel: 7.2.9-200.fc44.x86_64, Fedora 44
Compositor: driftwm (Wayland/smithay)
Display: eDP-1 (internal, direct), HDMI-A-1 (direct), and one MST hub on a DP port presenting as DP-3 in single-stream mode or DP-4 + DP-5 in dual-stream mode.

Summary:
After hotplug events on the MST hub, the system enters an unrecoverable state. Kernel log stops, nvidia-smi hangs indefinitely, VT switch (Ctrl+Alt+F3) does not work, but the machine is still pingable over SSH. nvidia-bug-report.sh hangs inside nvidia-smi. Only a hard power-off recovers the machine.

Trigger path:

  1. The MST hub re-enumerates its topology between single-stream and dual-stream modes: DP-3 (single-stream) goes disconnected/disabled, while DP-4 and DP-5 (dual-stream) come online. eDP-1 and HDMI-A-1 remain connected.

  2. Compositor requests modes via drm_mode_getconnector.

  3. DRM enters an atomic-commit retry loop for DP-4 and DP-5 (same CRTC, same mode, same FB reapplied ~5 times at 23:27:40).

  4. 3 seconds later driver gives up: nvidia-modeset: WARNING: GPU:0: Lost display notification (0:0x00000000); continuing.

  5. RM rwlock deadlock — see stacks below.

  6. TTY dead, nvidia-smi hangs forever, only hard reset works.

Specific trigger: The hub was switched from single-stream (DP-3) to dual-stream (DP-4 + DP-5) mode. This is a topology change on the MST hub side, and appears to be the specific operation that deadlocks the NVIDIA RM.

Deadlock evidence — kernel stacks captured live:

driftwm (compositor, victim — hung in mode probe):

nvkms_ioctl_from_kapi+0x58/0xe0 [nvidia_modeset]
GetDisplayMode+0x6e/0x210 [nvidia_modeset]
nv_drm_connector_get_modes+0xd5/0x180 [nvidia_drm]
drm_helper_probe_single_connector_modes+0x1f1/0x690
drm_mode_getconnector+0x34b/0x520
drm_ioctl_kernel+0xae/0x100
drm_ioctl+0x2e0/0x560

opera (DRM fd close path, caught in the deadlock):

os_acquire_rwlock_write+0x26/0x40 [nvidia]
portSyncRwLockAcquireWrite+0x10/0x40 [nvidia]
rmapiLockAcquire+0x264/0x350 [nvidia]
serverTopLock_Prologue+0x5c/0xe0 [nvidia]
serverFreeResourceTree+0x4a1/0x970 [nvidia]
rmapiFreeWithSecInfo+0x12b/0x2b0 [nvidia]
nvkms_call_rm+0x4c/0x80 [nvidia_modeset]
nvKmsKapiFreeSemaphoreSurface+0x44/0x90 [nvidia_modeset]
__nv_drm_semsurf_fence_ctx_destroy+0xca/0x250 [nvidia_drm]
drm_gem_object_release_handle+0x7f/0xf0
drm_gem_release+0x20/0x30
drm_file_free+0x213/0x2a0
drm_release+0x5c/0xe0

nvidia-smi #1 (RM free path, writer lock):

os_acquire_rwlock_write+0x26/0x40 [nvidia]
portSyncRwLockAcquireWrite+0x10/0x40 [nvidia]
rmapiLockAcquire+0x264/0x350 [nvidia]
serverTopLock_Prologue+0x5c/0xe0 [nvidia]
serverFreeResourceTreeLockAndFindResource+0x30/0x150 [nvidia]
rmapiFreeWithSecInfo+0x12b/0x2b0 [nvidia]
RmIoctl+0x419/0xe30 [nvidia]

nvidia-smi #2 (RM acquire path, reader lock):

os_acquire_rwlock_read+0x26/0x40 [nvidia]
portSyncRwLockAcquireRead+0x10/0x40 [nvidia]
rmapiLockAcquire+0x2ac/0x350 [nvidia]
serverTopLock_Prologue+0x5c/0xe0 [nvidia]

All four converge on portSyncRwLock. driftwm wchan: nvkms_ioctl_from_kapi, opera wchan: nvkms_close, both nvidia-smi in D state. VT switch dead, nvidia-smi exits 124 on timeout.

Attached:

Update: the deadlock is fixed in driver 615.78.08 (tested on kernel 7.2.10-200.fc44
with the same MST hub and driftwm). The single-stream ↔ dual-stream re-enumeration
no longer triggers “Lost display notification” and no longer deadlocks the RM.

Issue can be closed. Thanks.