The first call to IDXGIOutput::GetDisplayModeList after monitor hotplug freezes the system

And many other related operations, like EnumDisplaySettings, Direct3DCreate9(Ex), or simply opening the Windows display setting, will freeze the system for about 1 second.

Driver version: 555.99
OS version: Windows 11 23H2 22631.3737
GPU: NVIDIA GeForce RTX 4080 SUPER
Monitors setup: Dual 3840x2160@60Hz DisplayPort, one appears disconnected after display timeout

A Windows Performance Recorder trace shows it blocks on the nvidia kernel driver, the trace file is available upon request.