System hangs for half a second every once in a while in multiple monitor setup when one of them is turned off

Linux + X11,

With two monitors physically connected, but one of them turned off, say by xrandr of via settings provided by desktop environment, the system would hang for half a second every once in while, usually a couple time a minutes.

Specifically mouse, keyboard input would stop working for like half second in above scenario couple times a minutes.