More DisplayPorts connected, slower response after changing display settings


I have a workstation with 3x Quadro P4000. It is running Windows 10 Enterprise with latest NVIDIA drivers. I noticed that with multiple DisplayPort connected, the response time after changing the display settings (e.g. resolution, or plugin/plugout another monitor), is really long. The response time here I mean after implying the change, till the display is working properly again. As the display is kind of hanging first, but the mouse can move smoothly (I guess Windows in running ok). This is also getting worse with more DisplayPort connected:
4 ports: < 10s
8 ports: ~20s
12 ports: >50s

I tried also with Windows 10 Pro, and with 3x RTX4000, and the behaviors are same. I tried to monitor the CPU and memory usage and it looks like they are not the limiting factor (the display is hanging during the change, but after that I see the plot over the time). So I would like to ask if this is expected behavior. Thank you very much in advance.