Ubuntu 20.04 - not able to run 6/7 monitor setup with one X screen

Yes, this changes something.

The whole environment is crashing. First graphic card(0) all monitors are turning off and some console pointer (underscore) coming in, looks like whole screen session crashed while upper three monitors are still active. After around seconds login screen is shown and all monitors behave normal. But GDM is not stable in this state and login will fail, I required to manually restart it by opening terminal and restart the service. After that it worked again.

I did tried both, same outcome:

__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0 glxgears
__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0 __GLX_VENDOR_LIBRARY_NAME=nvidia glxgears

Only thing I found so far about logging is in dmesg really prominent, multiple times in row:

[drm:nv_drm_master_set [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000200] Failed to grab modeset ownership
[drm:nv_drm_master_set [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Failed to grab modeset ownership

Is there anything else I can look into? Or do you think this thing will not be able to run yet that easily?