What kind of output do you get if you use the interactive mode of simply using displaymodeselecotr --gpumode? It should show you all available modes.
Also, for Windows, you will need to change the driver model from TCC to WDDM as well by using nvidia-smi. Check the documentation that comes with the displaymodeselector tool for more details.
Regarding windows, I think the doc says first to change the gpumode to “graphics” and then do thing to the Windows Driver… No?
(this is the smi output with everything default)
Regarding the order of use of the two tools I think this is interchangeable.
But looking at your output I am at a loss right now as to why this does not work. I’ll reach out internally to get more eyes on this and how to debug further.