I am having issue with the nvench264 Media Foundation transform. I am trying to get a screen streaming application and I can make it work if I have only one screen configured. But when I set the MF_MT_FRAME_SIZE attribute on the output type to be the resolution of my virtual screen (witch is the sum of my double screen configuration), the SetOutputType method returns the MF_E_UNSUPPORTED_D3D_TYPE error. If I let the attribute to the default value (the resolution of my primary screen device), the method works well and the application is able to encode to h264. Also if I use some software tranform, I can again make the application working with my double screen config but the CPU usage is very high. That’s why I want to move this work to my GPU.
I know that this is possible because tools like OBS studio are able to encode my double screen using that same nvenc264 driver with no issue.
Can anyone provide me with some guidance on how to make this MFT work with other resolution/configuration than the default one ?
If you need more infos please ask. Thanks for your reply.