Performance difference between primary GPU and other GPUs in a muli-GPU system

I have two identical GPUs connected to my machine. I need to playback a set of images after some computation on GPU ( ComputeShader,Vertex shader and Pixel shader execution).

I am using Dx 11 for the rendering and image processing.

Issue:

If the direct X device (ID3D11Device) created with primary IDXGIAdapter1 ( GPU associated with my main display ) ==> Getting better performance ( 30 fps)

If I use my secondary GPU for device creation ==> Performance is degrading( less than 25 fps)

If I switch my main display to other GPU, then the performance also reverse.

Is this is an expected behavior? Can someone give some hint on the root cause of this behavior? Also is there any alternative to overcome this?