I don’t think that’s a problem. The frame rate drop because the AE alg need adjust to get longer time of exposure to get better preview.
You can try the v4l2-ctl to check if any frame rate control to set it as fixed frame rate to try, otherwise you may need consult with vendor.