3-frames frame size change delay on bitrate change

I’m running HEVC encoder with a variable target bitrate (the app is slightly changed code from Samples). Each N frames (N = 100 or 50) I change target bitrate and call Reconfigure(). Target bitrate is being switched between 1Mbs and 5Mbs. All the frames are P-Frames (gop = -1).
Each time I do it I watch that next 3 frames after the reconfig call are encoded to the size which matches the previous bitrate (i.e. frame size tends to be of the same order of magnitude as it was for the previous bitrate). On the 4th frame after reconfig frame size adjusts to the expected one.
In gstreamer with h265 encoder plugin there’s no such behavior: the frame size adjusts immediately after the reconfig.
Could somebody please explain and advise what happens and how can I get rid of this frame size lag in my app? TIA