Encoder produces much larger frames than requested

The attached graph shows the ratio between the frame size we request using setBitrate API (On NvV4L Encoder) and what we actually receive from the encoder. The blue dots are P-Frames and the Yellow ones are IDR.
The issue here is that we would expect the average per second will be around 1, meaning that if we request 2.5 Mbps from the encoder, IDRs will be larger, P-Frames will be smaller, but they will average at what we requested.
But in the graph we see t’o distinct scenario where we receive twice and even four times the bitrate we request - and on a stream of P-Frames.
This behaviour is highly unexpected.
Have you seen such behaviour or have any idea what might produce such large P-Frames?


For information, do you use Jetpack 4.6.4 or 5.1.2? Would like to confirm which release you are using.

Using Jetpack 4.4 [L4T 32.4.3]