H.264 CBR rate control produces lower than expected bitrates

I’m encoding H.264 video using CBR rate control and specifying the target bitrate using the averageBitRate parameter. With 512x512 resolution input, everything works as expected but with 256x256 input I get much lower than expected bitrates. The QP values used by the encoder appear to be well above zero (average QP per frame is 14 for one sample) and also double checked that constant zero QP encoding produces higher bitrates. I use NV_ENC_H264_PROFILE_HIGH_444_GUID and use the input buffer format NV_ENC_BUFFER_FORMAT_AYUV. Concerning rate control parameters, I set the multipass to NV_ENC_TWO_PASS_FULL_RESOLUTION. I have tried different presets as well as tuning infos to no avail. I’m using driver version 462.59 (newer drivers don’t handle AYUV input properly).

Am I missing some parameter setting or what is going on?