Nvv4l2decoder/nvv4l2h264enc with gst-rtsp-server latency increasement

Hi,
IDR frames have larger size than P frames, so it may be related to network bandwidth. Not sure if it helps but you may try:

Furthermore, you can set this property to have stricter bitrate:

  EnableTwopassCBR    : Enable two pass CBR while encoding
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false