Hi,
Please try to set nFrameMinQp and nFrameMaxQP in a limilted range and see if it helps. Also for balancing bitrate and video quality, you may try to configure CBR + virtual buffer size. Here is a topic for reference:
Random blockiness in the picture RTSP server-client -Jetson TX2 - #5 by DaneLLL