Hi,
Please refer to this post for setting encoder properties:
Random blockiness in the picture RTSP server-client -Jetson TX2 - #5 by DaneLLL
To get balance between bitrate and quality, we would suggest run in CBR mode and adjust virtual buffer size.