Hi,
There are other properties in x264enc plugin: x264enc
Please check and try to configure the properties. This would also need other users to share experience.
For using hardware encoder, we will suggest set to CBR mode and set virtual buffer size like:
Random blockiness in the picture RTSP server-client -Jetson TX2 - #5 by DaneLLL
Would need other users to share how to configure the same in x264enc