Hi,
Please set the properties in nvv4l2h264enc and try again:
maxperf-enable : Enable or Disable Max Performance mode
flags: readable, writable, changeable only in NULL or READY state
Boolean. Default: false
We suggest set maxperf-enable=1.
idrinterval : Encoding IDR Frame occurance frequency
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 256
We suggest set idrinterval=30.
And not sure if the linkage rtph264pay ! fpsdisplaysink is valid. Since we expect output of rtph264pay is packet-based, not frame-based. Not sure if it reports correct frame rate in this linkage, but it looks like you have seen correct result with x264enc.