Hi,
We have a reference setup with hardware encoder. Please take a look at
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL
In receiver side it cannot guarantee to receive the stream from very beginning, so decoding begins from first complete IDR frame and needs to have SPS/PPS with IDR frames.