Hi,
You may check glass-to-glass latency first. If the latency is from source, may check to reduce it.
Since video decoding begins from IDR frames, setting small idrinterval should help. For idrinterval=15 in 30 fps, the theoretical latency is around 500-600ms.