Hi,
The issue may not be in decoding/rendering. Decoder has to receive SPS/PPS and first complete IDR frame and then start decoding. It may be the IDR interval is too big in the h264 stream. For example, if IDR interval=60 frames, you will see maximum 2 second latency in 30 fps case. Suggest you check if you can set the source to IDR interval=15 for a try.