Tegra3 -- Problem: High latency for hardware decoding

Hardware:Nexus 7
OS Version: Android 4.1.2
Problem:
I use OpenMAX IL to decode H.264 video stream(640x480). Between I give a raw H.264 frame and I receive a YUV frame, it is about 2~4 seconds. The YUV frame is correct.
Question:
Is it really need so much time? Or I have made wrong configurations?