Tegra3 -- Problem: High latency for hardware decoding

Hardware:Nexus 7
OS Version: Android 4.1.2
I use OpenMAX IL to decode H.264 video stream(640x480). Between I give a raw H.264 frame and I receive a YUV frame, it is about 2~4 seconds. The YUV frame is correct.
Is it really need so much time? Or I have made wrong configurations?