Issues running OpenMAX AVC decoder on Tegra 3

It seems hat the decoder will start to produce decoded frames after several seconds (usually much more than 5 seconds). Is this normal? I’m trying to make a conferencing type system, so real time performance is crucial. I see the decoder detects the resolution correctly within 200mSec, but then it takes a while to actually get the frames. Any suggestions?
Arik