Missing frames from I420 -> H.264 via omxh264enc on R28.2.1?

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Can someone explain to me:

nvcamerasrc do-timestamp=true enable-meta=true fpsRange=“30.0 30.0” auto-exposure=1 aeLock=true sensor_id=0 name=nvcam0 ! video/x-raw(memory:NVMM), width=(int)4104, height=(int)3046, format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=2 ! video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)I420 ! queue ! identity name=tap0 ! omxh264enc control-rate=2 profile=8 bitrate=30000000 ! identity name=tap1 ! …

Why is the number of “handoff” callbacks on tap0 != tap1? i.e. there seems to be missing frames after the encoder? Why?

It’s usually off by one or two frames, i.e after the encoder I’m short a frame or two, sometimes three. I think the longer I let it play, the more I see one or two frames go missing every so often. Is this just a bug in the encoder?

NOTE: jetson_clocks.sh and nvpmodel -m 0 are both set.

ANOTHER ISSUE:

On pipeline stop, I see more packets in-core on my tap callback than what’s in the stream (ffprobe confirms). There seems to be some kind of flush bug on pipeline PLAYING → NULL state.

2

Hi

There are two reasons:
1 Encoder outputs codec specific data(with OMX_BUFFERFLAG_CODECCONFIG) at the very beginning. You shall see

Input                   Output
I420 frame 1(buffer 1)  Codec specific data(buffer 1)
I420 frame 2(buffer 2)  h264 frame 1(buffer 2)
I420 frame 3(buffer 3)  h264 frame 2(buffer 3)
I420 frame 4(buffer 4)  h264 frame 3(buffer 4)
...

2 Encoder keeps certain number of input frames as reference frames, so you may see frame (n) being returned back after frame (n+2) is fed into encoder.

3

EDIT: Wait, reference frames should be ultimately returned by the encoder as an I-frame right?

How many codec specific data buffers will I see? Just one or more than one? (I see more than one) The bottom line is as my recording progresses, I always see more in-core frames than what gets flushed out to my file (the actual H.264 stream) and it gradually gets bigger and bigger.

Is there anyway to determine what I420 frame is codec specific and what will become ultimately an H.264 frame?

4

Hi,
Only the first buffer is codec specific data.
Not sure how identity plugin works. It is very certain the encoder does not drop any frame. You can set ‘num-buffers=300’ in nvcamerasrc ad dump the encoded h264 stream. If you inspect encoded h264 stream through jm decoder, you will see encoded frame number is exactly 300.

5

I am able to reproduce that “num-buffers=300” does infact land 300 frames in the stream.

However, when I take num-buffers off, the results are very inconsistent:

This is what I do:

nvcamerasrc do-timestamp=true enable-meta=true fpsRange=“30.0 30.0” auto-exposure=1 aeLock=true sensor_id=0 name=nvcam0 ! identity name=countme0 ! … ! omxh264enc control-rate=2 profile=8 bitrate=30000000 ! identity name=countme1 …

Both countme0 and countme1 elements get the same “handoff” handler below, one passes 0 the other 1:

def on_handoff(identity, buf, id):
   """ id = 0 -> I420, 1 -> H.264 """
   return counts[id] += 1

That should count the number of GstBuffers that propagate through the pipeline. I have confirmed with the gstreamer team that my understanding is correct. And in fact if I wait till EOS with num-buffers=300 I see 300 handoff callbacks before and after the encoder.

HOWEVER, when I do a start/stop on my pipeline after a few seconds here are my results:

Number of frames outputted by nvcamerasrc: 574
Number of frames outputted by encoder: 569
Number of frames in the stream: 567

Number of frames in the stream is counted via:

ffprobe -v fatal -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 /path/to/file.mkv

So given that “num-buffers=300” is true, it looks like the encoder or something else has a drain issue, i.e. on the Gst play state moving from PLAYING → NULL, not all the buffers are being drained out to the file causing frame loss.

The reason why I say this is because when I actually wait for an EOS flush when num-buffers=X, it works. But if I stop the pipeline via:

recorder_pipeline.set_state(Gst.State.NULL)

It looks like not all the buffers are being flushed to the actual stream. MOREOVER, I am still short 5 frames above between I420 and H.264. Let’s pretend the first frame is the codec one so make it 4. Where did those 4 frames go? If I had to guess…the omx encoder is not flushing everything on state change to NULL but works when an EOS occurs?!

Try it! You can reproduce this very easily.

6

Hi,
Please wait for EOS before changing to NULL state. A similar sample is in
[url]CLOSED. Gst encoding pipeline with frame processing using CUDA and libargus - Jetson TX1 - NVIDIA Developer Forums
Without waiting for EOS, one issue is that qtmux does not complete the writing and generates an invalid mp4.

7

That does it. I have to emit a Gst.Event.new_eos() myself, catch, and then change state. That seems to give me consistent frame numbers in the stream.

Thanks again for your support DaneLLL.