How to set maximum bits per frame when encoding with omxh264enc?

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Given an input video called “input.yuv”, which is a set of raw YUV420p frames, I would like to produce an output h264 video with no individual frame exceeding “X” bits. An extra condition is that the output video must consist of only the initial I-frame and subsequent P-frames (i.e. iframeinterval > input frames, and no-B-Frames=false).

Since it seems like there are no other ways to control the maximum frame size beyond setting an average bitrate, I tested out the vbv-buffer size, which should theoretically specify the size of the virtual buffer while encoding. From my understanding, it should specify the maximum amount of bits in any one frame. It is defined as Virtual Buffer Size = vbv-size * (bitrate/fps), so at 10 Mbps and 30 fps, vbv-size=1 should produce a virtual buffer size of 333 kb.

Here is my GStreamer pipeline to encode the raw video.

BITRATE=10000000
GOPSIZE=400
VBVSIZE=30

gst-launch-1.0 filesrc location=input.yuv \
    ! videoparse width=1920 height=1080 framerate=2997/100 format=i420 \
    ! omxh264enc bitrate=$BITRATE no-B-Frames=false iframeinterval=$GOPSIZE vbv-size=$VBVSIZE \
    ! 'video/x-h264, stream-format=(string)byte-stream'    \
    ! h264parse    \
    ! qtmux        \
    ! filesink location=framesize-limit.mp4

And to inspect frames we can use ffprobe:

ffprobe framesize-limit.mp4 -show_frames | grep pkt_size

And further, to gather the frame sizes in bytes to a file:

ffprobe framesize-limit.mp4 -show_frames | grep pkt_size | cut -d = -f 2 > framesizes.csv

Experiment:

  1. Modulate vbv-size to 1 and 30.
  2. Modulate iframeinterval to 30 and over 300.
  3. Modulate bitrate to 10 Mbps and 500 kbps.
  4. Use ffprobe to dump frame data and inspect frame size

Results:
At a given iframeinterval and bitrate, vbv-size modulation does not change the resulting bits used to encode each frame. In fact, the frame sizes are identical. In the case of a high-motion video, the first 30 frames (1 second at 29.97 fps) are almost 50% over the specified bitrate. The first frame, for example, was over the 333 kb virtual buffer size, and was well over 400 kb.

I would like to know how I can cap the number of bits any given frame uses to some constant, say “X”. How can I set the maximum bits per frame when encoding with omxh264enc?

Notes:

  1. The polarity of the no-B-Frames seems backwards, but a test reveals that 'true' results in B-frames, while 'false' results in no B-frames.
2

Hello, jjohnston:
can you try the following pipeline:

gst-launch-1.0 filesrc location=input.yuv \
    ! videoparse width=1920 height=1080 framerate=2997/100 format=i420 \
    ! omxh264enc bitrate=$BITRATE control-rate=constant \
        no-B-Frames=false iframeinterval=$GOPSIZE vbv-size=$VBVSIZE \
    ! 'video/x-h264, stream-format=(string)byte-stream'    \
    ! h264parse    \
    ! qtmux        \
    ! filesink location=framesize-limit.mp4

br
ChenJian

3

Hello jachen,

Thanks for the response and suggestion. I’ve just run your suggested pipeline with the control-rate set to constant, and the GOP size as infinite. It looks like modulation of the vbv-size has an effect on the frame size now that the control-rate is constant. Beyond the first frame, the frames followed the expected maximum frame size threshold; however on the first frame for a vbv-size that results in a smaller virtual buffer, the frame size is much larger than expected.

vbv-size | bits in first frame | max frame size (bits) | difference
0	3209168		
1	2142968	333667	1809301
2	2173872	667334	1506538
3	2201936	1001001	1200935
4	2201936	1334668	867268
5	2201936	1668335	533601
6	2201936	2002002	199934
7	2201936	2335669	-133733
8	2201936	2669336	-467400
9	2201936	3003003	-801067
10	2201936	3336670	-1134734
11	2270168	3670337	-1400169
12	2431416	4004004	-1572588
13	2487960	4337671	-1849711
14	2595584	4671338	-2075754
15	2697376	5005005	-2307629
16	2783632	5338672	-2555040
17	2866800	5672339	-2805539

Is it possible to have the first frame respect the virtual buffer size? It happens to be an I-frame, and so it is large; subsequent tests with a GOP of 30 show that even subsequent I-frames’ sizes blow over the virtual buffer size.

Thanks!

Jonathan

4

Hello, Jonathan:
Would you please check why the first frame has a zero vbv-size?
With zero, driver will set it to default value.

br
ChenJian

5

Hi ChenJian,

In the table I pasted in my last post, I set the vbv-size to some value and am reporting the size of the first frame (the I-frame). Therefore, when the vbv-size is set to zero, the frame size for this case is 3209168. Since, like you said, zero sets the vbv-size to a default value, I did not include a “max frame size” for that case. This table was to demonstrate that the size of the virtual buffer is exceeded by I-frames.

Does the TX1 support setting a max frame size for a video stream?

Thanks,

Jonathan

6

Does vbv-size work properly without a constant bitrate mode (control-rate=constant)?

7

Any updates on this?
I too am finding that I frames exceed the VBV.

There are times when I need to set a low bitrate, but the I frames are totally eating up all the bandwidth.

One simple example is to encode random noise. The I picture always seems to come out visually perfect. I don’t want that, I want a degraded picture if needed to meet bandwidth requirements.

8

Hi phabsch,

I have had no resolution on this.

The new Jetpack has been released (JetPack SDK | NVIDIA Developer), which uses V4L2 to interact with the driver, but I haven’t done qualification on it yet to verify that it will overcome this problem.

9

Hello, all:
We have noticed this thread and are investigating this issue.
We will update the thread once there’s update.

thanks for patience.

br
ChenJian

10

Hi all
I wanted to know if there is a resolution on this issue which was raised a few months ago.
We also tackle this problem.
In our application, the frame rate is 30[FPS] and the maximal allowed bit-rate per frame is 2e6/30 [bits]. The settings in terms of GOP, I frame, P frame and B frame are the same as above.
If there is a resolution, can you please provide the correct pipe (in terms of the above example).
Curently, we see peaks of even 6[Mbits].
Thanks a lot
Amir

11

Hi Amir,
We will have the fix in CBR mode in next TX1 release.

12

Hi
Thanks fpr your reply.
Can You please specify an exact date for the next TX1 release?
Can we get a beta version (or something similar) for this bug fix becasue we are stuck on this issue?
Thanks a lot
Amir

13

Hi Amir,
Please try to adjust qp-range of I frames:
[url]https://devtalk.nvidia.com/default/topic/973295/jetson-tx1/constant-bitrate-help/post/5009319/#5009319[/url]

It should have improvement in size of I frames.