Constant Bitrate help

I need to the ability to encode and stream h.264 and h.265 at very controlled constant bitrates regardless of quality output. I understand encoding at a low bitrate can result in poor quality.

The main issue I am having though is I can’t get some of the video encodes (ones with high motion) to stay at a constant bitrate when set to a low bitrate.
I have tried many different gstreamer settings. I even compiled the newer omx source on my system to take advantage of EnableTwopassCBR which also didn’t seem to make a difference.

Running the following tests I have the following issues:

Test 1

gst-launch-1.0 v4l2src device=/dev/video0 do-timestamp=true io-mode=rw \
! "video/x-raw, width=1280, height=720, format=(string)UYVY, framerate=(fraction)60/1" \
! nvvidconv output-buffers=125 \
! 'video/x-raw(memory:NVMM), width=1280, height=720,format=I420, framerate=60/1'  \
! omxh265enc bitrate=500000 control-rate=constant vbv-size=1  EnableTwopassCBR=true iframeinterval=60\
! 'video/x-h265, stream-format=(string)byte-stream' \
! h265parse \
! queue \
! mpegtsmux alignment=7 \
! filesink location=/opt/out.ts

Issue 1:
This is a 720p60 stream with HIGH motion video which obviously makes it difficult to encode. As you can see I am requesting an encode at a bitrate of 500kbps. The resulting video though is 1.8-1.9mbps. Is there anyway to force the feed to stay at 500kbps? Maybe with dynamic frame dropping?
I’ve tried this at multiple resolutions and all show the same symptom as long as the video feed is high motion.

Test 2

gst-launch-1.0 v4l2src device=/dev/video0 do-timestamp=true io-mode=rw \
! "video/x-raw, width=1920, height=1080, format=(string)UYVY, framerate=(fraction)30/1" \
! nvvidconv output-buffers=125 \
! 'video/x-raw(memory:NVMM), width=1920, height=1080,format=I420, framerate=30/1'  \
! omxh264enc bitrate=500000 control-rate=constant vbv-size=1  EnableTwopassCBR=true iframeinterval=60\
! 'video/x-h264, stream-format=(string)byte-stream' \
! h264parse \
! queue \
! mpegtsmux alignment=7 \
! filesink location=/opt/out.ts

Issue 2:
This is a 1080p30 video with subtle motion. The encoder does stay at the requested 500kbps, but once every 2-3 seconds spikes up to 1.5 before going back down to 500kbps. Oddly H.265 does not exhibit this issue, only H.264.

Previously I asked for help with constant bitrate controls on this thread:
https://devtalk.nvidia.com/default/topic/948594/?comment=4928467

The answer was to set vbv-size to 1 which works for low motion video, but not for this high motion.

Would changing the quantization factor help? What is this temporal tradeoff and will that help?

Here is the mediainfo from an example of a 1080p30 stream set to be 500kbps.
As you can see the resulting bitrate is 745kbps because of spiking. Is there anyway to limit this?
This is a regular use case example without crazy amounts of motion. If i add extreme motion it only gets exponentially worse…
Can the system drop frames dynamically to meet this restriction if necessary?

File size : 7.36 MiB
Duration : 1 min 22 s
Overall bit rate mode : Variable
Overall bit rate : 745 kb/s

Video
ID : 65 (0x41)
Menu ID : 1 (0x1)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L5
Format settings, CABAC : Yes
Format settings, ReFrames : 1 frame
Format settings, GOP : M=1, N=60
Codec ID : 27
Duration : 1 min 21 s
Bit rate : 709 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Stream size : 6.87 MiB (93%)

Here are the high motion tests where my main issue arises and where you can truly see this mismatch of what i select as the targeted bitrate and what it actually results in. As you can see getting 2passCBR enabled helped somewhat, but it is still no where as low as the selected bitrate.

1080p30 500kbps targeted bitrate w/ Enable2Pass on.

gst-launch-1.0 v4l2src device=/dev/video0 do-timestamp=true io-mode=rw !\
"video/x-raw, width=1920, height=1080, format=(string)UYVY, framerate=(fraction)30/1" ! \
nvvidconv output-buffers=125 ! \
'video/x-raw(memory:NVMM), width=1920, height=1080,format=I420, framerate=30/1'  !  \
omxh265enc bitrate=500000 control-rate=2 vbv-size=1 iframeinterval=60  EnableTwopassCBR=1  ! \
'video/x-h265, stream-format=(string)byte-stream' !  \
h265parse ! \
queue ! \
mpegtsmux alignment=7 ! \ 
udpsink host=224.0.0.2 port=5003  sync=false

Media Info:
Format : MPEG-TS
File size : 16.6 MiB
Duration : 1 min 4 s
Overall bit rate mode : Variable
Overall bit rate : 2 170 kb/s

Video
ID : 65 (0x41)
Menu ID : 1 (0x1)
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L1@High
Codec ID : 36
Duration : 1 min 4 s
Bit rate : 2 063 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Stream size : 15.8 MiB (95%)

1080p30 500kbps targeted bitrate w/ Enable2Pass off.

gst-launch-1.0 v4l2src device=/dev/video0 do-timestamp=true io-mode=rw ! \
"video/x-raw, width=1920, height=1080, format=(string)UYVY, framerate=(fraction)30/1" ! \
nvvidconv output-buffers=125 ! \
'video/x-raw(memory:NVMM), width=1920, height=1080,format=I420, framerate=30/1'  !  \
omxh265enc bitrate=500000 control-rate=2 vbv-size=1 iframeinterval=60  EnableTwopassCBR=0  ! \
'video/x-h265, stream-format=(string)byte-stream' !  \
h265parse ! \
queue !   \
mpegtsmux alignment=7 ! \
udpsink host=224.0.0.2 port=5003  sync=false

Format : MPEG-TS
File size : 23.4 MiB
Duration : 1 min 2 s
Overall bit rate mode : Variable
Overall bit rate : 3 143 kb/s

Video
ID : 65 (0x41)
Menu ID : 1 (0x1)
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L1@High
Codec ID : 36
Duration : 1 min 2 s
Bit rate : 2 988 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Stream size : 22.3 MiB (95%)

Hi x1tester62,

Could you try to tune the qp-range?

gst-launch-1.0 nvcamerasrc num-buffers=300 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=I420’ ! omxh264enc bitrate=500000 qp-range=35,51:35,51:-1,-1 vbv-size=1 iframeinterval=60 ! qtmux ! filesink location=a.mp4

I tried the command attached and can get 500 bps video stream. Please give it a try.

Hey DaneLLL! Thanks for reaching out. Unfortunately I had previously tried that and many combinations of qp-range.

Again this symptom only happens with lots of motion on screen. I can provide you the output file if needed. Can you retry your test there w/ a difficult to encode to test pattern?

I ran the following:

gst-launch-1.0 v4l2src device=/dev/video0 do-timestamp=true io-mode=rw num-buffers=500 ! \
"video/x-raw, width=1920, height=1080, format=(string)UYVY, framerate=(fraction)30/1" ! \
nvvidconv output-buffers=125 ! \
'video/x-raw(memory:NVMM), width=1920, height=1080,format=I420, framerate=30/1'  !  \
omxh264enc bitrate=500000 qp-range=35,51:35,51:-1,-1 vbv-size=1 iframeinterval=60 ! \
qtmux ! \
filesink location=a.mp4

And here are the results in MediaInfo

Format : MPEG-4
Format profile : QuickTime
Codec ID : qt 2005.03 (qt )
File size : 7.49 MiB
Duration : 16 s 619 ms
Overall bit rate : 3 783 kb/s
Encoded date : UTC 2016-11-01 14:11:06
Tagged date : UTC 2016-11-01 14:11:06

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Baseline@L5
Format settings, CABAC : No
Format settings, ReFrames : 1 frame
Format settings, GOP : M=1, N=60
Muxing mode : Container profile=Baseline@2.1
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 16 s 619 ms
Bit rate : 3 778 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Variable
Frame rate : 30.086 FPS
Minimum frame rate : 14.851 FPS
Maximum frame rate : 50.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.061
Stream size : 7.49 MiB (100%)
Language : English
Encoded date : UTC 2016-11-01 14:11:06
Tagged date : UTC 2016-11-01 14:11:06

I think i’ve gotten the encoder (using my high motion video input) to a point where it simply can’t compress the image data anymore. That is what I am guessing at least.

So i think the only solution would be for the encoder to drop frames to meet this low bitrate.

So i believe having “OMX_Video_ControlRateConstantSkipFrames” work would be the only solution, but previously it was stated the the NVENC encoder doesn’t support this omx call? See here:
https://devtalk.nvidia.com/default/topic/948594/questions-on-gstreamer-on-tx1/
“Constant and constant-skip-frame: actually it is same, for bSkipframe is not handled in codec driver.”

Is there anyway to get this working in the encoder/driver?

Again I am not confident that is the only solution but I have seemingly exhausted all other options.

Hi x1tester62,
Please share a.mp4 you recorded in #5 for our reference. And are you on r24.2?

I am using 23.2 for soc-camera driver reasons at this time, but i am using a compile of gstreamer 1.6.0 i built myself as well as the omx-sources from 24.2 i built and installed.

Here is the video on a filedropper link. let me know if you would prefer a different way to provide the file.

http://www.filedropper.com/a_35

Hi x1tester62,
So you download gstomx1_src.tbz2 in
https://developer.nvidia.com/embedded/dlc/l4t-24-2-sources
, build and replace libgstomx.so?

If yes, could you apply 0002-fix-parsing-qp-range.txt and rebuild libgstomx.so?

In the a.mp4 attached, the I frames are quantized at 30, but it should be in 35-51. Please apply the patch and give it a try.
0002-fix-parsing-qp-range.txt (778 Bytes)

Hi x1tester62,
Another way is to change the 1080p setting in /etc/enctune.conf
H264:
Settings_1080P
IMinQP 35
IMaxQP 51
PMinQP 35
PMaxQP 51

And run the command without qp-range:
gst-launch-1.0 nvcamerasrc num-buffers=300 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=I420’ ! omxh264enc bitrate=500000 iframeinterval=60 ! qtmux ! filesink location=a.mp4

Hey DaneLLL. Thanks again so much for helping out. I really appreciate it!!!

Looks like editing the enctune.conf file does have an affect. I didn’t know this file existed. Is this documented anywhere, the parameter options and such?

After applying the changes to enctune.conf, it improved the bitrate reduction, but did only bring it down to 1.5mbps (down from the 3mbps from before).
Obviously this is because previously the max q was at 30 and a quantization of 30 with my test pattern resulted in 3mbps. Now the max is set to 51 and it is always being set to 51 which results in 1.5mbps. I confirmed this by setting both the min and max to 51.

I’ve tried changing the other various settings in that file (VBR, frameskip, etc) and none are bringing it any lower.

So now i truly am at the point i was claiming before where I can’t get any lower without dropping frames. Any ideas?

Also how did you check the quantinization factor of files?

Hi x1tester62,
I think you are right. 1.5mbps can be the lowest bitrate for high motion case of 1080p30. In general, the bitrate is suggested to be ~10mbps for 1080p30 to get good quality. If all frames are quantized at 50 and it still cannot go below 1.5mbps, it hits the compression limit. For getting lower bitrate, we have to drop frames.

Please try
gst-launch-1.0 nvcamerasrc num-buffers=300 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=I420’ ! omxh264enc bitrate=500000 iframeinterval=60 temporal-tradeoff=2 ! qtmux ! filesink location=a.mp4

And below is the definition of all types
typedef enum NVX_ENCODE_VIDEOTEMPORALTRADEOFF{
NVX_ENCODE_VideoEncTemporalTradeoffLevel_DropNone= 0,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop1in5,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop1in3,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop1in2,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop2in3,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Force32 = 0x7FFFFFFF
} NVX_ENCODE_VIDEOTEMPORALTRADEOFF;

To check the quantization factor, please do
1 Extract h264 via yamb http://yamb.unite-video.com/
2 Decode h264 via JM decoder http://iphome.hhi.de/suehring/tml/
(Download the source and compile)

./JM/bin/ldecode.exe -i <PATH_to_h264>

Codecvisa is also a useful tool to plot the bitrate: http://www.codecian.com/downloads.html

Thanks Dane,

I understand the suggested bitrates. I just have specific scenarios where bandwidth may be limited to 1Mbps. Usually this is results in quality of around quant 20-40, but occasionally there will be a large spike in motion resulting in it hitting 51 and in turn raising the bitrate over my bandwidth limit.
This results in my network dropping packets which in turn corrupts the video data since it isn’t selective with what packets it drops. Preferably frames would be dropped prior to this.

I see the temporal tradeoff option you have provided forces the system to always drops frames, but is there no way for the encoder to dynamically drop frames during these high moments of motion.

Pipeline being during high motion:
-raise quantinization until hits max.
-if bitrate is still not met
-drop frames until bitrate is met.

If that is not possible can anyone think of any work arounds? only thing i could think is for me to manually determine the bitrate and then using videorate throttle down the frame rate manually but this does seem complicated and cumbersome.

Hi x1tester62,
As we have provided the source of libgstomx.so (gstomx1_src.tbz2), you can implement the frame-dropping mechanism in gstomxh264enc.c or gstomxvideoenc.c like
1 NVX_ENCODE_VideoEncTemporalTradeoffLevel_DropNone by default
2 dynamically monitor the bitrate
3 when bitrate exceeds maximux, change to NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop1in2 (or others)
4 when bitrate lays back, return to default NVX_ENCODE_VideoEncTemporalTradeoffLevel_DropNone

Understood. I can implement that.

Only issue i currently am running into is dynamically monitoring the bitrate. Does anyone have an easy and efficient way to do this?

There doesn’t seem to be a standard way.

Hi x1tester62,
We suggest that you use
buf->omx_buf->nFilledLen
buf->omx_buf->nTimeStamp
in gst_omx_video_enc_handle_output_frame() to calculate the instant bitrate.

As you said, there is no standard way. The psuedo code below is for your reference:
total_time = buf->omx_buf->nTimeStamp - timestamp_base;
total_length += buf->omx_buf->nFilledLen;
instant_bitrate = total_length / total_time;

if (instant_bitrate > self->bitrate) {
// drop frames
} else {
// fall back to default
}

if (total_time > 2 seconds)
{
// reset every 2 seconds
total_length = 0;
timestamp_base = buf->omx_buf->nTimeStamp;
}

DaneLLL,

I was able to implement the bitrate monitor successfully and can now detect when a bitrate is too large. Thanks for the psuedo code. That helped alot.

A couple issues though:

  1. The Temporal Tradeoff settings in the omxh26*enc plugin requires the gstreamer state to be NULL or READY:
temporal-tradeoff   : Temporal Tradeoff value for encoder
                        flags: readable, writable, changeable only in NULL or READY state

Unfortunately this results in an unwanted delay and stagger since i must change the state of the pipeline to READY to make the change. In addition i have multiple pipelines going on (for example 1 high bitrate and 1 low bitrate parallel encode) and I have to change the state of the full pipeline to change this value. So other streams also get interrupted.

  1. To try and bypass this issue, I decided to put a ‘videorate’ plugin in between the v4l2src and nvvidconv plugins. I then can change the framerate fed into the encoder dynamically be changing the caps of videorate and nvvidconv. This allows for a dynamic drop of frames going into the encoder and the encoder responds accordingly with the print message:
NvxVideoEncoderSetParameterFramerate set to : fpsvalue

Unfortunately the encoder behaves oddly. You would assume the following behavior:
bitrate is set to 500kbps
Video is 60fps and output bitrate is 1mbps and won’t go lower.
Lower framerate to 30fps and output bitrate would cut in half resulting in 500kbps.

But that is not what happens. I dont know if the encoder is then lowering quantization factor or what, but the bitrate falls only a fraction of what it should. For example i may have to lower the framerate to 5-10fps in the above example Any ideas?

Is there a way to query the quantization factor dynamically?

  1. Sometimes when changing the framerate via videorate i get the following fault from the encoder:
VENC: TryProcessingData: 1619:  NvMMQueueDeQ pInBuf failed
VENC: NvMMLiteVideoEncDoWork: 2512: BlockSide error 0xa
Event_BlockError from 0BlockAvcEnc : Error code - a
Sending error event from 0BlockAvcEnc

Is there anyway to alleviate this and/or detect and properly reset? Is there a specific reset that can be sent to the encoder?

Another fault I am having is i must change both the nvvidconv caps as well as the videorate caps, and i believe sometimes a race condition occurs and causes no video to be produced. Is there a better way to change caps in sync? Maybe force a renegotiation?

Hi x1tester62,

temporal-tradeoff   : Temporal Tradeoff value for encoder
                        flags: readable, writable, changeable only in NULL or READY state

This is true when the user runs the gstreamer command with the original source code. But you have added new functionality in the source, it should work by calling gstomx_set_temporal_tradeoff().

Have yo tried to call it in runtime?

DaneLLL,

Understood. I have implemented that functionality. Still using the videorate plugin in combination w/ this temporal tradeoff.

Still getting the occasional crash (pInBuf failed error listed before) as well as this sometimes crashing the encoder:

NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 6, SyncPointValue = 0)

Do you have a quick reset capability or the ability to detect and recover from such an error?
I believe this happens randomly but prevalence is escalated when I change the framerate too many times in quick succession.