Constant Bitrate help

Autonomous Machines Jetson & Embedded Systems Jetson TX1
#5

Hey DaneLLL! Thanks for reaching out. Unfortunately I had previously tried that and many combinations of qp-range.

Again this symptom only happens with lots of motion on screen. I can provide you the output file if needed. Can you retry your test there w/ a difficult to encode to test pattern?

I ran the following:

gst-launch-1.0 v4l2src device=/dev/video0 do-timestamp=true io-mode=rw num-buffers=500 ! \
"video/x-raw, width=1920, height=1080, format=(string)UYVY, framerate=(fraction)30/1" ! \
nvvidconv output-buffers=125 ! \
'video/x-raw(memory:NVMM), width=1920, height=1080,format=I420, framerate=30/1'  !  \
omxh264enc bitrate=500000 qp-range=35,51:35,51:-1,-1 vbv-size=1 iframeinterval=60 ! \
qtmux ! \
filesink location=a.mp4

And here are the results in MediaInfo

Format : MPEG-4
Format profile : QuickTime
Codec ID : qt 2005.03 (qt )
File size : 7.49 MiB
Duration : 16 s 619 ms
Overall bit rate : 3 783 kb/s
Encoded date : UTC 2016-11-01 14:11:06
Tagged date : UTC 2016-11-01 14:11:06

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Baseline@L5
Format settings, CABAC : No
Format settings, ReFrames : 1 frame
Format settings, GOP : M=1, N=60
Muxing mode : Container profile=Baseline@2.1
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 16 s 619 ms
Bit rate : 3 778 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Variable
Frame rate : 30.086 FPS
Minimum frame rate : 14.851 FPS
Maximum frame rate : 50.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.061
Stream size : 7.49 MiB (100%)
Language : English
Encoded date : UTC 2016-11-01 14:11:06
Tagged date : UTC 2016-11-01 14:11:06

#6

I think i’ve gotten the encoder (using my high motion video input) to a point where it simply can’t compress the image data anymore. That is what I am guessing at least.

So i think the only solution would be for the encoder to drop frames to meet this low bitrate.

So i believe having “OMX_Video_ControlRateConstantSkipFrames” work would be the only solution, but previously it was stated the the NVENC encoder doesn’t support this omx call? See here:
https://devtalk.nvidia.com/default/topic/948594/questions-on-gstreamer-on-tx1/
“Constant and constant-skip-frame: actually it is same, for bSkipframe is not handled in codec driver.”

Is there anyway to get this working in the encoder/driver?

Again I am not confident that is the only solution but I have seemingly exhausted all other options.

#7

Hi x1tester62,
Please share a.mp4 you recorded in #5 for our reference. And are you on r24.2?

#8

I am using 23.2 for soc-camera driver reasons at this time, but i am using a compile of gstreamer 1.6.0 i built myself as well as the omx-sources from 24.2 i built and installed.

Here is the video on a filedropper link. let me know if you would prefer a different way to provide the file.

http://www.filedropper.com/a_35

#9

Hi x1tester62,
So you download gstomx1_src.tbz2 in
https://developer.nvidia.com/embedded/dlc/l4t-24-2-sources
, build and replace libgstomx.so?

If yes, could you apply 0002-fix-parsing-qp-range.txt and rebuild libgstomx.so?

In the a.mp4 attached, the I frames are quantized at 30, but it should be in 35-51. Please apply the patch and give it a try.
0002-fix-parsing-qp-range.txt (778 Bytes)

#10

Hi x1tester62,
Another way is to change the 1080p setting in /etc/enctune.conf
H264:
Settings_1080P
IMinQP 35
IMaxQP 51
PMinQP 35
PMaxQP 51

And run the command without qp-range:
gst-launch-1.0 nvcamerasrc num-buffers=300 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=I420’ ! omxh264enc bitrate=500000 iframeinterval=60 ! qtmux ! filesink location=a.mp4

#11

Hey DaneLLL. Thanks again so much for helping out. I really appreciate it!!!

Looks like editing the enctune.conf file does have an affect. I didn’t know this file existed. Is this documented anywhere, the parameter options and such?

After applying the changes to enctune.conf, it improved the bitrate reduction, but did only bring it down to 1.5mbps (down from the 3mbps from before).
Obviously this is because previously the max q was at 30 and a quantization of 30 with my test pattern resulted in 3mbps. Now the max is set to 51 and it is always being set to 51 which results in 1.5mbps. I confirmed this by setting both the min and max to 51.

I’ve tried changing the other various settings in that file (VBR, frameskip, etc) and none are bringing it any lower.

So now i truly am at the point i was claiming before where I can’t get any lower without dropping frames. Any ideas?

Also how did you check the quantinization factor of files?

#12

Hi x1tester62,
I think you are right. 1.5mbps can be the lowest bitrate for high motion case of 1080p30. In general, the bitrate is suggested to be ~10mbps for 1080p30 to get good quality. If all frames are quantized at 50 and it still cannot go below 1.5mbps, it hits the compression limit. For getting lower bitrate, we have to drop frames.

Please try
gst-launch-1.0 nvcamerasrc num-buffers=300 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=I420’ ! omxh264enc bitrate=500000 iframeinterval=60 temporal-tradeoff=2 ! qtmux ! filesink location=a.mp4

And below is the definition of all types
typedef enum NVX_ENCODE_VIDEOTEMPORALTRADEOFF{
NVX_ENCODE_VideoEncTemporalTradeoffLevel_DropNone= 0,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop1in5,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop1in3,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop1in2,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop2in3,
NVX_ENCODE_VideoEncTemporalTradeoffLevel_Force32 = 0x7FFFFFFF
} NVX_ENCODE_VIDEOTEMPORALTRADEOFF;

To check the quantization factor, please do
1 Extract h264 via yamb http://yamb.unite-video.com/
2 Decode h264 via JM decoder http://iphome.hhi.de/suehring/tml/
(Download the source and compile)

./JM/bin/ldecode.exe -i <PATH_to_h264>

Gstreamer: temporal-tradeoff=2
Would the Jetson Nano VPU be fit for our project
Invalid bitrate control for preset V4L2_ENC_HW_PRESET_MEDIUM
About "control-rate" of omxh264enc and nvv4l2h264enc
Problem adjusting bitrate during H265 encoding
H264 intra frame size
#13

Codecvisa is also a useful tool to plot the bitrate: http://www.codecian.com/downloads.html

#14

Thanks Dane,

I understand the suggested bitrates. I just have specific scenarios where bandwidth may be limited to 1Mbps. Usually this is results in quality of around quant 20-40, but occasionally there will be a large spike in motion resulting in it hitting 51 and in turn raising the bitrate over my bandwidth limit.
This results in my network dropping packets which in turn corrupts the video data since it isn’t selective with what packets it drops. Preferably frames would be dropped prior to this.

I see the temporal tradeoff option you have provided forces the system to always drops frames, but is there no way for the encoder to dynamically drop frames during these high moments of motion.

Pipeline being during high motion:
-raise quantinization until hits max.
-if bitrate is still not met
-drop frames until bitrate is met.

If that is not possible can anyone think of any work arounds? only thing i could think is for me to manually determine the bitrate and then using videorate throttle down the frame rate manually but this does seem complicated and cumbersome.

#15

Hi x1tester62,
As we have provided the source of libgstomx.so (gstomx1_src.tbz2), you can implement the frame-dropping mechanism in gstomxh264enc.c or gstomxvideoenc.c like
1 NVX_ENCODE_VideoEncTemporalTradeoffLevel_DropNone by default
2 dynamically monitor the bitrate
3 when bitrate exceeds maximux, change to NVX_ENCODE_VideoEncTemporalTradeoffLevel_Drop1in2 (or others)
4 when bitrate lays back, return to default NVX_ENCODE_VideoEncTemporalTradeoffLevel_DropNone

#16

Understood. I can implement that.

Only issue i currently am running into is dynamically monitoring the bitrate. Does anyone have an easy and efficient way to do this?

There doesn’t seem to be a standard way.

#17

Hi x1tester62,
We suggest that you use
buf->omx_buf->nFilledLen
buf->omx_buf->nTimeStamp
in gst_omx_video_enc_handle_output_frame() to calculate the instant bitrate.

As you said, there is no standard way. The psuedo code below is for your reference:
total_time = buf->omx_buf->nTimeStamp - timestamp_base;
total_length += buf->omx_buf->nFilledLen;
instant_bitrate = total_length / total_time;

if (instant_bitrate > self->bitrate) {
// drop frames
} else {
// fall back to default
}

if (total_time > 2 seconds)
{
// reset every 2 seconds
total_length = 0;
timestamp_base = buf->omx_buf->nTimeStamp;
}

#18

DaneLLL,

I was able to implement the bitrate monitor successfully and can now detect when a bitrate is too large. Thanks for the psuedo code. That helped alot.

A couple issues though:

  1. The Temporal Tradeoff settings in the omxh26*enc plugin requires the gstreamer state to be NULL or READY:
temporal-tradeoff   : Temporal Tradeoff value for encoder
                        flags: readable, writable, changeable only in NULL or READY state

Unfortunately this results in an unwanted delay and stagger since i must change the state of the pipeline to READY to make the change. In addition i have multiple pipelines going on (for example 1 high bitrate and 1 low bitrate parallel encode) and I have to change the state of the full pipeline to change this value. So other streams also get interrupted.

  1. To try and bypass this issue, I decided to put a ‘videorate’ plugin in between the v4l2src and nvvidconv plugins. I then can change the framerate fed into the encoder dynamically be changing the caps of videorate and nvvidconv. This allows for a dynamic drop of frames going into the encoder and the encoder responds accordingly with the print message:
NvxVideoEncoderSetParameterFramerate set to : fpsvalue

Unfortunately the encoder behaves oddly. You would assume the following behavior:
bitrate is set to 500kbps
Video is 60fps and output bitrate is 1mbps and won’t go lower.
Lower framerate to 30fps and output bitrate would cut in half resulting in 500kbps.

But that is not what happens. I dont know if the encoder is then lowering quantization factor or what, but the bitrate falls only a fraction of what it should. For example i may have to lower the framerate to 5-10fps in the above example Any ideas?

Is there a way to query the quantization factor dynamically?

  1. Sometimes when changing the framerate via videorate i get the following fault from the encoder:
VENC: TryProcessingData: 1619:  NvMMQueueDeQ pInBuf failed
VENC: NvMMLiteVideoEncDoWork: 2512: BlockSide error 0xa
Event_BlockError from 0BlockAvcEnc : Error code - a
Sending error event from 0BlockAvcEnc

Is there anyway to alleviate this and/or detect and properly reset? Is there a specific reset that can be sent to the encoder?

Another fault I am having is i must change both the nvvidconv caps as well as the videorate caps, and i believe sometimes a race condition occurs and causes no video to be produced. Is there a better way to change caps in sync? Maybe force a renegotiation?

#19

Hi x1tester62,

temporal-tradeoff   : Temporal Tradeoff value for encoder
                        flags: readable, writable, changeable only in NULL or READY state

This is true when the user runs the gstreamer command with the original source code. But you have added new functionality in the source, it should work by calling gstomx_set_temporal_tradeoff().

Have yo tried to call it in runtime?

#20

DaneLLL,

Understood. I have implemented that functionality. Still using the videorate plugin in combination w/ this temporal tradeoff.

Still getting the occasional crash (pInBuf failed error listed before) as well as this sometimes crashing the encoder:

NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 6, SyncPointValue = 0)

Do you have a quick reset capability or the ability to detect and recover from such an error?
I believe this happens randomly but prevalence is escalated when I change the framerate too many times in quick succession.

#21

Hi x1tester62,
Is it possible to use temporal tradeoff only? It may not be a valid usecase to change input framerate in runtime.

Please share the gst command with videorate for reference.

#22

Im doing it programmatically. I just change the caps on the videorate plugin to change the framerate but also have to change the caps on the nvvidconv plugin so it negotiates properly.

Id prefer to have more control than what temporal-tradeoff provides since it only provides frame dropping in set intervals and hence can’t be as exact as defining the framerate immediately.

#23

I am using TX1 with L4T R23.2.
I am using Qt 5.2.1 with Qt-gstreamer 1.2 for my gstreamer based application.

Below is the pipeline that i have created in Qt.

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, width=1920, height=1080, format=UYVY, framerate=60/1' \
! videorate ! 'video/x-raw, framerate=30/1' \
! nvvidconv ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=I420, framerate=30/1' \
! tee name=t ! queue ! nvoverlaysink sync=false async=false t. ! queue \
! omxh264enc bitrate=19000000 low-latency=1 control-rate=2 ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! qtmux ! filesink location=test.mp4 sync=false async=false -e

My camera sensor output 1080p60 video in UYVY format and in my Qt application, i want to dynamically change the filesink location.

So below is the way i change the location

m_queue2->sendEvent(QGst::EosEvent::create()); // here queue2 is the queue before omxh264enc
m_pipeline->setState(QGst::StateReady);
m_filesink->setState(QGst::StateNull);

if(changeFile == 0)
{
     m_filesink->setProperty("location", "/home/ubuntu/1.mp4");
     changeFile = 1;
}
else
{
     m_filesink->setProperty("location", "/home/ubuntu/2.mp4");
     changeFile = 0;
}
m_pipeline->setState(QGst::StatePlaying);

For Testing i have create a timer that called every 5 sec and change the location using above code.

This code works. But after few changeover, gstreamer pipeline hangs at while changing the location. Sometimes it crashes the whole Qt app, sometimes just hangs the pipeline until the next switchover.

And when this hangs i am getting below error. as mentioned in post 18 by x1tester62.

VENC: TryProcessingData: 1619:  NvMMQueueDeQ pInBuf failed
VENC: NvMMLiteVideoEncDoWork: 2512: BlockSide error 0xa
Event_BlockError from 7BlockAvcEnc : Error code - a
Sending error event from 7BlockAvcEnc

And this is due to videorate plugin. If i remove the videorate from the pipeline and connect v4l2src directly to nvvidconv this problem not occurs.

So whats the wrong here?
i think i need to take some extra care at the time before or after setting pipeline state to READY. So that there should be no sync issue.
or any other way so that i can change location?

#24