H.264/5 iframe pulsing

mark_umina · April 19, 2021, 10:49pm

Dear Support,

I’ve done everything I can imagine, and can’t stop the iframes from pulsing on any camera that I try - from MIPI to USB.

Here is my pipeline:

TX:
gst-launch-1.0 pylonsrc ! video/x-raw,width=2560,height=1440,format=UYVY ! nvvidconv ! nvv4l2h264enc bitrate=50000000 control-rate=0 vbv-size=450000 iframeinterval=30 ! 'video/x-h264, stream-format=(string)byte-stream' ! rtph264pay  ! udpsink host=192.168.1.157 port=5000

RX:
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! avdec_h264 skip-frame=5 ! autovideosink sync=false async=false -e

The issue restated is that the iframes cause a visible pulse that I cannot remove, regardless of bit rate, performance setting… etc…, I’ve been through many hours of testing.

Here is a sample video with the blip that I speak of, almost once per second:

video sample

Increasing the iframe interval is not a solution because then we see artifacts on movement.

I have looked into skipping the iframes on receive (which I would rather not do) but there is no such option:

Happy to discuss next steps with you.

BTW, the omxh265enc/omxh264enc encoders have been tested, but we are not able to get the same encoding speed out of them as the nvvidconv - frame rate too low.

Thanks in advance, and happy to discuss further.

DaneLLL · April 20, 2021, 3:25am

Hi,
We suggest set vbv-size in bitrate/fps ~ 2*(bitrate/fps). Your bitrate is 50Mbps and for 30fps, may set in the range 1666666 ~ 3333333.

mark_umina · April 20, 2021, 4:15am

Hi @DaneLLL Thank you for the response. I’ll give this a try tomorrow in the daylight.

mark_umina · April 20, 2021, 3:16pm

Hi @DaneLLL

I have tested with the suggested buffer sizes on the pipeline in my original post. it caused the digitization to get far worse.

Please let me know what other suggestions you may have.

So far we are not seeing great results using the nvv4l2h264enc/nvv4l2h264enc encoders, and the omxh264enc/omxh265enc are not able to attain 30fps at 2560 x 1440.

DaneLLL · April 21, 2021, 4:21am

Hi,
control-rate=0 is VBR mode. Please set control-rate=1 along with vbv-size and try again.

The commands in the post are corrected.

mark_umina · April 21, 2021, 8:34pm

Hi @DaneLLL

I’ve tried the suggestion above.

Both of these pipelines increase digitization to create an almost unusable picture:

gst-launch-1.0 pylonsrc ! video/x-raw,width=2560,height=1440,format=UYVY ! nvvidconv ! nvv4l2h264enc bitrate=50000000 control-rate=1 vbv-size=3333333 iframeinterval=30 ! 'video/x-h264, stream-format=(string)byte-stream' ! rtph264pay  ! udpsink host=192.168.1.157 port=5000

gst-launch-1.0 pylonsrc ! video/x-raw,width=2560,height=1440,format=UYVY ! nvvidconv ! nvv4l2h264enc bitrate=50000000 control-rate=1 vbv-size=1666666 iframeinterval=30 ! 'video/x-h264, stream-format=(string)byte-stream' ! rtph264pay  ! udpsink host=192.168.1.157 port=5000

For comparison, the original stream at the start of this post is usable outside of the pulsing on iframes.

The network is not limiting, it is a direct wired connection to a switch, and larger tens of Mb streams are able to make it through fine.

Is there anything else we can try here?

DaneLLL · April 22, 2021, 4:00am

Hi,
Another possible solution is to run in VBR mode and set qp-range:

  qp-range            : Qunatization range for P, I and B frame,
                         Use string with values of Qunatization Range
                         in MinQpP-MaxQpP:MinQpI-MaxQpI:MinQpB-MaxQpB order, to set the property.

Yo may try

... ! nvv4l2h264enc control-rate=0 qp-range="10,15:1,5:10,15" ! ...

mark_umina · April 22, 2021, 5:31am

Hi @DaneLLL

That’s very interesting, and something I haven’t tried. I’ll give that a run in the morning when there’s light.

Thanks!

mark_umina · April 23, 2021, 5:08am

I tried this and still had a lot of trouble with it. I have again tested the nvv4l2h264/5enc using different cameras and I still see the same issues with pulsing on Iframes and digitization being very frequent regardless of bit rate.

I have moved away from them and onto the omxh265enc again, which I have found to generate a better image, without pulsing.

gst-launch-1.0 pylonsrc ! video/x-raw,width=2560,height=1440,format=I420 ! videoconvert ! omxh265enc bitrate=50000  iframeinterval=3 preset-level=1 ! 'video/x-h265, stream-format=(string)byte-stream' ! rtph265pay ! udpsink host=192.168.1.157 port=5000

My remaining questions are:

What steps can I take to ensure fastest frame processing rate using the omxh265enc?
So far the omxh265enc seems to ignore the bitrate setting, am I missing something in the pipeline above?

Thanks in advance.

Honey_Patouceul · April 23, 2021, 12:10pm

I am away from any jetson and cannot check, but you would find ultra fast preset option with:

gst-inspect-1.0 omxh265enc

The problem is that your bitrates are very very low… It may be a pulsing cause. For your resolution and framerate, you may try a bitrate of 30000000 first.

mark_umina · April 24, 2021, 5:31am

Hi @Honey_Patouceul Thanks for the response! I’m using 1 for FastPreset, because 0 appears very grainy:

                           (0): UltraFastPreset  - UltraFastPreset for high perf
                           (1): FastPreset       - FastPreset
                           (2): MediumPreset     - MediumPreset
                           (3): SlowPreset       - SlowPreset

Regarding the bit rate, I will experiment with this again, but I’m almost certain I brought the bit rate that high.

However, the higher bit rates really slow down the resulting frame rate - as the encoders can’t process 30fps even at QuadHD when running high bit rates.

I wonder - what bit rate is the Nano specified at when claiming 4Kp30 here:

What I mean is, 4kp30 is an odd claim to make if it can only be performed at a bit rate that produces unusable image… right? I wonder if I am missing something - but I have shown all my pipelines above, and tested on multiple cameras. And my test resolution is nowhere near 4k.

mark_umina · April 24, 2021, 8:55pm

@Honey_Patouceul I had a closer look at your response today. in the nvv4l2h264enc example at the start of my post I’m actually running a bit rate higher than what you are suggesting. I’m running: bitrate=50000000

To test whether bit rate is causing the issue with pulsing on nvv4l2h264enc, I pumped it way up:

gst-launch-1.0 pylonsrc ! video/x-raw,width=2560,height=1440,format=UYVY ! nvvidconv ! nvv4l2h264enc bitrate=4294967294 control-rate=1 vbv-size=300000 iframeinterval=10 ! 'video/x-h264, stream-format=(string)byte-stream' ! rtph264pay  ! udpsink host=192.168.1.157 port=5000

Still no luck, still pulsing. For sake of time, I have to conclude after extensive testing that the nvv4l2h264enc is not going to provide a pulse free stream.

The omxh265enc encoder seems to be doing well now, and I will continue working toward speeding up the frame rate.

I would still be interested in hearing from NVidia on how the nominal 4kp30 specs were determined for the Nano. Still hopeful, as it is the faster encoder.

DaneLLL · April 26, 2021, 2:38am

Hi,
The recommendation of encoder setting is CBR + vbv-size or VBR + qp-range. We have tried with this YUVs:
https://media.xiph.org/video/derf/y4m/park_joy_1080p50.y4m
And the result looks fine. For further improvement, you can set

  preset-level        : HW preset level for encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstV4L2VideoEncHwPreset" Default: 1, "UltraFastPreset"
                           (0): DisablePreset    - Disable HW-Preset
                           (1): UltraFastPreset  - UltraFastPreset for high perf
                           (2): FastPreset       - FastPreset
                           (3): MediumPreset     - MediumPreset
                           (4): SlowPreset       - SlowPreset

However, this impacts encoding rate. Since your input resolution is 2560x1440, it may not be able to achieve 30fps in MediumPreset or SlowPreset.

Honey_Patouceul · April 26, 2021, 10:51am

@mark_umina, sorry if my post wasn’t clear. The low bitrate pulsing was related to omx encoders, not nvv4l2 ones.

With nvv4l2h264enc, excessive bitrate may result in some pulsing. My advice of 30M would work with Main or High profile, but would lead to pulsing with baseline. Reducing the bitrate would allow baseline profile without pulsing.

# These work fine
gst-launch-1.0 videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM),width=2560,height=1440,framerate=30/1' ! nvv4l2h264enc bitrate=30000000 control-rate=1 profile=Main ! nvv4l2decoder ! nvvidconv ! xvimagesink sync=false
gst-launch-1.0 videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM),width=2560,height=1440,framerate=30/1' ! nvv4l2h264enc bitrate=30000000 control-rate=1 profile=High ! nvv4l2decoder ! nvvidconv ! xvimagesink sync=false

# This has pulsing
gst-launch-1.0 videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM),width=2560,height=1440,framerate=30/1' ! nvv4l2h264enc bitrate=30000000 control-rate=1 profile=Baseline ! nvv4l2decoder ! nvvidconv ! xvimagesink sync=false

# Reducing bitrate to 10M, baseline works fine
gst-launch-1.0 videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM),width=2560,height=1440,framerate=30/1' ! nvv4l2h264enc bitrate=10000000 control-rate=1 profile=Baseline ! nvv4l2decoder ! nvvidconv ! xvimagesink sync=false