Video is jittery through "capture/encode/decode/render" in the same gstreamer pipeline

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Video is not smooth and jittery while executing “gst-launch-1.0 nvv4l2camerasrc ! ‘video/x-raw(memory:NVMM),width=3840,height=2160’ ! nvvidconv ! nvv4l2vp9enc bitrate=25000000 ! nvv4l2decoder ! nv3dsink”.

CPU overhead is low and no error/warn messages are displayed

If I change bitrate from 25M bps to 5M bps, some warning messages are displayed:

WARNING: from element /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0:
There may be a timestamping problem, or this computer is too slow.

Then I tried nvv4l2h265enc, video is also not smooth and jittery. And it seems bitrate parameter is not working for h265 encoder? (because video qualities with different bitrate valuse is the same)

If just capture/render, the video is very smooth at 4Kp30,
gst-launch-1.0 nvv4l2camerasrc ! ‘video/x-raw(memory:NVMM),width=3840,height=2160’ ! nvvidconv ! nv3dsink

My devkit runs R32.4.3.

3

Hi,
Please enable the property in encoder an decoder:

  maxperf-enable      : Enable or Disable Max Performance mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false

  enable-max-performance: Set to enable max performance
                        flags: readable, writable
                        Boolean. Default: false
4

Hi DaneLLL,

I just tried maxperf-enable=true and enable-max-performance=true, these two params can’t solve the issue

5

Hi,
Please try nvoverlaysink, or

... ! nv3dsink sync=false
6

Hi DaneLLL,

“nv3dsink sync=false” works
“nvoverlaysink” does not work
“nvoverlaysink sync=false” works

I don’t understand why “sync=false" is needed here for live source. How to interpret/explain such setting?
(“capture/sink” and “udpsrc/decode/sink” pipelines are all OK without “sync=false” setting on sink element)

7

Hi,
With sync=1, gstreamer synchronization mechanism is enabled and for 30fps,if encoding+decoding exceeds 30ms, it reports too slow.
Please try h264 or h265. See if it is specific to vp9. May also try small resolution like 1920×1080. Or execute sudo nvpmodel -m 0 and sudo jetson_clocks.

8

Before executing gstreamer pipelines, I have already enabled maxn power mode and max freq by jetson_clocks.
h264 / h265 have the same issue as vp9.
1920x1080 also has the same issue as 4K resolution.

In addition, h264 case displayed some warn messages:
WARNING: from element /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0:
There may be a timestamping problem, or this computer is too slow.

9

Hi,
Please set ts-offset and give it a try:

$ gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memoVMM),width=1920,height=1080' ! nvvidconv ! nvv4l2h264enc ! nvv4l2decoder ! nvoverlaysink ts-offset=500000000
11

Hi DaneLLL，ts-offset does not help.

12

We have tested it before sharing the pipeline. A bit strange it doesn’t work at your end. Are you able to try different value?

13

“videotestsrc->encode->decode->render” has no such issue.

gst-launch-1.0 videotestsrc pattern=ball ! ‘video/x-raw,width=1920,height=1080,framerate=30/1’ ! nvvidconv ! nvv4l2vp9enc ! nvv4l2decoder ! nv3dsink

I also have some other strange findings.

gst-launch-1.0 v4l2src ! nvvidconv ! nv3dsink
This pipeline also stuck the video and the video stream is updated very slowly. I can confirm that R32.3.x (jetpack 4.3) has no such issue. And gst-launch output some warning messages:

WARNING: from element /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0:
There may be a timestamping problem, or this computer is too slow.

If I add do-timestamp=true, it seems OK again.
gst-launch-1.0 v4l2src do-timestamp=true ! nvvidconv ! nv3dsink

May this issue be related to timestamp or clock things?
Or video capture driver? (we capture video through hdmi to csi converter chip)

14

Hi,
The source code of nvarguscamerasrc is in
https://developer.nvidia.com/embedded/L4T/r32_Release_v4.4/r32_Release_v4.4-GMC3/Sources/T186/public_sources.tbz2

By default it sets do-timestamp=true and is-live=true. You can try other setting and rebuild/replace

/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so

In 1920x1080p30 + setting ts-offset, wo don’t see any issue. Probably it is specific to the source. Setting sink to sync=false to disable gstreamer synchronization mechanism should be a fine solution, but if it is not acceptable in your usecase, please check the source code and try other settings.