Video playback is jittery when using multiple nv3dsink instances in the same program/process

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
,
1

Video playback is jittery when using multiple nv3dsink instances in the same program/process.
For example, “gst-launch-1.0 videotestsrc pattern=ball ! ‘video/x-raw,width=1920,height=1080,framerate=30/1’ ! tee name=t ! queue ! nvvidconv ! nv3dsink t. ! queue ! nvvidconv ! nv3dsink”

If replace one or all nv3dsink with xvimagesink, video playback is very smooth.
For example, “gst-launch-1.0 videotestsrc pattern=ball ! ‘video/x-raw,width=1920,height=1080,framerate=30/1’ ! tee name=t ! queue ! nvvidconv ! nv3dsink t. ! queue ! xvimagesink”

The below program has two pipelines and one nv3dsink per pipeline, the video playback is jittery. If replace one nv3dsink with xvimagesink, the playback will be smooth again.

#!/usr/bin/python3
import sys
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject

Gst.init(sys.argv)

a_pipeline = Gst.parse_launch('videotestsrc pattern=ball ! video/x-raw,width=1920,height=1080,framerate=30/1 '
#       '! xvimagesink'
       '! nvvidconv ! nv3dsink'
    )

b_pipeline = Gst.parse_launch('videotestsrc pattern=ball ! video/x-raw,width=1920,height=1080,framerate=30/1 '
#       '! xvimagesink'
       '! nvvidconv ! nv3dsink'
    )

a_pipeline.set_state(Gst.State.PLAYING)
b_pipeline.set_state(Gst.State.PLAYING)

loop = GObject.MainLoop()
try:
    loop.run()
except:
    print('Exitting...')
    b_pipeline.set_state(Gst.State.NULL)
    a_pipeline.set_state(Gst.State.NULL)
    loop.quit()
3

Hi,
The issue is known and we are checking internally. Suggest you use nvcompositor to composite sources into single video plane.

Related topics:
Multiple gstreamer pipelines slow down when in the same process - #10 by DaneLLL
Multiple instances of NvEglRenderer

5

Thanks for confirming this issue.

Is it possible that https://forums.developer.nvidia.com/t/video-is-jittery-through-capture-encode-decode-render-in-the-same-gstreamer-pipeline/156849/14 has the same root cause with this issue?

6

Hi,

It does not look like the same. If you don’t have multiple nv3dsink in single pipeline, it should work fine. And we have tried setting ts-offset and it helps. Not sure why it does not take effect with your sensor.

7

Hi DaneLLL,

When will this issue be fixed?
nvcompositor is not applicable to us.
Our application will have multiple windows on multiple monitors.

8

Hi,
The issue listed in the comment is fixed in Jetpack 4.5.1.

10

It’s a good news. I will try 4.5.1 release later

11

I just upgraded to 32.5.1 from 32.4.4. But there is a new issue.

$ gst-launch-1.0 videotestsrc pattern=ball ! nvvidconv ! nv3dsink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...

(gst-launch-1.0:5913): GLib-CRITICAL **: 15:37:34.766: g_array_remove_index: assertion 'index_ < array->len' failed

(gst-launch-1.0:5913): GStreamer-CRITICAL **: 15:37:34.766: gst_caps_remove_structure: assertion 'idx <= gst_caps_get_size (caps)' failed

(gst-launch-1.0:5913): GStreamer-CRITICAL **: 15:37:34.766: gst_caps_remove_structure: assertion 'idx <= gst_caps_get_size (caps)' failed

(gst-launch-1.0:5913): GStreamer-CRITICAL **: 15:37:34.766: gst_structure_set_parent_refcount: assertion 'refcount != NULL' failed

(gst-launch-1.0:5913): GStreamer-CRITICAL **: 15:37:34.766: gst_caps_features_set_parent_refcount: assertion 'refcount != NULL' failed
free(): invalid pointer
Aborted (core dumped)
12

without nvvidconv, the pipeline is OK, but CPU usage is higher considerably than the one in R32.4.4

gst-launch-1.0 videotestsrc pattern=ball ! nv3dsink

Even with h265 hardware decoding, the pipeline CPU usage is higher more than the one in R32.4.4

There is always a process occupying 100% cpu

13

Hi,
The nv3dsink plugin supports video/x-raw and video/x-raw(memory:NVMM). Please specify video/x-raw(memory:NVMM) and try again:

... ! video/x-raw(memory:NVMM) ! nv3dsink
14

If I add ‘video/x-raw(memory:NVMM)’ between nvvidconv and nv3dsink, there is no crash any more.

But in both videotestsrc and hardware video file decoding cases, gst-launch process cpu usage is very high compared with R32.4.4. One of cpus is always 100% usage

(gst-launch-1.0 filesrc location=Apink_MrChu.mp4 ! tsdemux ! h265parse ! nvv4l2decoder ! nv3dsink window-width=1280 window-height=720)

15

Hi,
We don’t observe the issue on r32.5.1/Xavier.

$ gst-launch-1.0 filesrc location= /opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_1080p_h265.mp4 ! qtdemux ! h265parse ! nvv4l2decoder ! nv3dsink window-width=1280 window-height=720

RAM 2189/31921MB (lfb 7167x4MB) SWAP 0/15960MB (cached 0MB) CPU [2%@2264,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265] EMC_FREQ 0%@2133 GR3D_FREQ 1%@1377 NVDEC 1075 VIC_FREQ 0%@115 APE 150 MTS fg 0% bg 1% AO@41C GPU@41.5C Tdiode@47C PMIC@100C AUX@40.5C CPU@42.5C thermal@41.4C Tboard@43C GPU 1218/1206 CPU 609/655 SOC 3655/3182 CV 0/0 VDDRQ 457/457 SYS5V 2184/2182
RAM 2189/31921MB (lfb 7167x4MB) SWAP 0/15960MB (cached 0MB) CPU [0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265] EMC_FREQ 0%@2133 GR3D_FREQ 1%@1377 NVDEC 972 VIC_FREQ 0%@115 APE 150 MTS fg 0% bg 1% AO@41C GPU@41.5C Tdiode@46.75C PMIC@100C AUX@40.5C CPU@42.5C thermal@41.4C Tboard@43C GPU 1218/1207 CPU 609/654 SOC 3503/3190 CV 0/0 VDDRQ 457/457 SYS5V 2184/2182
RAM 2189/31921MB (lfb 7167x4MB) SWAP 0/15960MB (cached 0MB) CPU [2%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265] EMC_FREQ 0%@2133 GR3D_FREQ 0%@1377 NVDEC 1075 VIC_FREQ 0%@115 APE 150 MTS fg 0% bg 1% AO@41C GPU@41.5C Tdiode@47C PMIC@100C AUX@40.5C CPU@42.5C thermal@41.4C Tboard@43C GPU 1219/1207 CPU 609/653 SOC 3351/3194 CV 0/0 VDDRQ 457/457 SYS5V 2184/2182

The CPU loading is low in tegrastats. Don’t observe one core is at 100%.

16

Such issue still exists in R32.6.1
The test command is

gst-launch-1.0 videotestsrc pattern=ball is-live=true ! 'video/x-raw,width=640,height=480,framerate=60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! tee name=t ! queue ! nv3dsink t. ! queue ! nv3dsink
17

Hi,
On r32.6.1, please run single nv3dsink in each process. If you have multiple sources, please use nvcompositor to composite the sources into single video plane.