Hello,
I’m writing a super simple video player using Python, PyGST, and PyGTK. This python app is containerized with Docker, and the videos to be played are 4K, so hardware acceleration is needed.
Here is a description of the issue I’m having:
When my script is run with a smaller video as a source on the regular docker runtime, the video plays inside the GTK window without a problem (it plays inside a DrawingArea and uses enable_sync_message_emission).
When run with a 4K video on the regular docker runtime, the playback stutters (obviously, since there is no acceleration), and stays inside the GTK DrawingArea.
However, when running it on the nvidia runtime using the --runtime nvidia flag, the video plays smoothly BUT outside of the application window (DrawingArea), and takes over the whole screen.
I need to be able to render other views on top of the video, so I’m looking for a way to ensure that the video will play in the window. I’ve spent days reading and trying different things, and I believe that the problem has to do with the fact that the sync-message::element message is never emitted by the GST playbin. This only happens with the nvidia runtime…
Any pointers are greatly appreciated.
Adding my Dockerfile:
FROM nvcr.io/nvidia/l4t-base:r32.6.1
ARG DEBIAN_FRONTEND=nointeractive
WORKDIR /app
RUN apt-get update
RUN yes | apt install libgirepository1.0-dev gcc libcairo2-dev pkg-config python3-dev gir1.2-gtk-3.0 python3-gst-1.0
RUN yes | apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio gstreamer1.0-tools gstreamer1.0-alsa gstreamer1.0-plugins-base gstreamer1.0-plugins-good
RUN apt-get -y install python3 python3-pip
RUN apt -y install git
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
CMD ["python3", "app.py"]
It might help to know what is the gstreamer pipeline, especially on sink/display side.
Especially if using gst-play or meta plugins such autovideosink, the nividia runtime may give other possible ways than without.
Adding python code + pipeline. It’s a simple playbin:
import sys, os
import gi
gi.require_version('Gst', '1.0')
gi.require_version('Gtk', '3.0')
gi.require_version('GstVideo', '1.0')
from gi.repository import Gst, GObject, Gtk
from threading import Timer
# Needed for window.get_xid(), xvimagesink.set_window_handle(), respectively:
from gi.repository import GdkX11, GstVideo
class GTK_Main(object):
def __init__(self):
window = Gtk.Window(Gtk.WindowType.TOPLEVEL)
window.set_title("Video-Player")
window.set_default_size(500, 400)
window.connect("destroy", Gtk.main_quit, "WM destroy")
vbox = Gtk.VBox()
window.add(vbox)
hbox = Gtk.HBox()
vbox.pack_start(hbox, False, False, 0)
self.entry = Gtk.Entry()
hbox.add(self.entry)
self.button = Gtk.Button("Start")
hbox.pack_start(self.button, False, False, 0)
self.button.connect("clicked", self.start_stop)
self.movie_window = Gtk.DrawingArea()
vbox.add(self.movie_window)
window.show_all()
self.player = Gst.ElementFactory.make("playbin", "player")
bus = self.player.get_bus()
bus.add_signal_watch()
bus.enable_sync_message_emission()
bus.connect("message", self.on_message)
bus.connect("sync-message::element", self.on_sync_message)
def start(self):
print("STARTING VIDEO", flush=True)
self.player.set_property("uri", "https://mylocation_in_aws")
self.player.set_state(Gst.State.PLAYING)
def start_stop(self, w):
if self.button.get_label() == "Start":
filepath = self.entry.get_text().strip()
if os.path.isfile(filepath):
filepath = os.path.realpath(filepath)
self.button.set_label("Stop")
self.player.set_property("uri", "file://" + filepath)
self.player.set_state(Gst.State.PLAYING)
else:
self.player.set_state(Gst.State.NULL)
self.button.set_label("Start")
def on_message(self, bus, message):
# print('on_message', message)
t = message.type
if t == Gst.MessageType.EOS:
self.player.set_state(Gst.State.NULL)
self.button.set_label("Start")
elif t == Gst.MessageType.ERROR:
self.player.set_state(Gst.State.NULL)
err, debug = message.parse_error()
print("Error: %s" % err, debug)
self.button.set_label("Start")
def on_sync_message(self, bus, message):
# print('on_sync_message', message)
if message.get_structure().get_name() == 'prepare-window-handle':
imagesink = message.src
imagesink.set_property("force-aspect-ratio", True)
imagesink.set_window_handle(self.movie_window.get_property('window').get_xid())
GObject.threads_init()
Gst.init(None)
w = GTK_Main()
Timer(2000, w.start())
Gtk.main()
Hi,
Please execute
$ export GST_DEBUG=*FACTORY*:4
And share what plugins are picked. For optimal performance it is better to run with nvoverlaysink on Jetson Nano.
Hi,
Thanks for your response.
Here is the output of the app after using the GST_DEBUG=*FACTORY*:4 flag:
Note: this is the output from the app inside a container on the nvidia runtime:
$ sudo docker run --runtime nvidia -e DISPLAY=$DISPLAY -e GST_DEBUG=$GST_DEBUG -v /tmp/.X11-unix/:/tmp/.X11-unix my-app
(app_2.py:1): dbind-WARNING **: 14:47:36.342: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-kBI29ZSbAq: Connection refused
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
nvbuf_utils: Could not get EGL display connection
0:00:04.491071255 1 0x4e89f30 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "playbin" named "player"
STARTING VIDEO
0:00:04.512160769 1 0x4e89f30 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "uridecodebin"
0:00:04.528835761 1 0x4e89f30 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "souphttpsrc" named "source"
0:00:04.531424253 1 0x4e89f30 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "typefind"
0:00:04.957523780 1 0x4a14320 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "decodebin"
0:00:04.957912174 1 0x4a14320 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "typefind" named "typefind"
0:00:04.958402289 1 0x4a14320 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "queue2"
0:00:05.005396679 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "qtdemux"
0:00:05.013643678 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "multiqueue"
0:00:05.017869393 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "h264parse"
0:00:05.019555888 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "capsfilter"
0:00:05.028332742 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "aacparse"
0:00:05.040403938 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "pulsesink"
0:00:05.050184511 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "alsasink"
0:00:05.063815167 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "jackaudiosink"
0:00:05.070611250 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "oss4sink"
0:00:05.073826683 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "openalsink"
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': No such file or directory
0:00:05.096707173 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "osssink"
0:00:05.293142518 1 0x7f7c4749e0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "avdec_aac"
0:00:05.385135579 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "nvoverlaysink"
0:00:05.418245141 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "nvv4l2decoder"
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
0:00:05.567857662 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "input-selector"
0:00:05.568867944 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "input-selector"
0:00:05.569236650 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "tee" named "audiotee"
0:00:05.570629076 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "bin" named "vbin"
0:00:05.570787569 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "queue" named "vqueue"
0:00:05.571667796 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "identity" named "identity"
0:00:05.573392051 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "videobalance" named "videobalance"
0:00:05.575124901 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "videoconvert" named "conv"
0:00:05.576923741 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "videoscale" named "scale"
0:00:05.590206734 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "videoconvert" named "conv"
0:00:05.590380539 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "videoscale" named "scale"
0:00:05.590647004 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "videoconvert" named "conv2"
0:00:05.622770815 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "bin" named "vdbin"
0:00:05.622853630 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "videoconvert" named "vdconv"
0:00:05.631734236 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "deinterlace" named "deinterlace"
0:00:05.635266551 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "autoaudiosink" named "audiosink"
0:00:05.635522234 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "fakesink" named "fake-audio-sink"
0:00:05.638212187 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "pulsesink" named "audiosink-actual-sink-pulse"
0:00:05.646516948 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "alsasink" named "audiosink-actual-sink-alsa"
0:00:05.647379571 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "jackaudiosink" named "audiosink-actual-sink-jackaudio"
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
0:00:05.651970710 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "oss4sink" named "audiosink-actual-sink-oss4"
0:00:05.652559004 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "openalsink" named "audiosink-actual-sink-openal"
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': No such file or directory
0:00:05.653369490 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "osssink" named "audiosink-actual-sink-oss"
0:00:05.653622673 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "fakesink" named "fake-audio-sink"
0:00:05.654132475 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "bin" named "abin"
0:00:05.654233103 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "queue" named "aqueue"
0:00:05.654644101 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "identity" named "identity"
0:00:05.655821105 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "volume" named "volume"
0:00:05.657071444 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "audioconvert" named "conv"
0:00:05.658696323 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "audioresample" named "resample"
0:00:05.659793065 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "audioconvert" named "conv"
0:00:05.659931454 1 0x7f782524a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "audioresample" named "resample"
Seems this is what I suspected, with nvidia runtime, nvoverlaysink is the prefered display sink (plugins can declare their ranking at build time and this would be used by meta-plugins such as playbin, but not sure nvoverlaysink plugin sources are available for rebuilding with a lower rank).
From what I’ve understood, your application is waiting for x window id, and nvoverlaysink is not X hosted.
You would have to try xvimagesink (may require nvvidconv for copying/converting from decoder output as NV12 in NVMM memory to system memory), or better try using nv3dsink instead.
I’m currently away from any Jetson for trying, but you may find some details about how to specify playbin video sink here.