Background: I am a software engineer with Huupe. Huupe.com is building the world’s first intelligent basketball backboard supporting games, training, and casting content. The entire backboard is a TV screen. Cameras and computer vision detect from where shots are taken and if the shots are made or missed. Huupes stream live and on demand video from the cloud, to other Huupes, and from casting mobile devices. Huupe prototypes are using Jetson Xaviers, but we are planning to upgrade to Jetson Orins.
What are the best tools for hardware accelerated streaming video on Jetsons?
We have heard a lot about GStreamer, is GStreamer the best option?
Our current application is built with React in Electron, but we are currently looking to migrate to another framework, so far we have considered Qt and GTK+.
What UI frameworks can be used with GStreamer? What framework is preferred on Jetsons?
Can we use GStreamer, or another multimedia player that uses hardware acceleration with Qt, GTK+, and/or Electron.
Is there another UI framework that allows us to build applications on the Jetson with hardware accerelated video streaming, WebRTC, and integration with native hardware such as bluetooth that exists that we should be considering?
Hi,
We support gstreamer and low-level jetson_multiemdia_api. It looks like jetson_multiemdia_api is better for your use-case. Please flash system image and install SDK Components on Xavier NX through SDKManager and you will see the samples in
/usr/src/jetson_multimedia_api
The samples demonstrate hardware functions like encoding, decoding, conversion, TensorRT. Please take a look and give it a try.
@DaneLLL,
Some additional questions along the same lines.
We are looking into layering our application with the video player, either the jetson multimedia api or GStreamer, some questions about this.
Can we overlay app UI over the video?
Can we display video in a window of an app, with either the jetson multimedia api, or GStreamer?
Can we concurrently show multiple videos in separate video windows? Either with a picture in picture in an application, or by layering the video player and application interface.
Can we use GStreamer or jetson multimedia api with WebRTC?
Let me share a little bit of my expertise on this to try to answer some of your questions:
“Can we use GStreamer, or another multimedia player that uses hardware acceleration with Qt, GTK+, and/or Electron”
R/Yes, I actually would say that for your use case GStreamer sounds like a great option for you due to its interoperability with the frameworks that you are currently considering (Qt, GTK+) and also the NVIDIA software stack (VPI, CUDA, Multimedia API, etc), its modularity and proven functionality. GStreamer has a strong community and support, and at RidgeRun we have been involved in dozens of GStreamer projects already deployed in production so I am confident in strongly recommending GStreamer.
“Is there another UI framework that allows us to build applications on the Jetson with hardware accerelated video streaming, WebRTC, and integration with native hardware such as bluetooth that exists that we should be considering?”
R/ I don’t have more UI frameworks recommendations apart from Qt or GTK+ but let me answer this partially by sharing that we also have a GStreamer plugin that proves hardware accelerated video streaming with WebRTC, you can read more about it at: GStreamer WebRTC Wrapper | WebRTC Wrapper | RidgeRun GStreamer WebRTC Wrapper - RidgeRun Developer Connection
Can we display video in a window of an app, with either the jetson multimedia api, or GStreamer?
R/Yes, this possible with both Qt and GTK+. Actually there is some basic tutorial documentation that demonstrates this functionality on how to integrate a GUI and GStreamer: Basic tutorial 5: GUI toolkit integration
Can we concurrently show multiple videos in separate video windows? Either with a picture in picture in an application, or by layering the video player and application interface.
R/Yes, this is a extension of the previous answer. GStreamer “media pipelines” paradigm offers the option to split (or merge) streams into multiple ones that ultimately could be displayed in separate windows.
Can we use GStreamer or jetson multimedia api with WebRTC?
R/Yes, please refer to my previous answer about our GStreamer-WebRTC plugin.
Your project sounds really exciting so if you are interested in the expertise that we can provide don’t hesitate to contact us.
Thanks for all of your insight and information, this is really helpful. I am going to try and utilize some of the tutorials and resources you recommended to get a basic application working with GStreamer. Thanks!