Best Frameworks for Jetson For Computer Vision/Streaming TV/IoT

[cp17]

Huupe would like your recommendations on platforms to best exploit Nvidia Jetsons for our intelligent basketball backboard.

OS: JetPack vs Yocto

• Yocto: We are considering using Yocto instead of Jetpack because we believe that it will be easier to make changes in Yocto when we build production systems.

Application Frameworks being considered::

• ReactJS in GTK WebKit with GJS/Typescript
• GTK/C++/Boost
• Electron/ReactJS/Chromium/Typescript

Our current net:

• We will choose GJS/ReactJS if we can display our app over hardware accelerated video with an acceptable level of effort for updating packages and level of UI processing overhead.
• We will use GTK/C++/Boost if we can overcome build issues.
• Else we will stick with the Electron/ReactJS/Chromium that we have used in our prototype if we can hardware accelerate video.

Background: Huupe.com is building an intelligent basketball backboard supporting games, training, and casting content. The entire backboard is a TV screen. Cameras and computer vision detect from where shots are taken and if the shots are made or missed. Huupes stream video from the cloud, to other Huupes, and from casting mobile devices. We will stress the Nvidia Jetson that is controlling the Huupe by running AI, video processing, and generating and displaying graphics and animations on the Huupe screen. Note that Huupes are controlled by mobile devices via BLE/wifi and will not require on screen UI controls. Huupe prototypes are using Jetson Xaviers, but we are planning to upgrade to Jetson Orins.

Alternatives being considered:

• ReactJS in GTK WebKit with GJS/Typescript

  • Pros:
    • ReactJS has higher app developer efficiency than GTK/Boost.
    • ReactJS has a larger app developer community than GTK/Boost.
    • Typescript use is growing more than C++.
    • More efficient than Electron/Chromium.
  • Concerns:
    • IPC between GTK and ReactJS hard to debug.
    • Updating packages can require significant effort.

• GTK/C++/Boost

  • Pros:
    • HW acceleration of UI
    • More efficient processing than ReactJS….
  • Electron/ReactJS/Chromium is used in our current prototype.
    • Concerns:
      • Overhead of Electron/Chromium in the long run as we add capabilities.
      • Chromium doesn’t support hardware acceleration for video.

Note: We are using React Native for our iOS and Android mobile app which is used to control the Huupe backboard.

[pwolfe1]

I would personally recommend Yocto over Jetpack for what you are doing as long as you are comfortable maintaining a Yocto build. The OE4T Tegrademo repository is a great starting point and if you start off by learning how to build this and then do incremental changes to it you will be able to ramp up with Yocto as your prototype becomes more mature.

With regard to your framework question, you may want to consider breaking up your stack a bit more. For example, you may find it easiest to do most of the UI work with React but still use a C++ based application for video processing. This really depends on what kind of sensors you are using and how high level their interfaces are as well. While you mention GTK I didn’t see any specific video processing framework mentioned in your post. I have had a fantastic experience using GStreamer on the Jetson platform and would highly recommend it. And while it is not fully open source, NVIDIA’s Deepstream plugin suite for GStreamer provides both a relatively high degree of customization as well as easy access to the hardware acceleration the Xavier/Orin are capable of. If you are doing all or most of your computation on CPU you are really not getting close to the full value out of these Jetson products so I would really recommend an approach that is able to utilize hardware acceleration. Apologies if I am missing something here and you have already sorted out your video processing solution.

I do not have as much experience running web applications on the Jetson hardware but it should be basically the same as running such an application in a generic environment.

Best of luck with your product and feel free to reply with follow up questions if you have any.