Making My Toddler’s Dream of Flying Come True with Jetson AGX Xavier and AI Tech

Robotics & Edge Computing Jetson & Embedded Systems Jetson Projects
1

Griffin is built using a pose estimation model, an object detection model created using Amazon SageMaker JumpStart, a gesture recognition system and a 3D game engine written in OpenGL running on an NVIDIA Jetson AGX Xavier, I built Griffin, a game that let my toddler use his body to fly as an eagle in a fantasy 3D world.

Based on Dexie’s interests in eagles and flying, and me not being able to go anywhere during COVID-19 on our two-week Christmas break, I thought it would be cool if I could build a system or game that could give him an eagle flying experience. Furthermore, to enhance the immersion, I wanted him to be able to use his body to control the eagle body — standing on a tree branch and flapping his wings to prepare for flight using his actual arms, jumping to take off and lifting his arms side to side to steer direction during mid-flight.

Dexie-Demo
Dexie-Demo1546×870 300 KB

To achieve the above, Griffin needed to have the following modules:

  • 3D Game Engine — to bring the 3D fantasy world with hills, mountains, blue skies and Griffin to life using a flight simulator written in OpenGL.
  • Body Posture Estimation — to constantly detect the player’s body posture as an input to the system to control Griffin using the OpenPose Pose Estimation Model and an SSD Object Detection Model.
  • Action Mapping and Gesture Recognition — to transform the body posture into a meaningful action and gesture, such as lifting the left/right wing, rolling the body side to side, jumping to take off, etc.
  • Communication System — to send the gesture input into the 3D game engine using a socket. I will discuss later why we need this.

Here is the list of the hardware required to run the system:

  • NVIDIA Jetson AGX Xavier — a small GPU-powered, embedded device that will run all the modules above. This is the perfect device for the job because it can support video and audio output via a simple HDMI port, and it has an ethernet port for easy internet access. You can even plug in a mouse and a keyboard to develop and debug right on the device, as it has a fully functional Ubuntu 18.04 OS.
  • TV (with an HDMI input and a built-in speaker) — as a display to the game engine.
  • Camera — Sony IMX327. It’s an awesome, tiny, Full HD ultra-low light camera. Honestly, I could have gone for a much lower end camera, as I only need 224x224 image resolution. However, since I already had this camera for another project, then why not use it?
  • Blu-Tack — to glue everything together and to make sure everything stays in place. :)

The full blog is available here

Full source code

Connect with me at https://www.linkedin.com/in/agustinus-nalwan

5 Likes
2

Hi @msubzero2000, great to see you back with another awesome project! Great job! Looks like a lot of fun!

3

thanks for the kind words dusty!

4

Thank you for the inspiring project.

Have you considered to combine your pose estimation/control with VR?
E.g. Birdly by Soniacs uses direct body control in their bird-flight VR world which could be substituted by a low cost pose control:

1 Like