Hello NVIDIA Developer Community,
I would like to share a real-time gesture-controlled 3D game experiment that I developed for the NVIDIA Jetson Orin Nano Super.
The project combines live UVC camera capture, CUDA image preprocessing, TensorRT gesture recognition, application-state processing and OpenGL 3D rendering.
The main goal is to build a low-latency camera-to-action pipeline in which recognized hand gestures immediately trigger attacks, defensive effects and character reactions inside a real-time 3D scene.
Real-Time Video Demonstration
The following video shows the current implementation.
The demonstration includes:
- Live UVC camera input
- Real-time hand-gesture recognition
- Fist and open-palm gesture inputs
- First-person weapon rendering
- Gesture-triggered fire and energy attacks
- 3D enemy hit reactions
- Player and enemy status displays
- Particle, flame and shield effects
- Battle-state transitions
- A victory screen after the target is defeated
The video is intended as a technical demonstration of the complete camera-to-gameplay pipeline rather than a formal performance benchmark.
Processing Pipeline
The current processing path is approximately:
V4L2 UVC Camera Capture
→ CUDA YUYV Image Preprocessing
→ TensorRT Gesture Inference
→ CUDA Detection Post-Processing
→ Gesture-to-Action State Mapping
→ CUDA Effect Data Generation
→ OpenGL 3D Rendering
The intention is to avoid unnecessary CPU-side image processing and reduce the delay between the physical hand gesture and the corresponding visual response.
Camera Capture and CUDA Preprocessing
The application captures frames from a UVC camera through V4L2.
The camera provides YUYV frames. CUDA kernels are used for operations including:
- YUYV-to-RGB conversion
- Letterbox resizing
- Image normalization
- Tensor layout conversion
- TensorRT input preparation
The current implementation uses V4L2 memory-mapped buffers and makes the captured data accessible to the CUDA processing path.
One of the main design goals is to minimize repeated frame copies between capture, inference and rendering stages.
TensorRT Gesture Recognition
The gesture-recognition model is executed with TensorRT.
The current gesture classes used by the game include:
- Fist
- Open palm
After inference, the detection results are decoded and filtered before being passed to the gameplay-action system.
A confidence threshold and short temporal validation are used to reduce accidental attacks caused by unstable detections.
Gesture-to-Action Mapping
Recognized gestures are converted into gameplay events.
For example:
Fist
→ Weapon attack or fire-based ability
Open Palm
→ Energy attack, defensive effect or alternate ability
The action system also manages:
- Gesture confidence
- Attack cooldown
- Player energy
- Enemy health
- Hit confirmation
- Attack state
- Recovery state
- Victory state
This separation allows the gesture-recognition thread to remain independent from the OpenGL rendering loop.
Multi-Threaded Runtime
The application is organized into several processing stages:
- Camera capture thread
- TensorRT inference thread
- Gesture and gameplay-state thread
- CUDA effect-update stage
- OpenGL rendering thread
The objective is to prevent camera capture or inference from directly blocking the rendering loop.
Shared frame, detection and action-state data are transferred between stages using synchronized buffers.
CUDA and OpenGL Rendering
OpenGL is used to render:
- The 3D environment
- The enemy character
- The first-person weapon
- HUD information
- Hit reactions
- Flame and energy effects
- Screen-facing attack effects
CUDA is used where appropriate to update dynamic effect data, particle positions and attack geometry.
A major area of investigation is the synchronization cost between CUDA-generated data and the OpenGL rendering stage.
End-to-End Latency
For this application, the important latency is not only TensorRT inference time.
The complete latency path includes:
Camera exposure and capture
→ V4L2 buffer delivery
→ CUDA preprocessing
→ TensorRT inference
→ Detection filtering
→ Gesture-state confirmation
→ Gameplay event generation
→ OpenGL rendering
→ Display presentation
A low inference time does not necessarily guarantee a low gesture-to-screen response time if synchronization or buffering introduces additional delay.
I am currently using tegrastats and Nsight Systems to investigate GPU utilization, CPU waits and synchronization gaps between these stages.
Current Optimization Areas
The main areas I am currently working on include:
- Reducing gesture-to-action latency
- Improving V4L2 buffer management
- Minimizing unnecessary memory copies
- Reducing CUDA and OpenGL synchronization stalls
- Improving multi-thread coordination
- Preventing repeated gesture activation
- Reducing transparent particle overdraw
- Stabilizing frame time during large attack effects
- Maintaining long-duration runtime stability
- Measuring the complete camera-to-display latency
Questions for the Community
I would appreciate feedback on the following technical questions:
-
For a UVC camera using V4L2 on Jetson Orin Nano, is V4L2 MMAP together with CUDA-accessible memory a reasonable approach, or would DMABUF, EGLImage or NvBufSurface provide a more efficient camera-to-CUDA path?
-
What is the recommended synchronization method when TensorRT inference, CUDA-generated effect data and OpenGL rendering run in separate stages?
-
Is double buffering or triple buffering generally recommended for transferring gesture and effect-state data between CUDA and OpenGL?
-
What is the best method for accurately measuring end-to-end gesture-to-display latency on Jetson?
-
For repeated flame and transparent energy effects, what rendering strategy is recommended to reduce overdraw on Jetson Orin Nano?
-
Which Nsight Systems traces are most useful for identifying stalls between camera capture, TensorRT inference, CUDA processing and OpenGL rendering?
If necessary, I can also prepare a minimal reproducible sample that isolates the camera, TensorRT gesture inference and action-trigger pipeline without including the complete game assets.
Thank you for reading.
Any feedback regarding Jetson camera processing, TensorRT integration, CUDA–OpenGL interoperability or real-time latency optimization would be greatly appreciated.