Hello NVIDIA Developer Community,
I would like to share a real-time zero-copy AI vision pipeline that I developed for the NVIDIA Jetson Orin Nano Super.
The main objective of this project is to reduce unnecessary memory transfers between camera capture, CUDA preprocessing, TensorRT inference, GPU post-processing and OpenGL rendering.
The current processing pipeline is:
V4L2 Camera Capture
→ CUDA Image Preprocessing
→ TensorRT Inference
→ CUDA Detection Decoding and NMS
→ CUDA–OpenGL Interoperability
→ OpenGL Real-Time Rendering
Real-Time Demonstration
The following video shows the complete pipeline running in real time on the Jetson Orin Nano Super.
The demonstration includes live UVC camera input, CUDA image preprocessing, TensorRT object detection, GPU-side post-processing and OpenGL visualization.
Project Architecture
The project is divided into several processing stages:
- V4L2 camera capture
- CUDA image preprocessing
- TensorRT inference
- CUDA detection decoding and NMS
- Application and detection-state processing
- OpenGL real-time rendering
The application uses a multi-threaded architecture so that camera capture, inference and rendering do not unnecessarily block one another.
Camera Capture
The camera input is captured through V4L2 using memory-mapped buffers.
The current camera format is YUYV. Instead of performing the complete preprocessing stage on the CPU, the captured frame data is made accessible to CUDA for GPU-side processing.
The preprocessing pipeline includes:
- YUYV-to-RGB conversion
- Letterbox resizing
- Image normalization
- Tensor layout conversion
- TensorRT input preparation
These operations are implemented with CUDA kernels.
TensorRT Inference
The preprocessed image is passed to a TensorRT FP16 engine for real-time inference.
The implementation uses the TensorRT I/O tensor API and enqueueV3.
After inference, the output remains on the GPU for additional processing.
The post-processing stage includes:
- Detection decoding
- Confidence filtering
- Bounding-box conversion
- CUDA-based non-maximum suppression
- Final detection-data generation
The purpose is to avoid transferring the complete TensorRT output back to the CPU for decoding and NMS.
CUDA–OpenGL Rendering
The rendering stage uses OpenGL for real-time visualization.
Where applicable, CUDA and OpenGL share graphics resources so that rendering data can be generated or updated without unnecessary intermediate CPU copies.
The visualization includes:
- Live camera frames
- Detection bounding boxes
- Confidence values
- Runtime status information
- Inference and rendering statistics
Multi-Threaded Design
The application currently separates the main workloads into multiple threads:
- Camera capture thread
- TensorRT inference thread
- Application-state processing thread
- OpenGL rendering thread
This design allows camera capture, inference and rendering to progress independently while using synchronization mechanisms to protect shared frame and detection data.
Zero-Copy Design Goals
The project attempts to minimize the following operations:
- Repeated camera-frame copies between CPU buffers
- CPU-based image preprocessing
- TensorRT output transfers for CPU post-processing
- Repeated CPU-to-GPU uploads for rendering
- Unnecessary synchronization between pipeline stages
The current implementation uses V4L2 MMAP buffers together with CUDA-accessible memory and GPU-side processing.
I understand that the term “zero-copy” can describe several different memory paths on Jetson.
In this project, it refers primarily to reducing explicit application-level copies and keeping preprocessing, inference, post-processing and rendering data on the GPU side whenever possible.
Current Development Focus
I am currently investigating several possible improvements:
- DMABUF-based camera buffer sharing
- EGLImage integration
- CUDA stream synchronization
- CUDA–OpenGL synchronization
- TensorRT execution latency
- Frame-buffer lifetime management
- End-to-end camera-to-display latency
- Long-duration stability
- More accurate GPU and CPU profiling
Questions for the Community
I would appreciate technical feedback on the following questions:
-
For a V4L2 UVC camera on Jetson Orin Nano, is registering memory-mapped buffers with CUDA a reasonable approach, or would a DMABUF/EGLImage pipeline provide better performance?
-
What synchronization method is recommended when CUDA updates resources that are subsequently rendered by OpenGL?
-
Are there Jetson-specific limitations or best practices for using CUDA–OpenGL interoperability in a long-running real-time application?
-
What is the most reliable way to measure complete camera-to-display latency on Jetson?
-
Are Nsight Systems and Nsight Compute sufficient for identifying synchronization gaps across capture, CUDA, TensorRT and OpenGL stages?
Thank you for reading.
Any technical feedback, performance suggestions or architecture recommendations would be greatly appreciated.