We’ve been using Jetpack 4.6.4 on the Jetson Nano SOM for the past few months in our efforts to reduce latency without much success. Our application pertains to the Healthcare Industry, where low latency is critical.
Currently, we’re dealing with a camera setup that produces frames at 400x400 resolution, 30 frames per second, with a glass-to-glass latency of approximately 130ms.
Here’s a breakdown of our latency issues:
Camera Latency: Around 33ms is attributed to the camera itself as the camera driver handles most ISP operations.
Source-level Latency: We’ve observed 30ms to 60ms latency at the source level through experiments.
Pipeline Latency: The rest of the latency is incurred within our pipeline, involving transformations like scaling, rotation, and overlay.
We’ve also compared our setup with an FPGA-based solution, which demonstrated lower latency.
In attempts to mitigate the issue, we’ve already experimented with the following:
Adjusting QUEUE_BUFFERS to 2.
Setting v4l2-ctl --set-ctrl low_latency_mode to 1.
Upon analyzing logs and other similar cases, we’ve discovered that the Jetson releases frames at the start of N+2 frame, retaining the first 2 frames in its internal queue to apply exposure. Since we manage exposure at the hardware level and don’t require software exposure control, it would greatly benefit us if frames could be released at the end of the N frame.
Upon reviewing the timestamps in vi2_fops.c, we have encountered some unexpected results that we would like to discuss further. Our camera setup is configured to deliver a resolution of 400x400 UYVY at 30fps.
To accurately measure latency, we have adopted the following methodology:
We are utilizing sample 12 of the Jetson Multimedia API.
We have adjusted the V4L2_BUFFERS_NUM parameter to 1.
Latency measurements are obtained using a smartphone to assess glass-to-glass latency.
Observations and Findings:
1. Without Patch (Queued Buffers = 2 & Frame Release at Start of N+2 Frame):
Each frame is enqueued after 33ms (as calculated from the difference in Start Capture times).
Frame release occurs at the start of the N+2 frame, yielding release timings of x+0 and x+33, where ‘x’ denotes the time when the Nth frame is enqueued into the V4L buffer.