Reduce latency from camera

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

I am seeing a large latency between the camera and HDMI output (as well as USB output to robotics controller). When I say large I looks to be 250ms for USB and about 225ms using a TX2 onboard camera (on the Xavier). I am getting 40fps with the webcam and about 30 with the TX2 camera.

I am using this https://github.com/naisy/realtime_object_detection

Any ideas on how to reduce this?

2

hello scbean,

may I know had you tried to boost system performance to check if there’s improvements.
you may refer to Supported Modes and Power Efficiency session for changing the power mode.
BTW, you should also update sensor device tree with tegra194-p2888-0001-p2822-0000-maxn.dtb if you need to configure to MaxN.
thanks

3

I was on MaxN mode, changed it to mode 6 (30w) and saw the latency increase a large amount and the frame rate of processing drop from 30FPS to about 7 FPS.

I am not sure what you mean by updating the sensor device tree, is there someone with more information on this?

Thanks!

4

hello scbean,

as you can see, mode-id=6 keep only two CPUs online.
my suggestion was boosting system performance by changing the power mode to MaxN, which is mode-id=0.
you may also refer to Topic 1044093 for the commands and details about updating MaxN device tree.
looking forward to your testing results.
thanks

5

With my system in MaxN I get the 250ms latency, so I need to also update the sensor device tree to be MaxN as well?

6

hello scbean,

yes, we need to use different bpmp dtb and kernel dtb for MaxN.
please replace the original device tree with tegra194-p2888-0001-p2822-0000-maxn.dtb.
thanks

7

So I finally got back to working on this and reflashed it with

sudo ./flash.sh -r -k kernel-dtb jetson-xavier-maxn mmcblk0p1

Using the info on the previously mentioned thread. Unfortunately, I see no reduction in latency, it is still is about 240ms with both the USB webcam and the TX2 camera.

I am assuming the updating of the kernel worked since on the host computer it something like *** The [kernel-dtb] has been successfully updated***

Any other ideas or is anyone else experiencing this much latency? Maybe it’s because of the problem I am using for object detection, although I am getting between 45-30 FPS on both cameras.

8

hello scbean,

  1. may I know how you measure the camera latency, could you please share the steps and also the commands.
  2. did you tried to remove object detection to check the latency again?
9

I am determining latency by pointing the webcam at a screen of a laptop and then displaying the video from the xavier on an adjacent screen. I take a photo which shows both timmers. I then subtract the time.

Using “cheese” to display the webcam I am getting roughly 60ms, so I guess there is a large amount of time being using for processing?

I am getting between 30-40FPS

(image)

10

hello scbean,

okay, you’re asking for capture-to-display latency.
please note that there’s buffer processing time included when sending image buffer to render display.

  1. may I know what’s your expectation results of the capture-to-display latency.
  2. could you please break-down the buffering time, by adding debug messages to check where’s bottleneck.
  3. please tried to reduce all other functions to evaluate the latency.
    for example,
$ gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,framerate=30/1,width=640,height=480 ! xvimagesink