I am trying to find the lowest-latency method for pulling frames out of LibArgus. My code, which I can benchmark, takes about 32 ms (two frames) from requesting LibArgus for a frame to sending it on the wire. The receiver benchmarks at most 16ms (one frame) of latency from starting to receive a frame to putting it on screen. However, when I benchmark my end to end latency using a photodiode latency tester, I get around 125ms of latency.
When we use a gstreamer pipeline with nvarguscamerasrc, which involves compressing and decompressing the video data and a program at the other end which displays it, our end-to-end latency is about 95 ms. Since I’m not doing any compression, and I’m using a much higher bandwidth link with my new program, I would expect to see at least the same performance on my end.
Looking at Low-Latency CSI Camera Stream - #10 by tolazzi and running argus_camera with --kpi, it claims to have 22 ms of latency. However, that seems to be latency from receiving frames in libArgus to displaying them on screen, because when I characterize argus_camera with the photodiode latency tester (flash a bright light at a photodiode, record the time it receives, have a second photodiode on the monitor that records the time it sees the camera output its flash) I get about 75ms. It seems unlikely that we could achieve 95 ms latency with that much latency just to output to the monitor on the Jetson.
Is there a recommended set of settings or interfaces to get minimum latency out of LibArgus? Clearly nvarguscamerasrc is very, very fast, and I’d like to make our application as low-latency as possible.
Right now, I capture frames using a repeating request to an ICaptureSession of an EGLStream. I noticed there is a Buffer stream type instead of an EGLStream, should I be using that instead? Should I not use a repeating request and instead make singular requests every frame? Are there any tips any users have to recommend how to speed up requesting frames from LibArgus? Is there an argus or multimedia API example that is known to be very low latency?
I am happy to answer any information about my own code base, but I am looking specifically for information about the libargus API and if there are any better methods to reduce latency.
I can understand that for specifically the test setup. However, I have a couple of follow-ups. Most importantly, we currently have a working pipeline that runs at 90ms glass-to-glass latency. We run a Gstreamer pipeline:
The 150 framrate isn’t actually true, it’s only 60 fps. We cannot run this camera faster.
I have a receiving program that uses NVDec to decode the stream and send it to a graphical program. That pipeline includes an encode step and a decode step on the receiving program, and yet it runs at 90ms total. My current work uses the same camera, does not have an encode/decode step, and uses a program written using LibArgus following several of the examples. It runs with about 135 ms glass-to-glass latency.
Is there anything known in LibArgus C++ implementations that runs higher latency than nvarguscamerasrc? I’m cutting out a huge chunk of work by removing the encode/decode step, but I have increased our latency by >40 ms!
I can always blindly try different things available in the Argus samples available, but I was hoping that there might be known tradeoffs with different ways to use LibArgus. Nvarguscamerasrc can clearly do the speed that I want, so I’m hoping that Argus has methods that work for the speeds i want to achieve.