Low-Latency CSI Camera Stream

Hello,
I’m trying to evaluate what is the absolute fastest way to output the image of an Mipi camera to a monitor via HDMI via a Jetson Nano. Right now I am using libargus via the nvarguscamerasrc gstreamer plugin with the following command:

gst-launch-1.0 nvarguscamerasrc exposuretimerange=“500000 500000” gainrange=“1 1” ispdigitalgainrange= “1 1” awblock=0 tnr_mode=0 ee-mode=0 ! ‘video/x-raw(memory:NVMM), width=1280, height=720,framerate=60/1’ ! nvoverlaysink

The camera in use is the Raspberry Pi V2 camera and the stream is displayed on a 60FPS monitor.

This gstreamer command results in a very low latency of around 50 ms [3 frames of the monitor].

However I need the additional control, that only the low-level C API of libargus provides but that is not exposed in the libarguscamerasrc gstreamer plugin (e.g. setting the the color correction matrix).

An basic example of how to use libargus with the C-API is provided in the tegra_multimedia_api under samples/09_camera_ jpeg_capture which is the basis for my test program. However even if I remove the JPEG thread of that example and just let the preview thread run, I can only get very few FPS and a very high delay in the playback of the camera image on the monitor.

My question is, why is using the C-API for libargus so much slower than the gstreamer plugin even though both should work similarly and second can I make the C-program as fast as the gstreamer pipeline?

Furthermore, what is the fastest way to get the result drawn to the screen? Is it via the NvEglRenderer? An X-server is not necessary for me.

I cannot comment on argus, but if gstreamer works fine for you, you may build the pipeline in your application so that you can perform your own processing (assuming with GPU in NVMM memory). See this opencv example (Not sure it was a good idea to use sobel on alpha channel, though, BGR should be enough).
You would adapt to your own case.

1 Like

Have a check with the argus_camera below result is I got from Pi imx219.

nvidia@nvidia-desktop:~$ ./argus_camera --kpi --sensormode=3
Executing Argus Sample Application (argus_camera)
Argus Version: 0.97.3 (multi-process)
PerfTracker: app initial 556 ms
PerfTracker 1: app intialized to task start 663 ms
PerfTracker 1: task start to issue capture 73 ms
PerfTracker 1: first request 169 ms
PerfTracker 1: total launch time 1462 ms
PerfTracker 1: frameRate 75.65 frames per second at 0 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 24 ms average, min 18 max 29
PerfTracker 1: frameRate 61.78 frames per second at 0 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 17 ms average, min 15 max 21
PerfTracker 1: frameRate 60.64 frames per second at 1 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 16 ms average, min 15 max 17
PerfTracker: display frame rate 52.62 frames per second
PerfTracker 1: frameRate 60.69 frames per second at 1 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 16 ms average, min 15 max 19
PerfTracker 1: frameRate 60.52 frames per second at 1 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 16 ms average, min 15 max 17
PerfTracker: display frame rate 60.04 frames per second
PerfTracker 1: frameRate 60.42 frames per second at 2 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 16 ms average, min 15 max 17
PerfTracker 1: frameRate 60.34 frames per second at 2 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 16 ms average, min 15 max 17
PerfTracker: display frame rate 60.00 frames per second
PerfTracker 1: frameRate 60.32 frames per second at 3 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 16 ms average, min 15 max 17
1 Like

With the same “argus_camera” program I get a very different result:

./argus_camera --kpi --sensormode=3
Executing Argus Sample Application (argus_camera)
Argus Version: 0.97.3 (multi-process)
PerfTracker: app initial 631 ms
PerfTracker 1: app intialized to task start 474 ms
PerfTracker 1: task start to issue capture 68 ms
PerfTracker 1: first request 1196 ms
PerfTracker 1: total launch time 2370 ms
PerfTracker 1: frameRate 2,01 frames per second at 0 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker: display frame rate 2,02 frames per second
PerfTracker 1: latency 51 ms average, min 51 max 53
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,02 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker 1: flush takes 2086 ms
PerfTracker 1: device close takes 8 ms
PerfTracker 1: total close takes 2095 ms

How can that be?

Thanks for that answer. That is kind of the fallback solution, but right now I specifically try to evaluate the performance of libargus, respectively the ISPs from NVIDIA which are only accessible via libargus.

Have a check with v4l2-ctl that the sensor cap can output 60fps correctlly.

nvidia@nvidia-desktop:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=720,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=300
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.39 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.19 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.13 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Yes it can a leat get very close. See here:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=720,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=300
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 58.41 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 58.20 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 58.33 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 58.25 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 58.20 fps

That is very weird. Why is the argus_camera output then so different for us?

Wow, that’s weird, which release for your problem.

I have the following release:

NVIDIA Jetson Nano (Developer Kit Version)
L4T 32.4.3 [ JetPack 4.4 ]
Ubuntu 18.04.5 LTS
Kernel Version: 4.9.140-tegra
CUDA 10.2.89
CUDA Architecture: 5.3
Argus Version: 0.97.3
Multimedia API: 32.2

The argus_camera programm produced a segfault when running at first, so I patched the headers “CameraDevice.h” and “Settings.h” as described by the mod JerryChang here: https://forums.developer.nvidia.com/t/l4t-r32-4-3-breaks-argus-camera-and-causes-segmentation-fault-sigsegv/141432/8. With these tiny patches the argus_camera program works.

I did not modify the system otherwise.

Can the patches be the source of the slowness?

I don’t think the source cause the problem.
Could you attached you binary here, I can verify it on my system.

Of course. Here is the binary (packed with gzip as otherwise I can’t upload it):
argus_camera.gz (480.1 KB)

Looks like your binary have problem. I just got 2 fps from this binary.

How can that be? That should not be possible.
I compiled that binary with the newest Multimedia API from the NVIDIA Webpage and the workflow excactly as described in the README.txt file in the tegra_multimedia_api/argus folder.
Can you give me a binary without that problem for testing purposes?

Have a check this binary it’s build from internal tree. I will try the release build later.

Just remove the .txt and chmod +x to run it.

argus_camera.txt (2.2 MB)

I tried your binary and it leads to the same result as you can see below:

./argus_camera --kpi --sensormode=3
Executing Argus Sample Application (argus_camera)
Argus Version: 0.97.3 (multi-process)
PerfTracker: app initial 656 ms
PerfTracker 1: app intialized to task start 436 ms
PerfTracker 1: task start to issue capture 64 ms
PerfTracker 1: first request 1213 ms
PerfTracker 1: total launch time 2371 ms
PerfTracker 1: frameRate 2,01 frames per second at 0 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker: display frame rate 2,02 frames per second
PerfTracker 1: latency 51 ms average, min 51 max 53
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,02 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker: display frame rate 2,00 frames per second
PerfTracker 1: flush takes 1715 ms
PerfTracker 1: device close takes 3 ms
PerfTracker 1: total close takes 1719 ms

Is maybe some shared library on the system broken in my case, which is linked into all the libargus executeables and this leads to the slow FPS?
Also why are the FPS exactly 2,00? tTat seems very weird to me. It’s not just slow, it’s exactly 2.

My system is really a fresh and unmodified install from the latest SD card image provided by NVIDIA.

My bad, attach the wrong file. Please try this one.

snchen@snchen-HP:~$ sha256sum /tmp/argus_camera
a1a0872c2801855ed837968fc0de5dde4be2c24c6f1f22f93a4196ad24b99652

argus_camera.txt (1.2 MB)

1 Like

./argus_camera --kpi --sensormode=3
Executing Argus Sample Application (argus_camera)
Argus Version: 0.97.3 (multi-process)
PerfTracker: app initial 584 ms
PerfTracker 1: app intialized to task start 561 ms
PerfTracker 1: task start to issue capture 67 ms
PerfTracker 1: first request 152 ms
PerfTracker 1: total launch time 1365 ms
PerfTracker 1: frameRate 72,26 frames per second at 0 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 17 ms average, min 16 max 20
PerfTracker 1: frameRate 60,38 frames per second at 0 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 15 ms average, min 15 max 18
PerfTracker 1: frameRate 60,39 frames per second at 1 Seconds
PerfTracker 1: framedrop current request 0, total 0
PerfTracker 1: latency 15 ms average, min 15 max 16
PerfTracker: display frame rate 60,31 frames per second
PerfTracker 1: frameRate 60,29 frames per second at 1 Seconds
PerfTracker 1: flush takes 87 ms
PerfTracker 1: device close takes 12 ms
PerfTracker 1: total close takes 100 ms

It works. But why? What is different?
Also your executeable is much smaller in file size. Mine is 2.2MB, yours us 1.2MB

There’s no problem for me to build from the public release source too.
The size is about 2.2MB too.
Attached my source here for reference.argus.tar.gz (2.0 MB)

Using the sources you provided I cannot compile with the commands as written in README.txt. The make command fails with:

In file included from /home/nico/testmod/samples/utils/gtk/Window.h:32:0,
from /home/nico/testmod/samples/utils/Window.h:34,
from /home/nico/testmod/apps/camera/renderer/Composer.h:36,
from /home/nico/testmod/apps/camera/renderer/Composer.cpp:40:
/home/nico/testmod/samples/utils/WindowBase.h:32:10: fatal error: Argus/Argus.h: No such file or directory
include <Argus/Argus.h>

I now managed to compile. I realized I have to copy it into the MMAPI folder to be able to compile.

And a wonder happened! The resulting executable runs smoothly with 60FPS!!!