Latency in video streaming (H.264)

Hi all,

I am using an embedded system with Tegra T30. An external camera is streaming H.264 video at 15fps and I get a delay of about 800ms.
If the camera switches to 5fps then the delay increases to almost 3s.

This is the pipeline that I am using:

gst-launch-0.10 -v udpsrc multicast-group=239.192.140.11 auto-multicast=true port=5004 ! application/x-rtp, media=video, payload=96, encoding-name=H264 ! queue ! rtph264depay ! nv_omx_h264dec ! nv_omx_overlaysink sync=false

In output, in the console, I see:

pnvsi -> nDecodeBuffer = 10

and this can clarify because I get more than 2s delay (latency) at 5fps.

Can I modify that parameter? How? Using Gstreamer?

Thanks
Marco

I don’t even know what the H.264 acceleration might be on a T30, but it would be typical on the newer Tegra SoCs that a plugin for gstreamer would need to be linked to one of the NVIDIA files (I’m pretty sure the T30 has H.264 acceleration, the question is if it is linked in to your gstreamer). Is this the Toradex Apalis T30?

Anyone know how to verify if gstreamer is using an NVIDIA version of H.264?

I’m not up to date with gstreamer, but to clarify, are you looking for a gstreamer option to control the low light slowdown via gstreamer on the assumption that this is a camera control option when it switches to low light behavior?

What do you mean exactly with “link”?
How can I check that?

Thanks

It’ll probably be much easier if someone familiar with gstreamer knows of a simple way to determine if the NVIDIA accelerated libraries are used in a particular pipeline, but I’ll give some information below on the more complicated part of the topic. Hopefully someone familiar with gstreamer will have a quick/easy way to know if H.264 in a given pipeline is hardware accelerated or not.

Almost all user space software will be dynamically linked against libraries, e.g., any “hello world” type application will use libc. The “strace” tool traces system calls to the kernel…these are function calls which don’t go to a library (this is the communications the program uses directly to the kernel). The ltrace tool traces library calls (and it is a library call which might optionally go to an NVIDIA hardware accelerated file versus a generic CPU-only file). These tools work on a running program and produce a LOT of output (think of it as a list of every function call which goes outside of the program, along with categorizing as either a call to the kernel or a call to a library). So ltrace would tell you about library calls. For an example, assuming “ls” is at “/usr/bin/ls”:

ltrace /usr/bin/ls

(the information is about what was passed to/from a library for individual function calls)

To statically determine what an executable file links to for a library (without needing to look at individual function calls) you would use ldd. Example:

ldd /usr/bin/ldd

Notice that ldd names libraries which are part of the system linker path. To see where the system linker looks to find those libraries:

ldconfig -p

(when ldd tells you which library it uses ldconfig can then be invoked to cross-reference where the file is physically located on the file system)

It is also possible that a program can dynamically load a plugin style module which won’t show up under ldd (it isn’t the system linker path used for loading…loading takes place during runtime by an explicit request to link to a file), although ltrace would show the function calls. This is basically because a library can call another library, and ldd does not recurse into other libraries. This is also why extracting this information under gstreamer might be difficult…some of it loads as a module and is not directly incorporated into gstreamer (thus ldd might not show the linking).

To demonstrate a library calling another library assume “ldconfig -p” shows libc.so.6 is located at “/lib64/libc.so.6” (this is where one Linux PC desktop distribution puts the file…it may differ on your system):

> ldd /lib64/libc.so.6
        /lib64/ld-linux-x86-64.so.2 (0x0000562881d57000)
        linux-vdso.so.1 (0x00007fffa9d22000)

This implies that libc.so.6 is linked to, and makes calls to, ld-linux-x86-64.so.2 and linux-vdso.so.1 on this particular system. If you found a particular library or module which your gstreamer uses to provide the H.264 functions, then you could run ldd on it and determine if the file used to fulfill that function is NVIDIA’s hardware accelerated version, or if the file is a CPU-only version. You could run ltrace on your pipeline and perhaps get a clue to what library to look for, but that would be a LOT of work.

So…this is why I’m asking if anyone familiar with gstreamer knows a way to quickly determine if a given gstreamer pipeline uses a hardware accelerated NVIDIA version of H.264.

Using gst-inspect you can find which library implements the plugin. The output below was done on a TK1, but it should be possible to do the same for your T30.

gst-inspect-0.10 | grep nv_omx_h264dec
<b>omx</b>:  nv_omx_h264dec: OpenMAX IL H.264/AVC video decoder

So here the plugin nv_omx_h264dec is implemented by library omx.

ldd /usr/lib/arm-linux-gnueabihf/gstreamer-0.10/libgst<b>omx.</b>so
	libEGL.so.1 => /usr/lib/arm-linux-gnueabihf/tegra-egl/libEGL.so.1 (0xb6e39000)
	libGLESv2.so.2 => /usr/lib/arm-linux-gnueabihf/tegra-egl/libGLESv2.so.2 (0xb6e23000)
	libX11.so.6 => /usr/lib/arm-linux-gnueabihf/libX11.so.6 (0xb6d3d000)
	libgstreamer-0.10.so.0 => /usr/lib/arm-linux-gnueabihf/libgstreamer-0.10.so.0 (0xb6c95000)
	libglib-2.0.so.0 => /lib/arm-linux-gnueabihf/libglib-2.0.so.0 (0xb6bcd000)
	libgobject-2.0.so.0 => /usr/lib/arm-linux-gnueabihf/libgobject-2.0.so.0 (0xb6b8f000)
	libgstvideo-0.10.so.0 => /usr/lib/arm-linux-gnueabihf/libgstvideo-0.10.so.0 (0xb6b74000)
	libgstinterfaces-0.10.so.0 => /usr/lib/arm-linux-gnueabihf/libgstinterfaces-0.10.so.0 (0xb6b60000)
	libgstbase-0.10.so.0 => /usr/lib/arm-linux-gnueabihf/libgstbase-0.10.so.0 (0xb6b18000)
	libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xb6b0d000)
	libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6a24000)
	libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6a09000)
	librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xb69fb000)
	libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xb698f000)
	libnvidia-glsi.so.21.5 => /usr/lib/arm-linux-gnueabihf/tegra/libnvidia-glsi.so.21.5 (0xb6927000)
	libnvrm.so => /usr/lib/arm-linux-gnueabihf/tegra/libnvrm.so (0xb690c000)
	libnvrm_graphics.so => /usr/lib/arm-linux-gnueabihf/tegra/libnvrm_graphics.so (0xb68f7000)
	libnvos.so => /usr/lib/arm-linux-gnueabihf/tegra/libnvos.so (0xb68de000)
	libxcb.so.1 => /usr/lib/arm-linux-gnueabihf/libxcb.so.1 (0xb68c4000)
	/lib/ld-linux-armhf.so.3 (0xb6f3f000)
	libgmodule-2.0.so.0 => /usr/lib/arm-linux-gnueabihf/libgmodule-2.0.so.0 (0xb68b9000)
	libxml2.so.2 => /usr/lib/arm-linux-gnueabihf/libxml2.so.2 (0xb67dc000)
	libpcre.so.3 => /lib/arm-linux-gnueabihf/libpcre.so.3 (0xb67a4000)
	libffi.so.6 => /usr/lib/arm-linux-gnueabihf/libffi.so.6 (0xb6796000)
	liborc-0.4.so.0 => /usr/lib/arm-linux-gnueabihf/liborc-0.4.so.0 (0xb673e000)
	libnvidia-rmapi-tegra.so.21.5 => /usr/lib/arm-linux-gnueabihf/tegra/libnvidia-rmapi-tegra.so.21.5 (0xb672b000)
	libXau.so.6 => /usr/lib/arm-linux-gnueabihf/libXau.so.6 (0xb6720000)
	libXdmcp.so.6 => /usr/lib/arm-linux-gnueabihf/libXdmcp.so.6 (0xb6714000)
	libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xb66f9000)
	liblzma.so.5 => /lib/arm-linux-gnueabihf/liblzma.so.5 (0xb66d9000)
	libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb66b8000)

Well, as said, in my pipeline I am using nv_omx_h264dec and I can see the video, no complaints from get-launch-0.10.
I will check as soon as possible.

Furthermore, do you know if it is possible to modify this parameter? I think it belongs to HW Decoder, not to GStreamer

pnvsi -> nDecodeBuffer = 10

Thanks
Marco

Assuming the nv_omx_h264dec is linked to hardware decoding (and it should be since it is the nv version) I would tend to suspect other parts of the system are causing the latency increase, e.g., data throughput or some other driver thread hogging CPU time. I’m still not clear if the issue is what it sounds like: Is it correct that performance is good at normal lighting, but the camera itself correctly switches to 5fps in low light, and in this condition latency hits? I.e., is it a latency issue and the 5fps itself is not the issue?

You might check on whatever BSP documents say about performance modes versus battery saving modes. Additionally you might try to renice your process to a higher priority (you can run htop on the Apalis T30, not sure what you are using), e.g., normal is “0”, renice to “-2” (meaning “not as nice”) would show some significant change if it is another process hogging the system (I wouldn’t increase the priority beyond “-5”, but you could try up to that).

Sorry, no cameras here to try, though someone might get a better idea if you mention the particular camera model and how it is connected (e.g., USB).

Yes, using gst-inspect-0.10 I get same output as you, but then my library is in /usr/lib/gstreamer-0.10 and not in /usr/lib/arm-linux-gnueabihf/gstreamer-0.10/ like by you.

Yes, when streaming at 15fps I have 800ms which in my opinion is also quite a lot.
When camera streams at 5fps (yes, it is correct), I have a latency of almost 2.5s which is too much.
I though it could be a problem of “wrong” pipeline, but then I saw this line in decoder output

pnvsi -> nDecodeBuffer = 10

And if 10 refers to number of frames which shall fill the buffer, it matches with the latency I see at 15fps and 5fps.

Sorry, I don’t know enough about gstreamer to be of much help. It does sound like the latency is from buffering, but I do not know how to control buffer depth.