How to find latency for decoding video and encoding the video H264 codec gstreamer on jetson Orin

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
,
1

Hi,

I have a video file. I require to perform the latency test between the time from video encoding to decoding. Unfortunately, we can’t get the raw image data as the jetson orin doesn’t have 4 pin port for accessing the leopard cameras. We are going to perform the video decoding of a video file and re-encode that video file. During this time we want to compute the latency between these pipeline, Also latency of decoding the video to raw data and latency of encoding raw data into video file.

Can anyone please let me know how to do this? How to write this pipeline? The below is a script for decoding video.

FILE=~/fov60_25.mp4
gst-launch-1.0 filesrc location=$FILE ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nv3dsink -e
2

Not sure I correctly understand ‘the latency test between the time from video encoding to decoding’.

If you have a video file in MP4 format, it is already encoded and muxed into MP4 container. So what you want to measure would be the latency of disk/filesystem/gst-filesrc + mp4 demuxing + h264 parsing + h264 decoding + nv3dsink + display controler + your monitor latency (refresh rate).

3

HI, I am trying to measure the latency for gstreamer to encode a video and the latency of gstreamer to decode a video for H264 compression. Hence, I require a streamer pipline inoder to caclulate that.

I heard of the tool called gstshark as below:

It helps to compute inter-latencies between pipeline. I did try this as follows:

FILE=~/fov60_25.mp4
GST_DEBUG="GST_TRACER:7" GST_TRACERS="interlatency" gst-launch-1.0 filesrc location=$FILE ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nv3dsink -e

I have a video file, trying to decode and find its latency to complete this pipeline. But I am getting this following error:

 0:00:00.022626699  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf89ca580 (cpuusage)
0:00:00.022708652  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf89ca640 (graphic)
0:00:00.022728172  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf89ca700 (proctime)
0:00:00.022743468  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf8c08040 (interlatency)
0:00:00.022757740  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf8c08100 (scheduletime)
0:00:00.022772268  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf8c081c0 (framerate)
0:00:00.022787660  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf8c08280 (queuelevel)
0:00:00.022801549  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf8c08340 (bitrate)
0:00:00.022814893  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0xaaaaf8c08400 (buffer)
0:00:00.022969166  7243 0xaaaaf8c00e60 TRACE             GST_TRACER gsttracerrecord.c:111:gst_tracer_record_build_format: interlatency.class, from_pad=(structure)"scope\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PAD\;", to_pad=(structure)"scope\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PAD\;", time=(structure)"scope\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.022994094  7243 0xaaaaf8c00e60 DEBUG             GST_TRACER gsttracerrecord.c:125:gst_tracer_record_build_format: new format string: interlatency, from_pad=(string)%s, to_pad=(string)%s, time=(string)%s;
./decoding.sh: line 4:  7243 Segmentation fault      (core dumped) GST_DEBUG="GST_TRACER:7" GST_TRACERS="interlatency" gst-launch-1.0 filesrc location=$FILE ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nv3dsink -e

Hence, I was asking for a proper pipeline for this or any other method.

4

You may try:

gst-launch-1.0 filesrc location=$FILE ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=file_out.mp4
5

Thank you, I can decode and encode the video. I am getting the output.mp4 file. Terminal output as follows:

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Redistribute latency...
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 846000 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.968119607
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

If we see clearly there is execution ended time available. “Execution ended after 0:00:00.968119607”. If I can get the execution started time. I am guessing I can the latency of the whole process.

How to get the execution start time?

6

Also when I run only decode code:


FILE=~/fov60_25.mp4
gst-launch-1.0 filesrc location=$FILE ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nv3dsink -e

The terminal output is:

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:12.900334212
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

The execution time for only decode (Execution ended after 0:00:12.900334212) seems greater than decode +encode.

I am confused here, why is it more for decode only? and how is it measure i.e. milli seconds or nano seconds?

7

I can’t check now, but my guess would be that it may be related to sync. Try adding sync=0 to nv3dsink.

8

adding file sync =0 to decoder just gave Execution ended after 0:00:06.566150473
as the time. which is greater than decoder + encoder.

What is sync=0 mean? Still this dosnt provide me the solution we require since the time is greater.

9

You may read:
https://gstreamer.freedesktop.org/documentation/base/gstbasesink.html?gi-language=c

Also note that my decoding-encoding pipeline had not -e flag as your decoding pipeline has.

Not sure what is your intention here. Sorry if it doesn’t work with your understanding.
You may better understand when using gst-shark tracers.
For reference, here using gst-shark v0.6.0 with R35.1.0:

git clone https://github.com/RidgeRun/gst-shark
cd gst-shark
git checkout release/v0.6.0
./autogen.sh --prefix /usr/ --libdir /usr/lib/aarch64-linux-gnu/
sudo make -j$(nproc) install
rm ~/.cache/gstreamer-1.0/registry.aarch64.bin
GST_DEBUG=*:4 gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstsharktracers.so

# If ok so far, try the example:
GST_DEBUG="GST_TRACER:7" GST_TRACERS="interlatency" gst-launch-1.0 videotestsrc ! queue ! videorate max-rate=15 ! fakesink sync=true

and further explore your case.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.