Question about encoding a USB camera stream to H.264 and decode it directly gstreamer

Question about encoding a USB camera stream to H.264 and decode it directly by gstreamer

We are trying to stream camera over RTSP by Jetson Xavier NX and we hope to reduce the latency.
Currently, around 150ms latency is considered caused by encode and decode process. Therefore we want to test the performance of the hardware encoder and decoder on Jetson Xavier NX.

For now, we have tested the hardware encoder by following command. And framerate of the output video is 20, which means at least 50ms is introduced during encode a frame averagely.
<1st command> gst-launch-1.0 -e v4l2src device=/dev/video4 num-buffers=200 ! ‘image/jpeg,width=1920,height=1080,framerate=25/1’ ! jpegdec ! nvvidconv ! nvv4l2h264enc ! h264parse ! queue2 ! qtmux ! filesink location=FHD_h264HW.mp4 -e

Then we tested the hardware decoder by following command. And we notice the fps is around 240, which means 4~5ms is reqired to decode a frame averagely.
<2nd command> gst-launch-1.0 filesrc location= jellyfish-5-mbps-1080p-h264.mkv ! matroskademux ! h264parse ! nvv4l2decoder enable-max-performance=1 ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

Next, we also want to try to encode a USB camera stream to H.264 and decode it directly by following command to evaluate the latency. But the process is stucked and no image can show up on the screen.
We also checked the status of encoder as while as decoder by jtop tools, and both hardware worked shortly and stoped.
<3rd command> gst-launch-1.0 -e v4l2src device=/dev/video4 ! ‘image/jpeg,width=1920,height=1080,framerate=25/1’ ! jpegdec ! nvvidconv ! nvv4l2h264enc maxperf-enable=1 ! h264parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! xvimagesink sync=0

Setting pipeline to PAUSED …
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Redistribute latency…
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
H264: Profile = 66, Level = 0
NVMEDIA: Need to set EMC bandwidth : 705000
NVMEDIA: Need to set EMC bandwidth : 705000
NVMEDIA_ENC: bBlitMode is set to TRUE

<Stucked here. And after press ctrl + C for the first time>

handling interrupt.
Interrupt: Stopping pipeline …
EOS on shutdown enabled – Forcing EOS on the pipeline

<Stucked here. after press ctrl + C for the second time, the process is terminated finally.>

So, my first question is how can I modify the 3rd command to encode then decode the video stream by gstreamer in the same pipeline.
The second question is what is the best method to measure the latency of H264 encoder and decoder.
The third question is how to reduce the latency introduced by decode and encode processes.

Hi,

This seems like a new issue introduced recently to the decoder, cause I’m sure this pipeline worked previously:

gst-launch-1.0 \
videotestsrc is-live=true ! nvvidconv ! nvv4l2h264enc ! nvv4l2decoder ! fakesink sync=false

I found a workaround streaming between two terminals on the same board with udp:

gst-launch-1.0 \
videotestsrc is-live=true ! nvvidconv ! \
nvv4l2h264enc bitrate=65507 insert-sps-pps=true maxperf-enable=true control-rate=1 ! \
h264parse ! \
udpsink port=5000 host=127.0.0.1 sync=false

gst-launch-1.0 \
udpsrc uri="udp://127.0.0.1:5000" ! \
h264parse ! \
nvv4l2decoder ! \
fakesink sync=false

For your second question, we use GstShark to measure interlatency. That will give you the most accurate measurement for just the encoder and decoder latency.

The parameters that most impact encoding latency are the additional values added to the H.264 header with properties like iframeinterval and idrinterval However, increasing these intervals will deteriorate image quality. It’s a trade-off between latency and quality, and you need to experiment with your specific use case to find the optimal values

Thanks for your reply Miguel!

You mentioned that new issue introduced to decoder. You mean the issue introduced to software related to decoder or the hardware decoder itself? Should I upgrade Jetpack or some package to solve the problem?

I have tried following command line in both Xavier NX and AGX Orin, and both platforms cannot get expected output.

gst-launch-1.0 \
videotestsrc is-live=true ! nvvidconv ! nvv4l2h264enc ! nvv4l2decoder ! fakesink sync=false

# OR add h264parse 
gst-launch-1.0 \
videotestsrc is-live=true ! nvvidconv ! nvv4l2h264enc ! h264parse ! nvv4l2decoder ! fakesink sync=false

We used to try the local UDP loop back method as you mentioned, and got a latency more than 150ms for H264 ecoding and decoding a 1080p video. So we want to check the latency induced by encoder and decoder without UDP loop back.

According to you suggestion, we tried GstShark, and it is very easy to use. We have got some results as following.

  1. 36ms for h264 encoding 30fps 1080p video:
GST_DEBUG="GST_TRACER:7" GST_TRACERS="interlatency" \
gst-launch-1.0 videotestsrc is-live=true do-timestamp=true num-buffers=50 ! \
'video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, framerate=(fraction)30/1' ! \
nvvidconv ! queue ! nvv4l2h264enc ! fakesink sync=true
  1. 15ms for h264 encoding 30fps 720p video:
GST_DEBUG="GST_TRACER:7" GST_TRACERS="interlatency" \
gst-launch-1.0 videotestsrc is-live=true do-timestamp=true num-buffers=50 ! \
'video/x-raw, format=(string)YUY2, width=(int)1080, height=(int)720, framerate=(fraction)30/1' ! \
nvvidconv ! queue ! nvv4l2h264enc ! fakesink sync=true
  1. 170ms for h264 decoding 30fps 1080p video:
# for the rtsp server:
gst-launch-1.0 videotestsrc is-live=1 ! \
video/x-raw,width=1920,height=1080,framerate=60/1 ! \
timeoverlay valignment=top halignment=right shaded-background=true font-desc="Sans, 72" ! \
tee name=t ! queue ! nvvidconv ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! \
h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5000 sync=0

# for the rtsp client:
GST_DEBUG="GST_TRACER:7" GST_TRACERS="interlatency" \
gst-launch-1.0 udpsrc port=5000 ! \
application/x-rtp,media=video,encoding-name=H264,payload=96 ! \
rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink sync=0 

The h264 decoding latency detected by GstShark is to big, since we hope to reduce the glass to glass latency to around 150ms. And we also noticed some webpages report very low encoding and decoding latency, for example:
Nvv4l2h264enc latency and preset-level. GStreamer Encoding Latency in NVIDIA Jetson Platforms[GStreamer Encoding in Jetson | GStreamer Encoding Latency | RidgeRun].

So what should I do to optimize the latency for stream video by h264.

Thanks very much!

You mentioned that new issue introduced to decoder. You mean the issue introduced to software related to decoder or the hardware decoder itself? Should I upgrade Jetpack or some package to solve the problem?

The pipeline used to work on JetPack 4.6. I’m not sure when it stopped working. I know this because I used it to debug another issue for a project using that JetPack version. The new issue seems related to the GStreamer element and not the hardware. The element isn’t pushing buffers and it doesn’t report any errors.

So what should I do to optimize the latency for stream video by h264.

The decoder latency does seem to be higher than expected, but I don’t think there is a significant way to optimize it besides setting low-latency-mode=true and skipping frames. Also, adding a leaky queue before the decoder can help with latency by dropping the oldest buffers: queue leaky=2 max-size-buffers=1

The encoding latency seems okay, but keep in mind that it can increase if you switch the videotestsrc for an actual video stream, because videotestsrc is really easy to encode since there is not a lot of spatial and temporal information in the stream.

Thanks Miguel.

Your information is very useful for me. I will give it a try.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.