How to decrease latency on Xavier NX H264 encoder?

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
1

Dear all,

I have an Xavier NX developper kit. I grab the image from an USB camera and I do some images enhancement on cuda. I would like to encode and stream in H264 the video flow.

here you can find my code :

cv:: String GStream = "appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! omxh264enc ! video/x-h264, stream-format=byte-stream ! h264parse ! rtph264pay pt=96 config-interval=1 ! udpsink host=192.168.1.1 port=5200 ";
cv::VideoWriter oStream;
oStream.open(GStream, cv::CAP_GSTREAMER, 0, 25, oResultImg.size(), true);
…My image enhancement code …
if(oStream.isOpened())
oStream << oResultImg;

Here, oResultImg.size() = 1224x1024.

Then I visualize the result in VLC on my windows computer with this sdp file :

m=video 5200 RTP/AVP 96
a=rtpmap:96 H264
a=framerate:25
c=IN IP4 127.0.0.1

My Xavier NX and my computer are linked via ethernet.
The problems are :

1. How can I change my pipeline in order the optimize the stream to reach a low latency streaming ? (MAX 100ms) I already have 800ms.

2. How can I visualize the stream without VLC (windows PC)? because I think VLC add latency.

Thank you very much for your answers. Fell free to ask me more details if I’m not clear about some points.

3

Hi,
We suggest use nvv4l2h264enc plugin and enable the property:

  maxperf-enable      : Enable or Disable Max Performance mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false

And received frames before the first IDR frame is dropped. You can set IDR/I frame interval to a smaller value.

In general we use gstreamer pipeline on Linux OS. For Windows OS, may need other users to share suggestion.

4

Thank you for you answer. I changed my pipeline by

"appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! nvv4l2h264enc marperf-enable=1 ! video/x-h264, stream-format=byte-stream ! h264parse ! rtph264pay pt=96 config-interval=1 ! udpsink ";

But the delay is still around 800ms. For information I already boost my Xavier NX performance with this command line :

sudo jetson_clocks

Before visualize the stream via windows I think I’m going to optimize it on linux. So I stream on the jetson xavier in local with this pipeline :

gst-launch-1.0 -e udpsrc port=5200 ! application/x-rtp, encoding-name=H264, payload=96 ! rtpjitterbuffer ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! xvimagesink

1. Are my decoding and encoding pipeline good ?
2. How can I set the IDR/I frame interval to a smaller value, as you said ?
3. I’m still interested in other user suggestion for the windows streaming visualize

5

Hi,
On Xavier, please execute sudo nvpmodel -m 0, sudo jetson_clocks and try this command:

gst-launch-1.0 -e udpsrc port=5200 ! application/x-rtp, encoding-name=H264, payload=96 ! rtpjitterbuffer ! rtph264depay ! h264parse ! nvv4l2decoder enable-max-performance=1 ! nvoverlaysink sync=0

For setting IDR/I frame interval, you would need to set the properties:

  iframeinterval      : Encoding Intra Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 30
  idrinterval         : Encoding IDR Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 256

idrinterval is added in r32.5. If you use previous r32.4.3 or r32.4.4, you may refer to the source code of r32.5, add the property, and rebuild/replace the lib. The source code is in
https://developer.nvidia.com/embedded/L4T/r32_Release_v5.0/sources/T186/public_sources.tbz2
https://developer.nvidia.com/embedded/L4T/r32_Release_v4.4/r32_Release_v4.4-GMC3/Sources/T186/public_sources.tbz2

6

Hi,

Thank you for your help. I succeed to reach low latency ~150ms glass to glass test.
For other users, I also succeed to stream on windows. I run this pipeline in the folder where gst-launch-1.0.exe is :

.gst-launch-1.0.exe -v udpsrc port=5200 caps=“application/x-rtp, payload=96” ! rtpjitterbuffer ! rtph264depay ! video/x-h264,stream-format=byte-stream,alignment=nal ! h264parse ! avdec_h264 ! fpsdisplaysink sync=false

2 Likes