How to decrease latency on Xavier NX H264 encoder?

Dear all,

I have an Xavier NX developper kit. I grab the image from an USB camera and I do some images enhancement on cuda. I would like to encode and stream in H264 the video flow.

here you can find my code :

cv:: String GStream = "appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! omxh264enc ! video/x-h264, stream-format=byte-stream ! h264parse ! rtph264pay pt=96 config-interval=1 ! udpsink host=192.168.1.1 port=5200 ";
cv::VideoWriter oStream;
oStream.open(GStream, cv::CAP_GSTREAMER, 0, 25, oResultImg.size(), true);
…My image enhancement code …
if(oStream.isOpened())
oStream << oResultImg;

Here, oResultImg.size() = 1224x1024.

Then I visualize the result in VLC on my windows computer with this sdp file :

m=video 5200 RTP/AVP 96
a=rtpmap:96 H264
a=framerate:25
c=IN IP4 127.0.0.1

My Xavier NX and my computer are linked via ethernet.
The problems are :

1. How can I change my pipeline in order the optimize the stream to reach a low latency streaming ? (MAX 100ms) I already have 800ms.

2. How can I visualize the stream without VLC (windows PC)? because I think VLC add latency.

Thank you very much for your answers. Fell free to ask me more details if I’m not clear about some points.

Hi,
We suggest use nvv4l2h264enc plugin and enable the property:

  maxperf-enable      : Enable or Disable Max Performance mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false

And received frames before the first IDR frame is dropped. You can set IDR/I frame interval to a smaller value.

In general we use gstreamer pipeline on Linux OS. For Windows OS, may need other users to share suggestion.

Thank you for you answer. I changed my pipeline by

"appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! nvv4l2h264enc marperf-enable=1 ! video/x-h264, stream-format=byte-stream ! h264parse ! rtph264pay pt=96 config-interval=1 ! udpsink ";

But the delay is still around 800ms. For information I already boost my Xavier NX performance with this command line :

sudo jetson_clocks

Before visualize the stream via windows I think I’m going to optimize it on linux. So I stream on the jetson xavier in local with this pipeline :

gst-launch-1.0 -e udpsrc port=5200 ! application/x-rtp, encoding-name=H264, payload=96 ! rtpjitterbuffer ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! xvimagesink

1. Are my decoding and encoding pipeline good ?
2. How can I set the IDR/I frame interval to a smaller value, as you said ?
3. I’m still interested in other user suggestion for the windows streaming visualize

Hi,
On Xavier, please execute sudo nvpmodel -m 0, sudo jetson_clocks and try this command:

gst-launch-1.0 -e udpsrc port=5200 ! application/x-rtp, encoding-name=H264, payload=96 ! rtpjitterbuffer ! rtph264depay ! h264parse ! nvv4l2decoder enable-max-performance=1 ! nvoverlaysink sync=0

For setting IDR/I frame interval, you would need to set the properties:

  iframeinterval      : Encoding Intra Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 30
  idrinterval         : Encoding IDR Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 256

idrinterval is added in r32.5. If you use previous r32.4.3 or r32.4.4, you may refer to the source code of r32.5, add the property, and rebuild/replace the lib. The source code is in
https://developer.nvidia.com/embedded/L4T/r32_Release_v5.0/sources/T186/public_sources.tbz2
https://developer.nvidia.com/embedded/L4T/r32_Release_v4.4/r32_Release_v4.4-GMC3/Sources/T186/public_sources.tbz2

Hi,

Thank you for your help. I succeed to reach low latency ~150ms glass to glass test.
For other users, I also succeed to stream on windows. I run this pipeline in the folder where gst-launch-1.0.exe is :

.gst-launch-1.0.exe -v udpsrc port=5200 caps=“application/x-rtp, payload=96” ! rtpjitterbuffer ! rtph264depay ! video/x-h264,stream-format=byte-stream,alignment=nal ! h264parse ! avdec_h264 ! fpsdisplaysink sync=false

1 Like