Gstreamer TCPserversink 2-3 seconds latency

Hello,
I’m trying to send a video stream with tcp, but I get 2-3 seconds of latency, and i’m looking to reduce it as much as possible. The received stream sometimes stop on a gray image and then receive a burst of frames in accelerate.
We have a Jetson NX Xavier devkit on Jetpack 4.5.1. I’m using it with nvpmodel -m 2 and jetson_clocks --fan.
I enabled VIC to run in max clock as described here

Command v4l2-ctl --list-formats-ext -d /dev/video0 give :
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘YUYV’
Name : YUYV 4:2:2
Size: Discrete 1920x1080
Interval: Discrete 0.017s (60.000 fps)

Here is my pipeline :
> std::string gst_out = "appsrc ! videoconvert ! queue ! nvvidconv ! nvv4l2h265enc maxperf-enable=1 ! h265parse ! matroskamux ! tcpserversink port=AAAA host=XXX.XXX.XXX.XXX sync=false async=false";

And my codec :
cv::VideoWriter out(gst_out, cv::VideoWriter::fourcc('F', 'M', 'P', '4'), double(fps), cv::Size(int(w), int(h)));
I tried to remove videoconvert with video/x-raw, format=YUYV, width=1920, height=1080, framerate=60/1, but it doesn’t work with this setup.

Videostream check with fpsdisplaysink

gst-launch-1.0 v4l2src ! videoconvert ! fpsdisplaysink text-overlay=0 video-sink=fakesink sink=0 -v
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING …
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 61,95, average: 61,95
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 63, dropped: 0, current: 60,01, average: 60,98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 93, dropped: 0, current: 60,00, average: 60,66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 124, dropped: 0, current: 60,00, average: 60,49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 154, dropped: 0, current: 60,00, average: 60,40
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 185, dropped: 0, current: 60,00, average: 60,33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 216, dropped: 0, current: 60,01, average: 60,28
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 247, dropped: 0, current: 59,99, average: 60,25
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 278, dropped: 0, current: 60,00, average: 60,22
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 309, dropped: 0, current: 60,00, average: 60,20
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 339, dropped: 0, current: 60,00, average: 60,18
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 370, dropped: 0, current: 60,00, average: 60,16
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 401, dropped: 0, current: 60,00, average: 60,15
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:07.000306499
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

Thank you !

Hi,
Please adjust the two properties in nvv4l2h265enc and try:

  iframeinterval      : Encoding Intra Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 30
  idrinterval         : Encoding IDR Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 256

The default setting may not be good for streaming, may try idrinterval=30 or15.

Hi DaneLLL,
I tried multiples value for idrinterval (15, 30) and iframeinterval (5, 10, 20, 50, 100, 200, 300, 600), but nothing changed.
Here is my code :
gstreamer_tcp.cpp (2.7 KB)

Where can I see the documentation about all the Gstreamer nvidia properties like idrinterval ? I can’t find them in the ACCELERATED GSTREAMER USER GUIDE ?
Here is a sample of what I am getting :

Don’t you think it can also come from the videoconvert plug-in ?

Hi,
Are you able to try UDP streaming? We have tried UDP in TX2 NX as server and x86 PC as client.
Server command:

$ gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,width=1280,height=720 ! timeoverlay valignment=4 halignment=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! tee name=t ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! h264parse ! rtph264pay ! udpsink host=10.19.106.10 port=5000 sync=0 t. ! queue ! nvegltransform ! nveglglessink sync=0

Client command:

$ gst-launch-1.0 udpsrc port=5000 ! 'application/x-rtp,encoding-name=H264,payload=96' ! rtph264depay ! avdec_h264 ! xvimagesink sync=0

The result(left is TX2 NX and right is x86 PC):

Could you try server commands in the cases:

  1. gst-launch-1.0 videotestsrc in UDP
  2. gst-launch-1.0 videotestsrc in TCP
  3. gst-launch-1.0 OpenCV in UDP

See if we can clarify where the latency is from by trying these cases.

And please run gst-inspect-1.0 nvv4l2h265enc to get all properties.

Huge Latency Gstreamer
High Latency in Gstreamer UDP Stream
How to Optimize omxh264enc to reduce RTSP latency
How to view Jetson nano live video from Android OS device
Jetson nano , pi camera, streaming in H265
Gstreamer network pipeline task
Improve performance Gstreamer pipeline for webrtc in Jetson AGX
Very long delay before receiving a decoded video if nvv4l2h265enc is used
Very long delay before receiving a decoded video if nvv4l2h265enc is used
GStreamer Manually Encode H264 and RTSP Stream with Pipeline
NVENC latency
How to transcode H264 rtp packet to H265 rtp packet
Streaming B01 Dual Camera Image over Network to VLC
Change video settings on RTSP streaming
Problem decoding H.265 video with nvv4l2decoder
Poor quality H264 stream and no GPU useage
Video encoder trade-offs, omxh264enc vs nvv4l2h264enc (Xavier NX)
RTSP streaming from Jetson cant play smoothy
GStreamer camera to multiple sinks / pipelines
Jetson x265 -> rtpay -> udpsink discards SPS/PSS
Gstreamer mpegtsmux slows down pipe enormously
Xavier NX for 4 cameras V4L2 capture, converter, H265 encode and RTP over UDP transfer
Udpsink drops 30 packets
Change video settings on RTSP streaming
Gstreamer 720x576 frame 200ms transmitting problem
Cannot stream 1080p video using GStreamer and appsrc
Jetson Nano FAQ
GStreamer pipeline to convert MJPEG to RTSP server stream with adaption of framerate
Result of multifilesink is executed only when an RTSP client is connected
Is OpenCV VideoCapture able to get branched appsink from test-launch
Encoding CSI camera with x264enc codec
Segfault when setting host and port on gstreamer's udpsink
Streaming opencv frames using gstreamer and reading them on node
Udpsink drops 30 packets
When reconnect udp port I can not get video stream again from udp port

Hi DaneLLL,
I am not able to stream UDP with these commands, I get the same screen as you when i’m streaming from the NX Xavier, but my Ubuntu 18.04 x86 PC doesn’t get the videotest :

gst-launch-1.0 udpsrc port=5000 ! 'application/x-rtp,encoding-name=H264,payload=96' ! rtph264depay ! avdec_h264 ! xvimagesink sync=0
Définition du pipeline à PAUSED...
Le pipeline est actif et n’a pas besoin de phase PREROLL…
Passage du pipeline à la phase PLAYING…
New clock: GstSystemClock
^Chandling interrupt.
Interruption : arrêt du pipeline…
Execution ended after 0:00:17.859331593
Définition du pipeline à PAUSED...
Définition du pipeline à READY (prêt)…
Définition du pipeline à NULL…
Libération du pipeline…

And no image is shown.

Also gst-launch-1.0 nvv4l2h265enc give me this on my NX Xavier :

$ gst-launch-1.0 nvv4l2h265enc
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:07.792176179
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Hi,
The command was not correct and I have fixed it. It should be gst-inspect-1.0.

For streaming to your host PC, do you change host=10.19.106.10 to IP address of the host PC?

Hi DaneLLL, thanks for fast reply,

It should be gst-inspect-1.0 .

It works, thanks.

  1. gst-launch-1.0 videotestsrc in UDP
    It works great, I have like 20ms latency.

  2. gst-launch-1.0 videotestsrc in TCP
    Please, correct me if i’m wrong : I have to specify the IP address of my NX, do I use tcpclientsrc or tcpserversrc :

gst-launch-1.0 tcpserversrc host=NX.Address.IP port=5000 ! 'application/x-rtp,encoding-name=H264,payload=96' ! rtph264depay ! avdec_h264 ! xvimagesink sync=0

I don’t receive image with this.
Transmitter pipeline (NX) :

gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,width=1280,height=720 ! timeoverlay valignment=4 halignment=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! tee name=t ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! h264parse ! rtph264pay ! tcpserversink host=NX.Address.IP port=5000 sync=0 t. ! queue ! nvegltransform ! nveglglessink sync=0
  1. gst-launch-1.0 OpenCV in UDP
std::string gst_out = "appsrc ! video/x-raw,width=1920,height=1080 ! videoconvert ! timeoverlay valignment=4 halignment=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1920,height=1080' ! tee name=t ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! h264parse ! rtph264pay ! udpsink host=PC.Address.IP port=5000 sync=0 t. ! queue ! nvegltransform ! nveglglessink sync=0

It works with 0-20ms latency (calculated with a few photo) notice the videoconvert, otherwise it wouldn’t push frame to the pipeline.
I also get some frame all blurred, it doesn’t happen too often thought, it can come from so much things, do you have any idea about this ?
So, it appears that the latency came from tcpserversink.

Thank you so much DaneLLL !

Now that the transmission is working really well, (I also changed the encoder from 264 to 265 and the quality is much higher). I can’t receive my stream with VLC using udp://@NX.Address.IP:PORT .
It seems that I need to define the packet size, but I can’t find which parameters can do it. Can you tell me if this is the correct way to read the stream with VLC, or is there a simpler way ?

Update : I managed to stream my video with UDP with Gstreamer on an application to VLC with a .sdp file, but I now have 500ms of latency, i use the same pipeline as before :

std::string gst_out = "appsrc ! video/x-raw,width=1920,height=1080 ! videoconvert ! timeoverlay valignment=4 halignment=1 ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080 ! tee name=t ! nvv4l2h265enc insert-sps-pps=1 idrinterval=15 ! h265parse ! rtph265pay ! udpsink host=PC.Address.IP port=5000 sync=0 t. ! queue ! nvegltransform ! nveglglessink sync=0

and my .sdp file :

v=0
m=video #PORT RTP/AVP 96
c=IN IP4 192.168.1.21
a=rtpmap:96 H265/90000

Can I reduce this latency ?

Hi,
Not sure but it seems like there is buffering mechanism in VLC. This would need other users to share experience.

It may help to run in CBR + setting virtual buffer size. It avoid burst bitrate between I and P frames and may offer better stability in limited network bandwidth. Please check the example of setting it:
Random blockiness in the picture RTSP server-client -Jetson TX2 - #5 by DaneLLL

Hi DaneLLL,
Thanks for advices,

it seems like there is buffering mechanism in VLC

Indeed, it looks like it, I could see some blurred frame regularly when I put previous pipeline into VLC. It gets corrected by adjusting the bitrate and setting the vbv-size as you told.
Now that I can use UDP, I now have 1-2 seconds latency with this pipeline :

std::string gst_out = "appsrc ! video/x-raw,width=1920,height=1080 ! videoconvert ! timeoverlay valignment=4 halignment=1 ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080 ! tee name=t ! nvv4l2h265enc EnableTwopassCBR=1 insert-sps-pps=1 idrinterval=2 bitrate=64000000 vbv-size=1600000 maxperf-enable=1 ! h265parse ! rtph265pay config-interval=1 pt=96 ! udpsink host=PC.Address.IP port=PORT sync=0 async=0  t. ! nvegltransform ! nveglglessink max-lateness=11000 sync=0";

Tried to remove these parameters : max-lateness=11000, EnableTwopassCBR=1, insert-sps-pps=1, idrinterval=2, it doesn’t seems to affect anything to the stream.

I didn’t use VLC in my solution, we receive the stream with the Windows Gstreamer library and I have something like 30ms latency with udpsink.
Here is a pipeline that works for me if someone wants to send it to VLC, but I couldn’t get better than 1s latency :

std::string gst_out = "appsrc is-live=1 ! video/x-raw, width=1920, height=1080 ! videoconvert ! timeoverlay valignment=4 halignment=1 ! nvvidconv ! video/x-raw(memory:NVMM), width=1920, height=1080 ! tee name=t ! nvv4l2h265enc EnableTwopassCBR=1 insert-sps-pps=1 idrinterval=15 iframeinterval=1000 bitrate=64000000 vbv-size=1600000 maxperf-enable=1 preset-level=1 ! h265parse ! rtph265pay config-interval=1 pt=96 ! udpsink host=IP.address port=PORTsync=0 async=0 t. ! nvegltransform ! nveglglessink max-lateness=11000 sync=0";

The receiver needs to open a .sdp file to read that stream. Adjusting the bitrate helps suppressing blurred frame.

Thank you DaneLLL for the help.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.