When I get video stream on localhost with udpsrc, the incoming video stream is lagged and mosaic

o.erdemir1997 · June 29, 2022, 6:26am

I have tried to get video stream on localhost with udpsrc.
When I get video stream on localhost with udpsrc, the incoming video stream is lagged and mosaic. I have change my h265 encoder and decoder than I have tried again. But nothing has changed

This one my server pipe:

gst-launch-1.0 v4l2src device=/dev/video8 io-mode=mmap ! video/x-raw width=1920, height=1080, framerate=25/1 ! nvvidconv ! omxh265enc bitrate=5000000 ! mpegtsmux ! udpsink host=10.0.3.30 port=11002

this one my reciever pipe:

gst-launch-1.0 udpsrc port=11002 ! video/mpegts, width=1920, height=1080, framerate=25/1 ! tsdemux ! h265parse, omxh265dec! nvvidconv ! omxh265enc bitrate=5000000 ! mpegtsmux name=mux ! filesink location=1.avi

DaneLLL · June 29, 2022, 7:52am

Hi,
Please try nvv4l2h265enc and nvv4l2decoder. May refer to this example:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL

Honey_Patouceul · June 29, 2022, 11:25am

Another thing to try if you want to keep UDP would be using MP2T-RTP:

Sender:

gst-launch-1.0 v4l2src device=/dev/video8 io-mode=mmap ! video/x-raw, width=1920, height=1080, framerate=25/1 ! nvvidconv ! nvv4l2h265enc insert-sps-pps=1 idrinterval=25 insert-vui=1 ! mpegtsmux ! rtpmp2tpay ! udpsink  host=10.0.3.30 port=11002 auto-multicast=0

Receiver:

# Display:
gst-launch-1.0 udpsrc port=11002 ! application/x-rtp,encoding-name=MP2T,payload=33,clock-rate=90000 ! rtpjitterbuffer latency=300 ! rtpmp2tdepay ! tsdemux ! decodebin ! autovideosink

It is unclear why your receiver decodes and re-encodes into same properties.
Also note that currently you save with avi file extension, but you would save in ts format.
You would use avimux for creating an avi file, but it doesn’t support H265 (H264 is supported). For H265, you would use mastroskamux for MKV as container.

gst-launch-1.0 udpsrc port=11002 ! application/x-rtp,encoding-name=MP2T,payload=33,clock-rate=90000 ! rtpjitterbuffer latency=300 ! rtpmp2tdepay ! tsdemux ! h265parse ! matroskamux ! filesink location=test_h265.mkv

o.erdemir1997 · July 2, 2022, 8:20am

I couldn’t find how to implement this “rtpmp2tpay” function(elements).
I’m developing in C language and I couldn’t find how it was written in the documents, in which procedure.
Can you help me how to add it to the code? Information on this element on the Internet is very limited.

Honey_Patouceul · July 2, 2022, 9:50am

The simplest would be to use gstreamer C API such as:

#include <gst/gst.h>

int main (gint argc, gchar * argv[])
{  
	gst_init (&argc, &argv);
	GMainLoop *loop = g_main_loop_new (NULL, FALSE);

	const gchar *pipeline_str = "v4l2src device=/dev/video8 io-mode=mmap ! video/x-raw, width=1920, height=1080, framerate=25/1 ! nvvidconv ! nvv4l2h265enc insert-sps-pps=1 idrinterval=25 insert-vui=1 ! mpegtsmux ! rtpmp2tpay ! udpsink  host=10.0.3.30 port=11002 auto-multicast=0";
	GstElement *pipeline = gst_parse_launch (pipeline_str, NULL);
	if (!pipeline) {
		g_error ("Failed to create pipeline\n");
		exit(-1);
	}

	/* Ok, successfully created the pipeline, now start it */
	gst_element_set_state (pipeline, GST_STATE_READY);
	gst_element_set_state (pipeline, GST_STATE_PLAYING);

	/* wait until it's up and running or failed */
	if (gst_element_get_state (pipeline, NULL, NULL, -1) == GST_STATE_CHANGE_FAILURE) {
		g_error ("Failed to go into PLAYING state");
		exit(-2);
	}

	g_print ("Running ...\n");
	g_main_loop_run (loop);

	return 0;
}

and build with:

gcc -Wall -o gst_testlaunch_rtpmp2t -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include gst_testlaunch_rtpmp2t.cpp -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0

If you want to implement by yourself, gstreamer is open source and you can find code from github (repo gst-plugins-good).

o.erdemir1997 · July 2, 2022, 10:47am

Not this way, I’m building a deep structure, I define the elements one by one, in short I dont use gst parse launch, is rtpmp2tpay gst_element_factory defined as make. I’m not sure if we define the Src and Sink values as gst_caps_new_simple. can i get this information… G_OBJECT_SET What object are the values attached to payload ,pay, src …etc ?

Honey_Patouceul · July 2, 2022, 1:59pm

You may try something like (I don’t know your camera format, used YUY2 here):

#include <stdbool.h>
#include <gst/gst.h>

static GMainLoop *loop;

gint
main (gint   argc,
      gchar *argv[])
{
  GstElement *pipeline, *vidsrc, *conv, *enc, *mux, *pay, *udpsink;
  GstCaps *vidsrc2conv_caps, *conv2enc_caps, *enc2mux_caps, *mux2pay_caps, *pay2udp_caps;

  /* init GStreamer */
  gst_init (&argc, &argv);
  loop = g_main_loop_new (NULL, FALSE);

  /* setup pipeline */
  pipeline = gst_pipeline_new ("pipeline");

  vidsrc = gst_element_factory_make ("v4l2src", "vidsrc");
  g_object_set (G_OBJECT (vidsrc), "device", "/dev/video8", NULL);
  g_object_set (G_OBJECT (vidsrc), "io-mode", 2, NULL);

  vidsrc2conv_caps = gst_caps_from_string("video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)25/1, format=(string)YUY2");

  conv = gst_element_factory_make ("nvvidconv", "conv");

  conv2enc_caps = gst_caps_from_string("video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12");

  enc = gst_element_factory_make  ("nvv4l2h264enc", "enc");
  g_object_set (G_OBJECT (enc), "insert-sps-pps", true, NULL);
  g_object_set (G_OBJECT (enc), "insert-vui", true, NULL);
  g_object_set (G_OBJECT (enc), "idrinterval", 25, NULL);

  enc2mux_caps = gst_caps_from_string("video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)25/1");

  mux = gst_element_factory_make ("mpegtsmux", "mux");

  mux2pay_caps = gst_caps_from_string("video/mpegts, systemstream=(boolean)true, packetsize=(int)188");

  pay = gst_element_factory_make ("rtpmp2tpay", "pay");

  pay2udp_caps = gst_caps_from_string("application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)MP2T, payload=(int)33");

  udpsink = gst_element_factory_make ("udpsink", "udpsink");
  g_object_set (G_OBJECT (udpsink), "host", "10.0.3.30", NULL);
  g_object_set (G_OBJECT (udpsink), "port", 11002, NULL);
  g_object_set (G_OBJECT (udpsink), "auto-multicast", false, NULL);

  gst_bin_add_many (GST_BIN (pipeline), vidsrc, conv, enc, mux, pay, udpsink, NULL);

  if (!gst_element_link_filtered(vidsrc, conv, vidsrc2conv_caps)) {
        g_printerr("Fail to gst_element_link_filtered vidsrc -> conv\n");
        return -1;
  }

  if (!gst_element_link_filtered(conv, enc, conv2enc_caps)) {
        g_printerr("Fail to gst_element_link_filtered conv -> enc\n");
        return -1;
  }

  if (!gst_element_link_filtered(enc, mux, enc2mux_caps)) {
        g_printerr("Fail to gst_element_link_filtered enc -> mux\n");
        return -1;
  }

  if (!gst_element_link_filtered(mux, pay, mux2pay_caps)) {
        g_printerr("Fail to gst_element_link_filtered mux -> pay\n");
        return -1;
  }

  if (!gst_element_link_filtered(pay, udpsink, pay2udp_caps)) {
        g_printerr("Fail to gst_element_link_filtered pay -> udpsink\n");
        return -1;
  }
  /* Ok, pipeline successfully created */

  /* This will output details of pipeline going to play */
  g_signal_connect(pipeline, "deep-notify", G_CALLBACK(gst_object_default_deep_notify), NULL);

  /* Set pipeline playing */
  gst_element_set_state (pipeline, GST_STATE_PLAYING);
  g_main_loop_run (loop);

  /* Clean up */
  gst_element_set_state (pipeline, GST_STATE_NULL);
  gst_object_unref (GST_OBJECT (pipeline));
  g_main_loop_unref (loop);

  return 0;
}

o.erdemir1997 · July 4, 2022, 11:32am

Ty for helping sir…
My Code little different.
I used gst_caps_new_simple, Is it possible for me to use this function (gst_caps_new_simple) multiple ?
Pipeline = gst_element_link_many = src, conv, enc,muxer, pay,overlay, tee,
queue_udp, udpsink,
queue_file, filesink,
. ERROR Gstreamer-CRITICAL gst_element_link_pads_full: assertion GST_IN_ELEMENT (dest) failed

o.erdemir1997 · July 4, 2022, 12:17pm

In short, can I use gst_caps_new_simple 2 times at the same time.
example = main_caps = gst_caps_new_simple (“video/x-raw-yuv”,
“format”, GST_TYPE_FOURCC, fourcc,
“framerate”, GST_TYPE_FRACTION,
dec->info.fps_numerator, dec->info.fps_denominator,
“pixel-aspect-ratio”, GST_TYPE_FRACTION, par_num, par_den,
“width”, G_TYPE_INT, dec->width, “height”, G_TYPE_INT, dec->height,
“color-matrix”, G_TYPE_STRING, “sdtv”,
“chroma-site”, G_TYPE_STRING, “jpeg”, NULL);
g_object_set(G_OBJECT(src), “caps”, main_caps, NULL);
gst_caps_unref (main_caps);

second_caps = gst_caps_new_simple (“video/mpegts”,
“packetsize”,G_TYPE_INT, 188,
“systemstream”, G_TYPE_BOOLEAN, TRUE,
NULL);
g_object_set(G_OBJECT(pay), “caps”, second_caps, NULL);
gst_caps_unref (second_caps);
…
Is such a definition possible?

Honey_Patouceul · July 4, 2022, 8:40pm

There are plugins that have properties. You can use g_object_set for setting properties of a previously created plugin.

There are caps, that define the format of data between 2 plugins. These caps must match both producer plugin SRC pad and consumer SINK pad.

It may get confusing when some plugins such as udpsink have a caps property.
What you are trying can only work with these plugins.
rtpmp2tpay doesn’t have such a caps property (nor have nvvidconv, nvv4l2h264enc,…).
You would check a plugin’s properties and SRC and SINK caps with:

gst-inspect-1.0 the_plugin_you_want_to_inspect

Furthermore this wouldn’t ensure that caps are matching next plugin input until you’ve linked to it.

You may better consider adapting your code than expecting someone to adapt your code ;-)
The example above should be enough for a working solution to topic’s question. If you still experience mosaicing because an intra frame was lost, then move to TCP transport.
You can also use a debugger for checking caps returned by the example if you really want using gst_caps_new_simplefor building caps, but it may not be easy (especially for NVMM memory attribute).

o.erdemir1997 · July 5, 2022, 6:05am

I understand, just know gst_caps_new_simple used 2 more …
rtpmp2tpay What can we use instead, I didn’t ask for code, I just wanted to learn the correct usage, you wrote it as code.
Ty so much.

o.erdemir1997 · July 5, 2022, 11:05am

The structure I built is very deep, my code is already written, but I could not solve the problem caused by rtpmp2pay and I needed information about multiple caps, many explanations are available in their own document , i just wanted help about the information limited ones .

Honey_Patouceul · July 5, 2022, 8:57pm

I fail to understand what you’re asking for.
If it is just using gst_caps_new_simple here are the caps before and after rtpmp2tpay:

...

  //mux2pay_caps = gst_caps_from_string("video/mpegts, systemstream=(boolean)true, packetsize=(int)188");
  mux2pay_caps = gst_caps_new_simple("video/mpegts", 
	"systemstream", G_TYPE_BOOLEAN, true,
	"packetsize", G_TYPE_INT, 188,
	NULL);

  pay = gst_element_factory_make ("rtpmp2tpay", "pay");

  //pay2udp_caps = gst_caps_from_string("application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)MP2T, payload=(int)33");
  pay2udp_caps = gst_caps_new_simple("application/x-rtp",
	 "media", G_TYPE_STRING, "video",
	 "encoding-name", G_TYPE_STRING, "MP2T",
	 "clock-rate", G_TYPE_INT, 90000,
	 "payload", G_TYPE_INT, 33,
	 NULL);

  udpsink = gst_element_factory_make ("udpsink", "udpsink");
  g_object_set (G_OBJECT (udpsink), "host", "10.0.3.30", NULL);
  g_object_set (G_OBJECT (udpsink), "port", 11002, NULL);
  g_object_set (G_OBJECT (udpsink), "auto-multicast", false, NULL);

...

For summarizing up on this topic, I think that:

You’re seeing mosaicing because an I frame was lost in UDP traffic (Note that UDP is just datagrams with no control-flow).
There are many possible causes for hogging your UDP stack or network.
So streaming MP2TS directly on UDP may not be reliable.
A first step would be using RTP-MP2T with rtpmp2tpay on sender and rtpmp2tdepay on receiver. This may help.
If not enough, or if loosing an I frame is not affordable, you would have to move to TCP transport.

If not enough, you may better explain your expectation into another topic, but as it may be a gstreamer more than a Jetson-related issue, you may better post it into gstreamer devel forum.

jinpri61 · July 7, 2022, 12:32pm

Hi again. The above pipeline works just great on command line but when we try to use the same receive pipeline in python script, the saved video flows real fast. It does not match the original 25 fps. Also the timestamp does not match. Any support would be really appreciated.

Honey_Patouceul · July 7, 2022, 8:01pm

Please help to keep this topic clear with an answer to topic’s title as it is what next users will find when searching.

Please help creating another topic for your issue which is different from this one.

system · August 3, 2022, 1:39am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Receive RTP (Pipeline demo) DeepStream SDK gstreamer	23	1620	August 14, 2023
UDP video from ffmpeg to gstreamer Jetson AGX Orin gstreamer , ffmpeg	8	1497	February 13, 2024
Segfault when setting host and port on gstreamer's udpsink Jetson Nano gstreamer	14	2547	September 19, 2021
deepstream4.0 with rtsp of sink makes the picture distortion with high bitrate DeepStream SDK	22	2428	October 12, 2021
How to analyze video by source UDP? DeepStream SDK	4	1003	October 12, 2021
Delay in video Jetson Nano gstreamer	5	2699	October 18, 2021
Deepstream 3.0 C API Jetson AGX Xavier	10	2502	May 30, 2019
Gstreamer TCPserversink 2-3 seconds latency Jetson Xavier NX camera , opencv , gstreamer	12	18157	October 3, 2021
RTSP stream not work "Stream format not found, dropping the frame" DeepStream SDK rtsp , gstreamer , nvbugs	24	1658	January 3, 2024
Streammux can not stop Jetson TX2 gstreamer	4	706	October 18, 2021

When I get video stream on localhost with udpsrc, the incoming video stream is lagged and mosaic

Related topics