Jetson TX2 RTSP Streaming, FFmpeg or Gstreamer?

Hello,

I am using the Jetson TX2 dev board to work on real-time video processing.
Thanks to many other topics, I wrote this code :

#include "iostream"
#include "string"

#include "opencv2/opencv.hpp"
#include "opencv2/core.hpp"

int main()
{
// VideoCapture Pipe
std::string Cap_pipeline("nvarguscamerasrc ! "
"video/x-raw(memory:NVMM), width=1920, height=1080,format=NV12, framerate=30/1 ! "
“nvvidconv ! video/x-raw,format=I420 ! appsink”);

// VideoWriter Pipe
std::string Stream_Pipeline("appsrc is-live=true ! autovideoconvert ! "
"omxh264enc control-rate=2 bitrate=10000000 ! video/x-h264, "
"stream-format=byte-stream ! rtph264pay mtu=1400 ! "
“udpsink host=192.168.1.102 port=5000 sync=false async=false”);

cv::VideoCapture Cap(Cap_pipeline,cv::CAP_GSTREAMER);
cv::VideoWriter Stream(Stream_Pipeline, cv::CAP_GSTREAMER,
30, cv::Size(1920, 1080), true);

// check for issues
if(!Cap.isOpened() || !Stream.isOpened()) {
std::cout << “I/O Pipeline issue” << std::endl;
}

while(true) {
cv::Mat frame;
Cap >> frame; //read last frame
if (frame.empty()) break;

  cv::Mat bgr;
  cv::cvtColor(frame, bgr, cv::COLOR_YUV2BGR_I420);

  //video processing

  Stream.write(bgr);// write the frame to the stream

  char c = (char)cv::waitKey(1);
  if( c == 27 ) break;

}

Cap.release();
Stream.release();

return 0;
}

With this code, I can recover, process and live-stream the video coming from the onboard CSI camera to a client PC.
To read the stream, I can use GStreamer with the following command :
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264 ! autovideosink sync=false async=false -e

or FFmpeg or VLC using an SDP file.

However, I would like to use an RTSP stream, so that a do not have to use an SDP file, which would more easy for the future.
I have two questions :

  • Is there a way to use a code similar to ./test- launch from gst-rtsp-server to stream the cv::Mat processed with openCV in RTSP?
  • Which library has better performance between FFmpeg and GStreamer for h264/h265 RTSP video streaming?

OpenCV version: 4.3.0 (instaled with JEP script : https://github.com/AastaNV/JEP/tree/master/script)
Jetpack version: 4.4 DeepStream

Thanks

hi,
The questions are more about OpenCV. Suggest go to OpenCV forum.

You may say you have a pipeline to run UDP streaming:

// VideoWriter Pipe
std::string Stream_Pipeline("appsrc is-live=true ! autovideoconvert ! "
"x264enc ! video/x-h264, "
"stream-format=byte-stream ! rtph264pay mtu=1400 ! "
“udpsink host=192.168.1.102 port=5000 sync=false async=false”);

And ask for suggestion of changing it to RTSP streaming. Once you have a working pipeline, you may replace x264enc with hardware encoder nvv4l2h264enc.

We are deprecating omx plugins and please use v4l2 plugins for video encoding/decoding.

1 Like

Hi @DaneLLL

Thank you for your help, I will ask this question on the OpenCV forum.

You may use the videowriter with a gstreamer pipeline producing the h264 stream and sending to shmsink, so that you can get it from shmsrc inside test-launch:

#include <iostream>

#include <opencv2/opencv.hpp>
#include <opencv2/videoio.hpp>

int main ()
{
  //setenv ("GST_DEBUG", "*:3", 0);

  /* Setup capture with gstreamer pipeline from onboard camera converting into BGR frames for app */
  const char *gst_cap =
    "nvarguscamerasrc  ! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)640, height=(int)480, framerate=(fraction)30/1 ! "
    "nvvidconv    ! video/x-raw,              format=(string)BGRx ! "
    "videoconvert ! video/x-raw,              format=(string)BGR  ! "
    "appsink";

  cv::VideoCapture cap (gst_cap, cv::CAP_GSTREAMER);
  if (!cap.isOpened ()) {
    std::cout << "Failed to open camera." << std::endl;
    return (-1);
  }
  unsigned int width = cap.get (cv::CAP_PROP_FRAME_WIDTH);
  unsigned int height = cap.get (cv::CAP_PROP_FRAME_HEIGHT);
  unsigned int fps = cap.get (cv::CAP_PROP_FPS);
  unsigned int pixels = width * height;
  std::cout << " Frame size : " << width << " x " << height << ", " << pixels <<
    " Pixels " << fps << " FPS" << std::endl;

  cv::VideoWriter h264_shmsink
     ("appsrc is-live=true ! queue ! videoconvert ! video/x-raw, format=RGBA ! nvvidconv ! "
      "omxh264enc insert-vui=true ! video/x-h264, stream-format=byte-stream ! h264parse ! shmsink socket-path=/tmp/my_h264_sock ",
     cv::CAP_GSTREAMER, 0, fps, cv::Size (width, height));
  if (!h264_shmsink.isOpened ()) {
    std::cout << "Failed to open h264_shmsink writer." << std::endl;
    return (-2);
  }

  /* Loop for 3000 frames (100s at 30 fps) */
  cv::Mat frame_in;
  int frameCount = 0;
  while (frameCount++ < 3000) {
    if (!cap.read (frame_in)) {
      std::cout << "Capture read error" << std::endl;
      break;
    }
    else {
      h264_shmsink.write (frame_in);
    }
  }

  h264_shmsink.release();
  cap.release ();

  return 0;
}

Build, then launch it. It would write into /tmp/my_h264_sock.

So now you would launch your RTSP server with:

./test-launch "shmsrc socket-path=/tmp/my_h264_sock ! video/x-h264, stream-format=byte-stream, width=640, height=480, framerate=30/1 ! h264parse ! video/x-h264, stream-format=byte-stream ! rtph264pay pt=96 name=pay0 "

Just checked from localhost with:

gst-launch-1.0 -ev rtspsrc location=rtsp://127.0.0.1:8554/test ! application/x-rtp, media=video, encoding-name=H264 ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! videoconvert ! xvimagesink

…seems to work fine so far, at least in this case.

Got surprizing results with nvv4l2 encoder/decoder, so I propose using OMX plugins for now with this case.
For using VLC on receiver side, some extra work may be required (EDIT: mainly need to use property do-timestamp=1 for shmrc in test-launch pipeline).

Note that sockets are created by shmsink, but won’t be deleted if a shmsrc is still connected.
So after each trial when your app and test-launch are closed, remove any remaining socket with:

rm /tmp/my_h264_sock*

before trying again.

Hy @Honey_Patouceul

Indeed, this solution could work. I think I will do this temporary, but I’ll try to merge openCV and test-launch.c later to obtain something more user-friendly.

Thank you for your help,
Matteo Luci

If it can help anyone, I used the appsrc.c example of GstRtspServer to write this code :

#include "gstreamer-1.0/gst/gst.h"
#include "gstreamer-1.0/gst/gstmessage.h"
#include "gstreamer-1.0/gst/rtsp-server/rtsp-server.h"

#include "glib-2.0/glib.h"
#include <gstreamer-1.0/gst/app/app.h>


#include <opencv2/opencv.hpp>
#include <iostream>
#include <string>

#define W 1920
#define H 1080
#define FPS 30

typedef struct
{
	cv::VideoCapture *cap;
	cv::Mat *lastFrame;
	int *numberFrames;
	GstClockTime timestamp;
} MyContext;

// should be private data of c++ class
int numberFrames = 0;
cv::Mat lastFrame;

std::string launchString = "nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)" +
		std::to_string(W) + ", height=(int)" +
    	std::to_string(H) + ", format=(string)NV12, framerate=(fraction)" +
    	std::to_string(FPS) + "/1 ! nvvidconv flip-method=" +
    	std::to_string(0) + " ! video/x-raw, width=(int)" +
    	std::to_string(W) + ", height=(int)" +
    	std::to_string(H) + ", format=(string)BGRx"
    	" ! videoconvert ! video/x-raw, format=(string)BGR ! appsink sync=false";

cv::VideoCapture cap = cv::VideoCapture(launchString, cv::CAP_GSTREAMER);

/* called when we need to give data to appsrc */
static void
need_data (GstElement * appsrc, guint unused, MyContext * ctx)
{
	if (ctx->cap->isOpened()) {
		if (ctx->cap->read(*(ctx->lastFrame))) {

			GstBuffer *buffer;
			uint64_t size=W*H*4; // Image size * deth of BGRx;
			GstFlowReturn ret;
			buffer = gst_buffer_new_allocate (NULL, size, NULL);
			GstMapInfo map;
			gint8 *raw;

			gst_buffer_map (buffer, &map, GST_MAP_WRITE); // make buffer writable
			raw = (gint8 *)map.data;

			for (int i = 0; i<H; i++) {
				cv::Vec3b* ptr = ctx->lastFrame->ptr<cv::Vec3b>(i);
				for (int j = 0; j<W; j++) {
					uint64_t offset = ((i*W)+j)*4;
					raw[offset] = ptr[j][0];
					raw[offset+1] = ptr[j][1];
					raw[offset+2] = ptr[j][2];
					raw[offset+3] = 127;
				}
			}
			gst_buffer_unmap (buffer, &map);

			/* increment the timestamp every 1/FPS second */
			GST_BUFFER_PTS (buffer) = ctx->timestamp;
			GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale_int (1, GST_SECOND, FPS);
			ctx->timestamp += GST_BUFFER_DURATION (buffer);

			g_signal_emit_by_name (appsrc, "push-buffer", buffer, &ret);
			gst_buffer_unref (buffer);
        }
	}
}

/* called when a new media pipeline is constructed. We can query the
 * pipeline and configure our appsrc */
static void
media_configure (GstRTSPMediaFactory * factory, GstRTSPMedia * media, gpointer user_data)
{

	// should be incremented once on each frame for timestamping


	GstElement *element, *appsrc;
	MyContext *ctx;

	/* get the element used for providing the streams of the media */
	element = gst_rtsp_media_get_element (media);

	/* get our appsrc, we named it 'mysrc' with the name property */
	appsrc = gst_bin_get_by_name_recurse_up (GST_BIN (element), "mysrc");

	/* this instructs appsrc that we will be dealing with timed buffer */
	gst_util_set_object_arg (G_OBJECT (appsrc), "format", "time");
	/* configure the caps of the video */
	g_object_set (G_OBJECT (appsrc), "caps",
	gst_caps_new_simple ("video/x-raw",
				"format", G_TYPE_STRING, "BGRx",
				"width", G_TYPE_INT, W,
				"height", G_TYPE_INT, H,
				"framerate", GST_TYPE_FRACTION, FPS, 1, NULL), NULL);

	ctx = g_new0 (MyContext, 1);
	ctx->timestamp = 0;
	ctx->cap = &cap;
	ctx->lastFrame = &lastFrame;
	ctx->numberFrames = &numberFrames;

	/* make sure ther datais freed when the media is gone */
	g_object_set_data_full (G_OBJECT (media), "my-extra-data", ctx, (GDestroyNotify) g_free);

	/* install the callback that will be called when a buffer is needed */
	g_signal_connect (appsrc, "need-data", (GCallback) need_data, ctx);
	gst_object_unref (appsrc);
	gst_object_unref (element);
}

int main (int argc, char *argv[])
{
	GMainLoop *loop;
	GstRTSPServer *server;
	GstRTSPMountPoints *mounts;
	GstRTSPMediaFactory *factory;

	gst_init (&argc, &argv);

	loop = g_main_loop_new (NULL, FALSE);

	/* create a server instance */
	server = gst_rtsp_server_new ();
	/* get the mount points for this server, every server has a default object
	* that be used to map uri mount points to media factories */
    mounts = gst_rtsp_server_get_mount_points (server);

	/* make a media factory for a test stream. The default media factory can use
	* gst-launch syntax to create pipelines.
	* any launch line works as long as it contains elements named pay%d. Each
	* element with pay%d names will be a stream */
	factory = gst_rtsp_media_factory_new ();
	gst_rtsp_media_factory_set_launch (factory,
			"( appsrc name=mysrc is-live=true ! videoconvert ! omxh265enc ! rtph265pay mtu=1400 name=pay0 pt=96 )");
	gst_rtsp_media_factory_set_shared (factory, TRUE);
	/* notify when our media is ready, This is called whenever someone asks for
	* the media and a new pipeline with our appsrc is created */
	g_signal_connect (factory, "media-configure", (GCallback) media_configure, NULL);

	/* attach the test factory to the /test url */
	gst_rtsp_mount_points_add_factory (mounts, "/test", factory);

	/* don't need the ref to the mounts anymore */
	g_object_unref (mounts);

	/* attach the server to the default maincontext */
	gst_rtsp_server_attach (server, NULL);

	/* start serving */
	g_print ("stream ready at rtsp://127.0.0.1:8554/test\n");
	g_main_loop_run (loop);

	return 0;
}

All this code can be put in a C++ class and the GMainloop can loop in a separate thread.
With this code, it is possible to send cv::mat data to an RTSP stream, and therefore, any processing can be done on those frames.

I think that there is a lot of improvement that must be done to optimize all this, but it is a working base.

2 Likes