How to get H264 streaming from Gstreamer pipeline (every frame)?

We are using gstreamer to write the processed video on the local computer. But now we want to send this output over the network without writing on the local computer, so that anyone can access this output using IP. Right now we are using following code to write video using gstreamer. (attached image)

So How we can post over the network rather than writing on the local system? What gstreamer functiuon we can use to get every frame and then send over the network?

The easiest way would be to use RTP stream over UDP. You would have to use a multicast address for several clients.
Posting image prevents from copying code…so I’d say:
In you videoWriter, replace:
matroskamux by rtph264pay config-interval=1
and
filesink with udpsink host=$IP_TO_STREAM_TO port=5000.

You may also add insert-sps-pps=1 or insert-vui=1 into omxh264enc options.

Hi Sir, Thanks for your prompt reply , Now it’s not writing the output on the system , so it might be sending the output over the network successfully, Now How I can receive/view the output on the other system or VLC media player?
I tried to receive on the VLC media player using 192.168.1.5:5000 but failed.
I am using Xavier NX and it’s IP is 192.168.1.5
I also attached current code for your convenience. 1.cpp (2.2 KB)

You would try on receiver something like:

#On Jetson:
gst-launch-1.0 -v udpsrc port=5000 ! 'application/x-rtp, encoding-name=H264, payload=96' ! rtph264depay ! nvv4l2decoder ! nvvidconv ! nvoverlaysink

#On any host
gst-launch-1.0 -v udpsrc port=5000 ! 'application/x-rtp, encoding-name=H264, payload=96' ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! xvimagesink

2 Likes

Sir actually we are already using other IP camera and receive it’s processed video on VLC media player or any other computer using a program only with the IP address. For example we can see output in the VLC media player using “rtsp://192.168.1.88/live/3”
Now lets say my receiver is windows os and I want to receive output in the VLC media player. Do you mean I should use only
"
#On any host
gst-launch-1.0 -v udpsrc port=5000 ! ‘application/x-rtp, encoding-name=H264, payload=96’ ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! xvimagesink "
in the VLC media player or I need to combine it with another piece of code or need to run this command in the terminal?

+1 When I am trying on Jetson cmd "gst-launch-1.0 -v udpsrc port=5000 ! ‘application/x-rtp, encoding-name=H264, payload=96’ ! rtph264depay ! nvv4l2decoder ! nvvidconv ! nvoverlaysink
"
It does not show the output , it stuck at the following stage shown in the image.

+We have verified that it’s sending information now. So there is only problem on the receiving end. You can check the screenshot

You may try omxh264dec instead of nvv4l2decoder:

gst-launch-1.0 -v udpsrc port=5000 ! ‘application/x-rtp, encoding-name=H264, payload=96’ ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! nvoverlaysink
1 Like

So nice of you sir. We are able to receive the output on the same device (Xavier NX) with the given command.

But we want to receive output on any other computers in the office. for example using VLC media player installed on windows os on another system. Now How to receive on other computers? Can we use IP adress?

we are accessing another camera on all computers using vlc media player with “rtsp://192.168.1.88/live/3”.

Can we do something like this?

The ultimate goal is we process video on NX , post over the network and anyone can receive it with IP address.

This is another topic.
In short, you would create on receiver a sdp file with:

m=video 5000 RTP/AVP 96
c=IN IP4 127.0.0.1
a=rtpmap:96 H264/90000

(here just using localhost, you would adapt for your case)

and then you would just use:

vlc test.sdp

RTSP is another way of RTP streaming. You may try this.

1 Like

Sir we have tired with the .sdp file “vlc test.sdp” but it is not showing anything. See the screenshot of the .sdp file and error.

Update: Sir we are able to receive the output with test.sdp

m=video 5000 RTP/AVP 96 c=IN IP4 127.0.0.1 a=rtpmap:96 H264/90000

But now the problem is 'only given IP in the udpsink host=192.168.1.45 port=5000"’ is able to receive. no one else is able to receive it. that’s mean we are able to send to only one IP at a time. We want to just send over the network and anyone could receive it. How to send it for everyone on the local network?

You would try a multicast address such as:

gst-launch-1.0 -ev nvarguscamerasrc ! nvv4l2h264enc insert-vui=1 ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=224.1.1.1 auto-multicast=true port=5000

Sir we have tried gst-launch-1.0 -ev nvarguscamerasrc ! nvv4l2h264enc insert-vui=1 ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=224.1.1.1 auto-multicast=true port=5000 on the receiver end but still unable to receive. There is errr

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:557 No cameras available
Got EOS from element "pipeline0".
Execution ended after 0:00:00.077923699

Do we also need to change something on the sender’s side?

As I mentioned earlier this is the part of the code on the sender side

const char*  gst_writer = "appsrc ! video/x-raw, format=(string)BGR ! \
		videoconvert ! video/x-raw, format=(string)I420 ! \
	        nvvidconv ! video/x-raw(memory:NVMM) ! \
		omxh264enc insert-sps-pps=1 ! h264parse ! \
        rtph264pay config-interval=1 ! udpsink host=192.168.1.45 port=5000";

and only the given IP. udpsink host=192.168.1.45 is able to receive the output with

m=video 5000 RTP/AVP 96 c=IN IP4 127.0.0.1 a=rtpmap:96 H264/90000

I also attached complete code file.
1.cpp (2.3 KB)

Seems you had a process still using the camera, or tried an unsupported mode, or you’ve crashed nvargus camera deamon, or …

In my case it works using a Jetson with a monitor attached and a GUI session opened (this might make a diference for argus) for streaming with:

gst-launch-1.0 -ev nvarguscamerasrc ! nvv4l2h264enc insert-vui=1 ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=224.1.1.1 auto-multicast=true port=5000

Then, from another host on same LAN, I’ve edited my test.sdp so that I give multicast address 224.1.1.1:

m=video 5000 RTP/AVP 96
c=IN IP4 224.1.1.1
a=rtpmap:96 H264/90000

then using vlc test.sdp works.

If it doesn’t work, you may ask your network admin for a multicast address and firewall rules if any.

Hi sir, This problem still not resolved, we are able to send to only one client at a time.
Now I have another question:

How can pass the opencv frame to gstreamer for encoding and get the buffer /frame information to use in another variable?

For example this function receives input frame from video and then perform some processing and then return processed frame, now how i can encode this frame with gstreamer to h264 and then pass to another variable?
I do not know so many technical details about gstreamer so I am sorry if there are any mistakes in my question.

 while(1)
    {
        if (!writer.isOpened()) 
        {
            std::cout<<"Failed to open writer."<<std::endl;
            return (-2);
        }
        if (!cap.read(frame_in)) 
        {
            std::cout<<"Capture read error"<<std::endl;
            break;
        }
        else  
        {
            frame_in.copyTo(frame_out);
            cv::imshow("MyCameraPreview",frame_in);
            writer.write(frame_out);
        }
        if (char(cv::waitKey(1)) == 'q') break;	
    }

    cap.release();
    // writer.release();

    return 0;
}

Not sure what gets wrong in your case.

The following example works fine for me. Opencv code reading nvarguscamera and sending to RTP/UDP multicast 224.1.1.1:5000:

#include <iostream>

#include <opencv2/opencv.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>

int main ()
{
  setenv ("GST_DEBUG", "*:3", 0);

  /* Setup capture with gstreamer pipeline from onboard camera converting into BGR frames for app */
  const char *gst_cap =
    "nvarguscamerasrc  ! video/x-raw(memory:NVMM), format=(string)NV12, width=(int)640, height=(int)480, framerate=(fraction)30/1 ! "
    "nvvidconv    ! video/x-raw,              format=(string)BGRx ! "
    "videoconvert ! video/x-raw,              format=(string)BGR  ! "
    "appsink";

  cv::VideoCapture cap (gst_cap, cv::CAP_GSTREAMER);
  if (!cap.isOpened ()) {
    std::cout << "Failed to open camera." << std::endl;
    return (-1);
  }
  unsigned int width = cap.get (cv::CAP_PROP_FRAME_WIDTH);
  unsigned int height = cap.get (cv::CAP_PROP_FRAME_HEIGHT);
  unsigned int fps = cap.get (cv::CAP_PROP_FPS);
  unsigned int pixels = width * height;
  std::cout << " Frame size : " << width << " x " << height << ", " << pixels <<
    " Pixels @" << fps << " FPS" << std::endl;


  /* video writer to RTP/UDP sink multicast at 224.1.1.1 */
  cv::VideoWriter gst_udpsink("appsrc ! video/x-raw, format=BGR, pixel-aspect-ratio=1/1 ! queue ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! nvv4l2h264enc insert-vui=1 ! video/x-h264, stream-format=byte-stream, alignment=au ! h264parse ! video/x-h264, stream-format=byte-stream ! rtph264pay pt=96 config-interval=1 ! application/x-rtp, media=video, encoding-name=H264 ! udpsink host=224.1.1.1 port=5000 auto-multicast=true ", 0, fps, cv::Size (width, height));
  if (!gst_udpsink.isOpened ()) {
    std::cout << "Failed to open gst_udpsink writer." << std::endl;
    return (-8);
  }


  /* Loop for 3000 frames (100s at 30 fps) */
  cv::Mat frame_in, frame_out;
  int frameCount = 0;
  while (frameCount++ < 3000) {
    if (!cap.read (frame_in)) {
      std::cout << "Capture read error" << std::endl;
      break;
    }
    else {
       
       // Here you would make your processing
       // frame_out = YourFrameProcess(frame_in);
       // For this example, no processing
       frame_out = frame_in;

       gst_udpsink.write(frame_out);
    }
  }
  gst_udpsink.release();
  cap.release ();

  return 0;
}

When running, from any host on same LAN, this works for me:

gst-launch-1.0 -v udpsrc multicast-group=224.1.1.1 auto-multicast=true port=5000 ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer ! rtph264depay ! h264parse ! avdec_h264 ! xvimagesink

And it also works from any host with VLC using a sdp file as in my previous post:

vlc test.sdp

If not working, be sure your firewall accepts incoming connection from 224.1.1.1 and to destination port 5000.

[EDIT: you may also try to increase kernel receive socket max buffer size on receiver before with:

sudo sysctl -w net.core.rmem_max=26214400

]

1 Like

So nice of you sir, Now it can send to everyone on the same LAN with your new code.

Now I have another question. How I can get every frame after gstreamer encoding? we do not want to send to send directly, we want to pass that encoded frame to our project and then send through it, because that is fast.

So Now the only question is : How to get gstreamer encoded frame that can use in other variables/functions rather then uploading video stream over network?

Not sure what you mean… In my example, you would just call your processing for each received frame before sending it to the RTP stream:

       // Here you would make your processing
       frame_out = YourFrameProcess(frame_in);
       // For this example, no processing
       //frame_out = frame_in;

So you would have to define your own function for processing a received frame.

Sir in the previous example we was doing these steps.

1- Receive input video from webcam, decode using gstreamer.
2- Pass this decoded frame to opencv functions for some pre-processing
3- Encode these pre-processed frames using gstreamer and send over the network.

Now we want to just change the 3rd step. We want to get every H264 encoded frame and use it in another function. Just skip the sending/saving process.
Because we have our own sending/save functions, we want to get H264 encoded frame and pass to those functions.

So how to get every encoded H264 frame. ? we do not need gst_writer, we need just H264 encoding.

Can this function edit for this purpose? or is there any other gstreamer function that we can use?

const char*  gst_writer = "appsrc ! video/x-raw, format=BGR, pixel-aspect-ratio=1/1 ! \
    queue ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! nvv4l2h264enc insert-vui=1 ! \
    video/x-h264, stream-format=byte-stream, alignment=au ! h264parse !\
    video/x-h264, stream-format=byte-stream ! rtph264pay pt=96 config-interval=1 ! \
    application/x-rtp, media=video, encoding-name=H264 ! \
    udpsink host=224.1.1.1 port=5000 auto-multicast=true";