Streaming videos from opencv to gstreamer on udp sink?

Hi,
I’m currently trying to stream videos perform some opencv operations and republish these jpeg as rtsp streams. What I currently have figured out,

  1. I can publish a stream using gstreamer’s videotestsrc and publish jpeg frames to the udp sink, and when I run gstreamer pipeline I can see the exact stream, no issues.

  2. Running gstreamer pipeline on the terminal on a webcam, and publish to udp sink and view them on a separate pipeline. The command is as follows,
    Sender Pipeline,

     gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg,width=640,height=480,framerate=30/1 ! \
     jpegparse !   rtpjpegpay ! udpsink host=127.0.0.1 port=5000
    

Receiver Pipeline,

gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=JPEG,payload=26 ! \
    rtpjpegdepay ! jpegdec !   autovideosink

Right now to the issue that I face, the code is as follows,

#include <iostream>
#include <gst/rtsp-server/rtsp-server.h>
#include "opencv2/highgui.hpp"
#include "opencv2/imgproc.hpp"

using namespace std;
using namespace cv;

int main()
{
VideoCapture cap("videotestsrc pattern=7 ! video/x-raw,format=BGR,width=640,height=480,framerate=30/1 !   appsink",CAP_GSTREAMER);

VideoWriter out("appsrc ! autovideoconvert ! videoscale ! video/x-raw,format=I420,width=1280,height=720,framerate=30/1 ! jpegenc ! rtpjpegpay ! udpsink host=127.0.0.1 port=5001",CAP_GSTREAMER,0,30,Size(1280,720),true);

if(!cap.isOpened() || !out.isOpened())
{
    cout<<"VideoCapture or VideoWriter not opened"<<endl;
    exit(-1);
}

Mat frame;
Mat resized;

while(true) {

    cap.read(frame);

    cv::resize(frame, resized, cv::Size(1280, 720));

    if(frame.empty())
        break;

    out.write(frame);

    // imshow( "Frame", resized);
    //
    // char c=(char)waitKey(25);
    // if(c==27)
    //   break;
}
cap.release();

}

In the above code in the line where I read frames with gstreamer’s videotestsrc the pipeline works without an issue, but with mp4 it seems to be an issue. I’ll attach a sample image below for reference,

Link to the output with artifacts.

However I don’t see the issue with videotestsrc, I get the same feed. My receiver pipeline is same as above,
gst-launch-1.0 -v udpsrc port=5001 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=JPEG, payload=26 ! rtpjpegdepay ! jpegdec ! xvimagesink sync=0

Please take a look and let me know if I have issues anywhere in the receiver pipeline or the way I’m writing images to the udp sink. Any help would be appreciated. Thanks.

Do you mean your pipeline does not get correct display with mp4 format source? Can you show the pipeline you used for mp4 file?

@Fiona.Chen, the pipeline is in the above code in the Videowriter element of opencv. There is one thing I would like to mention after putting hours of work into it, what I observer is that images were getting streamed perfectly fine at 640480 using an rtsp url as my source, when I did the same by reading my input source from a file, at the same resolution I was getting the artifacts that I linked above. So now I have the problem partially solved. However what I noticed is that although streaming with 640480 works fine with rtsp source, higher resolution streaming is problematic. The feed is choppy and it looks like a lot of frames are being dropped. Please refer to the above code attached for the pipeline. Thanks

Can you upload the local file? Which format is it?

@Fiona.Chen, sorry I currently do not have the authority to share a video due to client constraints. But the format was an mp4 video obtained from a 360 deg camera. Could you please help me with the other issue, since I wouldn’t exactly be using a file, since mostly everything would be done with rtsp source as my input source and this isn’t causing an issue. What I am currently facing is pushing images to a udp sink of higher resolution. At a lower resolution everything works fine. But as soon as I increase the resolution to 1280*720 and start the receiver pipeline, the gstreamer output is choppy. Any idea how would I be able to achieve a smooth flow at a higher resolution. Thanks

The pipeline is OK for UDP streaming. Seems the network performance is bad so the frame dropping happens. So maybe you need to check the network performance first. If the network throughput is OK, you may try set larger “buffer-size” for “udpsrc” plugin to improve packet lost.