Converting video frames in proto

Hi,
I am a beginner and trying to replace a camera input with a video stream. So similar to the input/output images in OpenCV Edge Detector, I am writing a node to read video frames with OpenCV and as output I need to send them as a proto. But I don’t need any of the protos (pinhole, distortion etc.) encoded in ColorCameraProto, just the image itself. How can I convert an OpenCV matrix to ImageProto? Is it even possible to use ImageProto alone?
Thanks!

Hi @e_lif_kaya,
look at this post about converting a RGB camera proto to a grey scale camera proto using open cv (code inspired by the open cv sobel sample),
also take a look at this post to convert a camera proto to a image proto.
Hope this can help you,
tell me if you need some clarifications about my codes or any other advice,
Regards,
Planktos

Bu that’s the problem, both examples are converting camera images, which are already formatted as protos. So you can just get the information from input_image, using functions like input.getDistortion() etc. and move the attributes from the input_image to output_image easily.

In my case there is no camera input, but just a frame, like (using opencv):

VideoCapture cap(file_path);
Mat frame;
cap >> frame;

So my aim is to convert this simplpe frame I read, and transmit it as an output proto. I can’t really use ColorCameraProto in that case, can I?

//Build a new ColorCameraProto
 auto output = tx_output_image().initProto();
//Pinhole
  auto pinhole = output.initPinhole();
  ToProto(Vector2d(3147.54098361, 3160.49382716), pinhole.getFocal());
  ToProto(Vector2d(519.4719202793973, 693.0736876181506), pinholeL.getCenter());
  pinhole.setCols(1280);
  pinhole.setRows(960);
// Distortion
  Vector5d disto;
  disto << 0.01, 0.01, 0.01, 0, 0;
  auto distortion = output.initDistortion();
  distortion.setModel(DistortionProto::DistortionModel::BROWN);
  ToProto(disto, distortion.getCoefficients());
//Image
 output.setCols(1280);
 output.setRows(960);
 Image3ub output_image(960,1280);//create a new 3 channels image
cv::Mat image =  cv::Mat(rows, cols, CV_8UC3, const_cast<void*>(static_cast<const void*>(output_image.data().pointer())));//point the mat to proto
//modify your cv Mat there
 ToProto(std::move(output_image), output, tx_output_image().buffers());
//Send the ColorCameraProto
 tx_output_image().publish(rx_start().acqtime());

This code is not tested so if you get an error tell me, should be easy to solve,
You could use the open cv function copy() to copy your input mat to the output mat.(Copy(input_image,output_image);)
Regards

Wow amazing, many thanks! I’ll try it now and let you know, if that works, I’ll mark it as a solution.

Soo, it kinda worked. I added Colorspace to the proto as the ColorCameraViewer was throwing an error and not displaying anything on its channel. Copy(input,output) was not necessary, since there was no input proto I needed to own before transmitting. Now the result: It displays the video not clearly, a couple of frames can still be seen but as if the video data is corrupted. Looks like watching TV with bad signal, a lot of distortion noise included. I tried it with many different videos. Here is my tick function below. Any suggestions for improving the video quality? Nevertheless this is a huge improvement for my project, thanks a lot again for helping!

void VideoStream::tick(){
    cv::Mat frame;

    // Read a frame of the video
    cap.read(frame);
    ASSERT(!frame.empty(), "Frame is empty" );
    LOG_INFO("Frame read!");

    const size_t col = frame.cols;
    const size_t row = frame.rows;

    // Build a new ColorCameraProto
    auto output = tx_output_image().initProto();

    // Pinhole
    auto pinhole = output.initPinhole();
    ToProto(Vector2d(3147.54098361, 3160.49382716), pinhole.getFocal());
    ToProto(Vector2d(519.4719202793973, 693.0736876181506), pinhole.getCenter());      
    pinhole.setCols(col);
    pinhole.setRows(row);

    // Distortion
    Vector5d disto;
    disto << 0.01, 0.01, 0.01, 0, 0;
    auto distortion = output.initDistortion();
    distortion.setModel(DistortionProto::DistortionModel::BROWN);
    ToProto(disto, distortion.getCoefficients());

    // Image
    output.getImage().setCols(col);
    output.getImage().setRows(row);

    // Create a new 3 channels image
    Image3ub output_image(row,col);

    // set colorspace
    output.setColorSpace(ColorCameraProto::ColorSpace::RGB);

    //point the mat to proto - where I pointed it to the current video frame
    frame = cv::Mat(row, col, CV_8UC3, const_cast<void*>(static_cast<const void*>
        (output_image.data().pointer())));

    // modify your cv Mat there
    ToProto(std::move(output_image), output.getImage().tx_output_image().buffers());

    //Send the ColorCameraProto
    tx_output_image().publish();

}

You may try with smiple cast

Do you check opening in start ?

  // Check if camera opened successfully
  if(!cap.isOpened()){
    cout << "Error opening video stream or file" << endl;
    return -1;
  }

Is it fine when using imshow ?

    Mat frame;
    // Capture frame-by-frame
    cap >> frame;
    // If the frame is empty, break immediately
    if (frame.empty())
      break;
    // Display the resulting frame
    imshow( "Frame", frame );

If you are stuck with this issue you could use Deepstream to call Gstreamer then modify your image with opencv in a codelet. I use a Gstreamer pipeline to get my video sources and it works perfectly !

Yes thank you, just adding the Gstreamer in the json has solved this!