Hi,
I am a beginner and trying to replace a camera input with a video stream. So similar to the input/output images in OpenCV Edge Detector, I am writing a node to read video frames with OpenCV and as output I need to send them as a proto. But I don’t need any of the protos (pinhole, distortion etc.) encoded in ColorCameraProto, just the image itself. How can I convert an OpenCV matrix to ImageProto? Is it even possible to use ImageProto alone?
Thanks!
Hi @e_lif_kaya,
look at this post about converting a RGB camera proto to a grey scale camera proto using open cv (code inspired by the open cv sobel sample),
also take a look at this post to convert a camera proto to a image proto.
Hope this can help you,
tell me if you need some clarifications about my codes or any other advice,
Regards,
Planktos
Bu that’s the problem, both examples are converting camera images, which are already formatted as protos. So you can just get the information from input_image, using functions like input.getDistortion() etc. and move the attributes from the input_image to output_image easily.
In my case there is no camera input, but just a frame, like (using opencv):
VideoCapture cap(file_path);
Mat frame;
cap >> frame;
So my aim is to convert this simplpe frame I read, and transmit it as an output proto. I can’t really use ColorCameraProto in that case, can I?
//Build a new ColorCameraProto
auto output = tx_output_image().initProto();
//Pinhole
auto pinhole = output.initPinhole();
ToProto(Vector2d(3147.54098361, 3160.49382716), pinhole.getFocal());
ToProto(Vector2d(519.4719202793973, 693.0736876181506), pinholeL.getCenter());
pinhole.setCols(1280);
pinhole.setRows(960);
// Distortion
Vector5d disto;
disto << 0.01, 0.01, 0.01, 0, 0;
auto distortion = output.initDistortion();
distortion.setModel(DistortionProto::DistortionModel::BROWN);
ToProto(disto, distortion.getCoefficients());
//Image
output.setCols(1280);
output.setRows(960);
Image3ub output_image(960,1280);//create a new 3 channels image
cv::Mat image = cv::Mat(rows, cols, CV_8UC3, const_cast<void*>(static_cast<const void*>(output_image.data().pointer())));//point the mat to proto
//modify your cv Mat there
ToProto(std::move(output_image), output, tx_output_image().buffers());
//Send the ColorCameraProto
tx_output_image().publish(rx_start().acqtime());
This code is not tested so if you get an error tell me, should be easy to solve,
You could use the open cv function copy() to copy your input mat to the output mat.(Copy(input_image,output_image);)
Regards
Wow amazing, many thanks! I’ll try it now and let you know, if that works, I’ll mark it as a solution.
Soo, it kinda worked. I added Colorspace to the proto as the ColorCameraViewer was throwing an error and not displaying anything on its channel. Copy(input,output) was not necessary, since there was no input proto I needed to own before transmitting. Now the result: It displays the video not clearly, a couple of frames can still be seen but as if the video data is corrupted. Looks like watching TV with bad signal, a lot of distortion noise included. I tried it with many different videos. Here is my tick function below. Any suggestions for improving the video quality? Nevertheless this is a huge improvement for my project, thanks a lot again for helping!
void VideoStream::tick(){
cv::Mat frame;
// Read a frame of the video
cap.read(frame);
ASSERT(!frame.empty(), "Frame is empty" );
LOG_INFO("Frame read!");
const size_t col = frame.cols;
const size_t row = frame.rows;
// Build a new ColorCameraProto
auto output = tx_output_image().initProto();
// Pinhole
auto pinhole = output.initPinhole();
ToProto(Vector2d(3147.54098361, 3160.49382716), pinhole.getFocal());
ToProto(Vector2d(519.4719202793973, 693.0736876181506), pinhole.getCenter());
pinhole.setCols(col);
pinhole.setRows(row);
// Distortion
Vector5d disto;
disto << 0.01, 0.01, 0.01, 0, 0;
auto distortion = output.initDistortion();
distortion.setModel(DistortionProto::DistortionModel::BROWN);
ToProto(disto, distortion.getCoefficients());
// Image
output.getImage().setCols(col);
output.getImage().setRows(row);
// Create a new 3 channels image
Image3ub output_image(row,col);
// set colorspace
output.setColorSpace(ColorCameraProto::ColorSpace::RGB);
//point the mat to proto - where I pointed it to the current video frame
frame = cv::Mat(row, col, CV_8UC3, const_cast<void*>(static_cast<const void*>
(output_image.data().pointer())));
// modify your cv Mat there
ToProto(std::move(output_image), output.getImage().tx_output_image().buffers());
//Send the ColorCameraProto
tx_output_image().publish();
}
You may try with smiple cast
Do you check opening in start ?
// Check if camera opened successfully
if(!cap.isOpened()){
cout << "Error opening video stream or file" << endl;
return -1;
}
Is it fine when using imshow ?
Mat frame;
// Capture frame-by-frame
cap >> frame;
// If the frame is empty, break immediately
if (frame.empty())
break;
// Display the resulting frame
imshow( "Frame", frame );
If you are stuck with this issue you could use Deepstream to call Gstreamer then modify your image with opencv in a codelet. I use a Gstreamer pipeline to get my video sources and it works perfectly !
Yes thank you, just adding the Gstreamer in the json has solved this!