Delay acumulation with gstreamer and opencv

Hi,
I’m trying to paint some circles in the image with opencv.
I get the stream with gstreamer and I try to process the image with opencv with the next one:

GstElement* conv=gst_bin_get_by_name(GST_BIN(gst_pipeline), “myconv”);
GstPad *src_pad=gst_element_get_static_pad (conv, “src”);
gst_pad_add_probe(src_pad, GST_PAD_PROBE_TYPE_BUGFFER, frame_processing, NULL,NULL);

In the frame_processing function I have something like:

static GstPadProbeReturn frame_processing (GstPad * pad, GstPadProbeInfo * info, gpointer u_data)
{

GstBuffer *buffer=(GstBuffer *) info->data;
GstMapInfo map={0};
int dmabuf_fd=0;
gst_buffer_map (buffer, &map, GST_MAP_READ);
ExtractFdFromNvBuffer((void *)map.data, &dmabuf_fd);
{

EGLImageKHR egl_image;
egl_image=NvEGLImageFromFd(egl_display, dmabuf_fd);
CUresult status;
CUeglFrame egl_frame;
CUgraphicsResource p_resource=NULL;
cudaFree(0);
status=cuGraphicsEGLRegisterImage(&p_resource, egl_image, CU_GRAPHICS_MAP_RESOURCE_FLAGS_NONE);
status=cuGraphicsResourceGetMappedEglFrame(&egl_frame, p_resource, 0, 0);
status=cuCtxSynchronize();
cv::cuda::GpuMat d_mat(H, W, CV_8UC4, egl_frame.frame.pPitch[0]);
cv::Mat img;
d_mat.download(img);
circle(img, cv::Point(500,500), 300, cv::Scalar(0,255,0),-1);
d_mat.upload(img);
status=cuCtxSynchronize();
status=cuGraphicsUnregisterResource(p_resource);

}
gst_buffer_unmap(buffer, &map);
return GST_PAD_PROBE_OK;

}

When I try to make something like this, I see how the video slows down and accumulate a lot of delay, I think the reason is the computing cost to transform the cuda mat to opencv mat, because if I remove that, the video works fine.
Anyone knows any solution to that?

Thanks in advance.

Hi,
Looks like the circle() function executes on CPU buffers. You may try to get cv::Mat instead of cv::cuda::GpuMat. Please refer to

and run sudo nvpmodel-m 0 and sudo jetson_clocks to get max CPU performance.

cv::cuda::GpuMat is better if you use cuda filter function calls.

Sorry for the delay in respondering, but I can’t work with that before.
I tried putting the nvpmodel-m0 and sudo jetson_clocks but I get the same results.

On the other hand, I have checked how many frames I get by second and I see that I have ~36 so I should see the video well and smooth, but I keep seeing how the video is continuously slow down and accumulating delay.
Maybe the problem is in the pipelines? I’m feel very lost with that problem.

Hi,
If your source is a video file, you may try this simple code:

It is RTSP source. For video file source, it should work by replace rtspsrc ! rtph264depay with filesrc. You may check if you observe the same accumulation in running it.

My video must come from a rtsp source, I can’t change that.
There is a way to make that don’t accumulate delay?

Hi,
We have seen performance issues if there are heavy operations in prob functions. A similar topic:

Is it possible to link to appsink plugin and use OpenCV functions in it?