How to use OpenCV+CUDA in deepstream prob

[chrichard]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) T4 dGPU
• DeepStream Version 6.3
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

The plan is to use OpenCV’s CUDA method to process GPU data in the callback function(in osd prob), as shown below, but the output image of DeepStream has not changed. Please help see what the reason is.

---

GstPadProbeReturn post_process(GstPad *pad, GstPadProbeInfo *info,
gpointer u_data)
{
NvDsMetaList *l_frame = NULL;
GstBuffer *buf = (GstBuffer *)info->data;
GstMapInfo in_map_info;
NvBufSurface *surface = NULL;
NvDsMetaList *l_obj = NULL;
NvDsObjectMeta *obj_meta = NULL;

if (!gst_buffer_map(buf, &in_map_info, GST_MAP_READWRITE))
{
    g_print("Error: Failed to map gst buffer\n");
    gst_buffer_unmap(buf, &in_map_info);
    return GST_PAD_PROBE_OK;
}

NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf); 
surface = (NvBufSurface *)in_map_info.data;
for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
     l_frame = l_frame->next)
{
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
    int offset = 0;
    fps_count++;

    cv::cuda::GpuMat gpu_blur, gpu_frame;
    gint frame_width = (gint)surface->surfaceList[frame_meta->batch_id].width;
    gint frame_height = (gint)surface->surfaceList[frame_meta->batch_id].height;
    gint frame_step = surface->surfaceList[frame_meta->batch_id].pitch;
    void *frame_data = surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0];

    gpu_frame = cv::cuda::GpuMat(frame_height, frame_width,
                                 CV_8UC3, surface->surfaceList[frame_meta->batch_id].dataPtr, frame_step);
    cv::cuda::resize(gpu_frame, gpu_frame, mosaicSize); 
    cv::cuda::resize(gpu_frame, gpu_frame, cv::Size(frame_width,frame_height)); 
}
gst_buffer_unmap(buf, &in_map_info);
return GST_PAD_PROBE_OK;

}

[yuweiw]

Which sample did you run, our demo or your own? Could you attach the whole demo code to us?

[chrichard]

deepstream_test1_app.cpp (12.9 KB)

I modified the code from test1 ,I hope to manipulate the GPU’s image via cuda+opencv , instead of downloading it to the CPU for processing.
(1)output video at “rtsp://localhost:8554/ds-test”
(2)i donnot know how to modify image direct on GPU
(3)work on T4 GPU,not jetson.

[yuweiw]

Just from your code, you didn’t change the raw data of surface->surfaceList[frame_meta->batch_id].dataPtr. So the output image of DeepStream has not changed.
You can use NvBufSurfTransform API to resize the image. You can refer to our source code: opt\nvidia\deepstream\deepstream-6.3\sources\gst-plugins\gst-dsexample\gstdsexample_optimized.cpp.

[chrichard]

Thanks, I change the raw data , it works well.
But the image processing was not as fast as I thought, and I continued to deal with this problem.

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

OK. And we suggest you use the NvBufSurfTransform API to resize the image.