Populate NvBufSurface from cv::Mat

[twangbarang]

• Hardware Platform: GPU
• DeepStream Version: 5.0.0
• TensorRT Version: 7.0.0.11
• NVIDIA GPU Driver Version (valid for GPU only): 460.32.03

I edited the gst-nvinfer plugin to do some pre-processing before inference. I actually want to exchange the buffer for an inference on a detected bbox in an sgie. All that happens in gst_nvinfer_process_objects just before calling get_converted_buffer.

Here is what I’m doing:

1. Create a copy of the actual buffer
2. Create a cv::Mat from that buffer
3. Do some postprocessing on that mat
4. Create and init a NvBufSurface
5. Copy cv::Mat content from 2 into NvBufSurface

I’m following this thread but unfortunately it does not work for me

Here’s my code:

NvBufSurface *nvbuf_copy = nullptr;
cv::Mat mat;
NvBufSurfaceCreateParams create_params;
create_params.gpuId = in_surf->gpuId;
create_params.width = in_surf->surfaceList[frame_meta->batch_id].width;
create_params.height = in_surf->surfaceList[frame_meta->batch_id].height;
create_params.size = 0;
create_params.colorFormat = NVBUF_COLOR_FORMAT_BGR;
create_params.layout = in_surf->surfaceList[frame_meta->batch_id].layout;
create_params.memType = NVBUF_MEM_CUDA_UNIFIED;
NvBufSurfaceCreate(&nvbuf_copy, 1, &create_params);

NvBufSurfTransformParams transform_params;
NvBufSurfTransformRect src_rect = {0, 0, in_surf->surfaceList[frame_meta->batch_id].width,
in_surf->surfaceList[frame_meta->batch_id].height};
NvBufSurfTransformRect dst_rect = {0, 0, in_surf->surfaceList[frame_meta->batch_id].width,
in_surf->surfaceList[frame_meta->batch_id].height};

transform_params.src_rect = &src_rect;
transform_params.dst_rect = &dst_rect;

NvBufSurfTransform_Error err = NvBufSurfTransform(in_surf, nvbuf_copy, &transform_params);
if (err != NvBufSurfTransformError_Success) {
GST_ELEMENT_ERROR(nvinfer, STREAM, FAILED, (“Error for NvBufSurfTransform”),
(std::to_string(err).c_str()));
return GST_FLOW_ERROR;
}

if (NvBufSurfaceMap(nvbuf_copy, 0, 0, NVBUF_MAP_READ) != 0) {
GST_ELEMENT_ERROR(nvinfer, STREAM, FAILED, (“Error mapping buffer”), (NULL));
return GST_FLOW_ERROR;
}

NvBufSurfaceSyncForCpu(nvbuf_copy, 0, 0);

mat = cv::Mat(in_surf->surfaceList[frame_meta->batch_id].height,
in_surf->surfaceList[frame_meta->batch_id].width, CV_8UC3,
nvbuf_copy->surfaceList[frame_meta->batch_id].mappedAddr.addr[0],
nvbuf_copy->surfaceList[frame_meta->batch_id].pitch);

cv::Mat transformation_matrix(2, 3, CV_32F);
get_transformation_matrix(vals, dst_vals, 5, transformation_matrix);
cv::Mat warped(mat.rows, mat.cols, CV_32F);
cv::warpAffine(mat, warped, transformation_matrix, warped.size());

cv::Rect roi(0, 0, 112, 112);
cv::Mat crop = warped(roi);

NvBufSurface *inf_buf = nullptr;
create_params.gpuId = in_surf->gpuId;
create_params.width = 112;
create_params.height = 112;
create_params.size = 0;
create_params.colorFormat = NVBUF_COLOR_FORMAT_BGR;
create_params.layout = NVBUF_LAYOUT_PITCH;
create_params.memType = NVBUF_MEM_CUDA_UNIFIED;
NvBufSurfaceCreate(&inf_buf, 1, &create_params);

// cf.
// OpenCV Mat to NvBufSurface (to use in NvBufSurfTransform) - #13 by DaneLLL
inf_buf->numFilled = 1;

NvBufSurfaceMemSet(inf_buf, 0, 0, 0);

if (NvBufSurfaceMap(inf_buf, 0, 0, NVBUF_MAP_READ_WRITE) != 0) {
GST_ELEMENT_ERROR(nvinfer, STREAM, FAILED, (“Failed mapping buf inf_buf”), (NULL));
return GST_FLOW_ERROR;
}

if (NvBufSurfaceSyncForCpu(inf_buf, 0, 0) != 0) {
GST_ELEMENT_ERROR(nvinfer, STREAM, FAILED, (“Failed syncing inf_buf for CPU”), (NULL));
return GST_FLOW_ERROR;
}

memcpy(inf_buf->surfaceList[0].mappedAddr.addr[0], crop.ptr(), 112 * 112 * 3);

Questions:

1. Is there a way to copy the data without this intermediate nvbuf_copy? Meaning in this part

mat = cv::Mat(in_surf->surfaceList[frame_meta->batch_id].height,
in_surf->surfaceList[frame_meta->batch_id].width, CV_8UC3,
nvbuf_copy->surfaceList[frame_meta->batch_id].mappedAddr.addr[0],
nvbuf_copy->surfaceList[frame_meta->batch_id].pitch);

is there any way to copy directly from in_surf? The respective in_surf->surfaceList[frame_meta->batch_id].mappedAddr.addr[0] was null in my case and mapping using a preceding NvBufSurfaceMap did not work.

2. How would I populate the inf_buf object with the crop Mat data? The suggested solution from the mentioned thread did not work as NvBufSurfaceSyncForCpu returns -1.

[Fiona.Chen]

@twangbarang

Is there a way to copy the data without this intermediate nvbuf_copy? Meaning in this part

Depends on what you want to do. nvinfer input surfaces are either Nvidia NV12 format or RGBA format. I don’t understand why you want to read the data in this way.

@twangbarang

2. How would I populate the inf_buf object with the crop Mat data? The suggested solution from the mentioned thread did not work as NvBufSurfaceSyncForCpu returns -1.

Do you mean you want to replace the content of ‘in_buf’ with the cropped video? If so, it is a big mistake. nvinfer works in ‘in-place’ mode, the input buffer is passed to downstream plugins as it is. You can only populate the same resolution and same format content in ‘in_buf’, or else, the will be unpredictable error happen to the pipeline.

Why do you want to replace the content of ‘in_buf’?

[twangbarang]

Please follow this thread to get the idea of what I actually want to achieve.

I pass in a frame with n bboxes/object_metas. For each of those bboxes I want to create another NvBufSurface or finally GstInferFrame because I want to execute inference on those frames, not on the original ones.

Hope this is somehow clear? If not i’ll elaborate.

[Fiona.Chen]

nvinfer has already implemented bbox cropping and resize, you don’t need to add the same function in this place, it will mess the flow. Please don’t touch ‘in_buf’ since you only want to do preprocessing for inference.

The crop and resize happens inside convert_batch_and_push_to_input_thread(), and please read the code carefully to make sure you know the correct data flow.

Currently we don’t have sample for how to add new pre-process in nvinfer, only a simple introduction of nvinfer flow. DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

[twangbarang]

Please note the buffer I created is named inf_buf, not in_buf. I’m not trying to alter this last one.

What I want, I guess thats the way to achieve what I’m seeking for, is to change

GstInferFrame frame;
frame.converted_frame_ptr = memory->frame_memory_ptrs[idx];
frame.scale_ratio_x = scale_ratio_x;
frame.scale_ratio_y = scale_ratio_y;
frame.obj_meta = (nvinfer->classifier_async_mode) ? nullptr : object_meta;
frame.frame_meta = frame_meta;
frame.frame_num = frame_num;
frame.batch_index = frame_meta->batch_id;
frame.history = obj_history;
frame.input_surf_params = (nvinfer->classifier_async_mode)
							? nullptr
							: (in_surf->surfaceList + frame_meta->batch_id);
batch->frames.push_back(frame);

to take the buffer from cv::Mat mat with additional adapted meta data, so that the buffer lying in mat is used for inference and not the buffer in in_buf and the tensor created from the inference is attached to the original object_meta.

nvinfer has already implemented bbox cropping and resize, you don’t need to add the same function in this place, it will mess the flow. Please don’t touch ‘in_buf’ since you only want to do preprocessing for inference.

As far as I read and understood the cropping and resizing is not capable of the affine transformation I want to apply to the bbox as you can find in the linked thread.

[twangbarang]

So the questions remains: How could I create an NvBufSurface object from an cv::Mat?

[Fiona.Chen]

@twangbarang

How could I create an NvBufSurface object from an cv::Mat ?

You can not create a NvBufeSurface from a cv::Mat. NvBufSurface is a HW buffer(or you can say is a GPU buffer).

[twangbarang]

@Fiona.Chen

@twangbarang

How could I create an NvBufSurface object from an cv::Mat ?

You can not create a NvBufeSurface from a cv::Mat . NvBufSurface is a HW buffer(or you can say is a GPU buffer).

But isn’t it what people did in this thread? I mean it has to be possible to populate a NvBufSurface with custom data.

[Fiona.Chen]

Please read the code. The NvBufSurface is created with NvBufSurfaceCreateParams， memory type is NVBUF_MEM_DEFAULT. The buffer is not created from opecv.

The NvBufSurface can be mapped and read by CPU.

[twangbarang]

So I’m having this now:

NvBufSurface *inf_buf = nullptr;
NvBufSurfaceCreateParams create_params2;
create_params2.gpuId = in_surf->gpuId;
create_params2.width = 112;
create_params2.height = 112;
create_params2.size = 0;
create_params2.colorFormat = NVBUF_COLOR_FORMAT_BGR;
create_params2.layout = NVBUF_LAYOUT_PITCH;
create_params2.memType = NVBUF_MEM_DEFAULT;
if (NvBufSurfaceCreate(&inf_buf, 1, &create_params2) != 0) {
GST_ELEMENT_ERROR(nvinfer, STREAM, FAILED, (“Failed creating inf_buf”), (NULL));
return GST_FLOW_ERROR;
}

// cf.
// OpenCV Mat to NvBufSurface (to use in NvBufSurfTransform) - #13 by DaneLLL
inf_buf->numFilled = 1;

NvBufSurfaceMemSet(inf_buf, 0, 0, 0);

auto map_err = NvBufSurfaceMap(inf_buf, 0, 0, NVBUF_MAP_READ_WRITE);
if (map_err != 0) {
GST_ELEMENT_ERROR(nvinfer, STREAM, FAILED, (“Failed mapping buf inf_buf: %d\n”, map_err),
(NULL));
return GST_FLOW_ERROR;
}

which unfortunately does not work as NvBufSurfaceMap returns -1. NvBufSurfaceMap yields

nvbufsurface: mapping of memory type (0) not supported

@Fiona.Chen

The buffer is not created from opecv.

I just want to create an NvBufSurface which is populated by an cv::Mat like in the linked thread but when following this one I got the upper error.

Do you have any other idea why this is not working?

[Fiona.Chen]

@twangbarang There is a complete sample of how to use CPU to read NvVufSurface. Please refer to the sample: DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums