Hello everyone,
I’m creating a preprocessing element that performs some opencv operations on the image, and this element will come just before nvinfer. The logic of the element is as follows:
- Obtain a cv::Mat from NvBufSurface
- Perform opencv operations
- Create a new NvBufSurface
- Populate the new surface with the data from the cv::Mat
- Unref the input buffer
The reason I’m discarding the original buffer is because I need to send the original frames along with a few copies of the frame, with some filters applied.
There are examples on how to convert NvBufSurface to cv::Mat (present in dsexample.cpp), but no examples on how to do the reverse, i.e., cv::Mat to NvBufSurface. I have looked at a few topics on the forum that discuss similar issues, like this one and this one, and also looked at the nvbufsurface.h header, and managed to put together this code:
// creating NvBufSurface
NvBufSurfaceCreateParams create_params;
NvBufSurface* surface = NULL;
create_params.gpuId = gpu_id;
create_params.width = width;
create_params.height = height;
create_params.size = 0;
create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
create_params.layout = NVBUF_LAYOUT_PITCH;
#ifdef __aarch64__
create_params.memType = NVBUF_MEM_DEFAULT;
#else
create_params.memType = NVBUF_MEM_CUDA_UNIFIED;
#endif
NvBufSurfaceCreate(&surface, 1, &create_params);
output = gst_buffer_new_wrapped_full((GstMemoryFlags) 0, (gpointer) surface, sizeof(NvBufSurface), 0, sizeof(NvBufSurface), NULL, NULL);
// adding NvDsBatchMeta to new buffer
NvDsBatchMeta* batch_meta = nvds_create_batch_meta(1);
NvDsMeta* meta = gst_buffer_add_nvds_meta(*output, batch_meta, NULL, nvds_batch_meta_copy_func, nvds_batch_meta_release_func);
meta->meta_type = NVDS_BATCH_GST_META;
batch_meta->base_meta.batch_meta = batch_meta;
batch_meta->base_meta.copy_func = nvds_batch_meta_copy_func;
batch_meta->base_meta.release_func = nvds_batch_meta_release_func;
batch_meta->max_frames_in_batch = 1;
// copy logic:
GstMapInfo in_map_info;
GstMapInfo out_map_info;
NvBufSurface* in_surface;
NvBufSurface* out_surface;
NvDsBatchMeta* out_batch_meta;
NvDsBatchMeta* in_batch_meta;
NvDsFrameMeta* in_frame_meta;
gst_buffer_map(inbuf, &in_map_info, GST_MAP_READ);
gst_buffer_map(outbuf, &out_map_info, GST_MAP_WRITE));
in_surface = (NvBufSurface*) in_map_info.data;
out_surface = (NvBufSurface*) out_map_info.data;
out_batch_meta = gst_buffer_get_nvds_batch_meta(outbuf);
in_batch_meta = gst_buffer_get_nvds_batch_meta(inbuf);
in_frame_meta = nvds_get_nth_frame_meta(in_batch_meta->frame_meta_list, 0);
std::unique_ptr<cv::Mat> cv_img = self->image_converter->get_mat_from_surface(in_surface, 0);
// ...perform opencv operations
// memset memory
if (NvBufSurfaceMemSet(out_surface, 0, 0, 0) != 0) {
GST_ELEMENT_ERROR(self, STREAM, FAILED, ("Failed memset NvBufSurface"), (NULL));
return GST_FLOW_ERROR;
}
// map buffer, since we are using one of NVBUF_MEM_CUDA_UNIFIED, NVBUF_MEM_SURFACE_ARRAY or NVBUF_MEM_HANDLE
if (NvBufSurfaceMap(out_surface, 0, 0, NVBUF_MAP_WRITE) != 0) {
GST_ELEMENT_ERROR(self, STREAM, FAILED, ("Failed to map output NvBufSurface"), (NULL));
return GST_FLOW_ERROR;
}
// sync for CPU if on jetson
if (out_surface->memType == NVBUF_MEM_SURFACE_ARRAY || out_surface->memType == NVBUF_MEM_HANDLE) {
NvBufSurfaceSyncForCpu(out_surface, 0, 0);
}
// convert cv::Mat to RGBA
cv::cvtColor(*cv_img, *cv_img, cv::COLOR_BGR2RGBA);
// copy data
memcpy(out_surface->surfaceList[0].mappedAddr.addr[0], cv_img->ptr(), cv_img->total() * cv_img->elemSize());
out_surface->numFilled = 1;
// sync for device if on jetson
if (out_surface->memType == NVBUF_MEM_SURFACE_ARRAY || out_surface->memType == NVBUF_MEM_HANDLE) {
NvBufSurfaceSyncForDevice(out_surface, 0, 0);
}
// unmap surface
if (NvBufSurfaceUnMap(out_surface, 0, 0) != 0) {
GST_ELEMENT_ERROR(self, STREAM, FAILED, ("Failed to unmap output NvBufSurface"), (NULL));
return GST_FLOW_ERROR;
}
// adding NvDsFrameMeta to the batch meta
NvDsFrameMeta* frame_meta = nvds_acquire_frame_meta_from_pool(out_batch_meta);
nvds_add_frame_meta_to_batch(out_batch_meta, frame_meta);
frame_meta->pad_index = 0;
frame_meta->source_id = 0;
frame_meta->buf_pts = 0;
frame_meta->ntp_timestamp = 0;
frame_meta->frame_num = 0;
frame_meta->batch_id = 0;
frame_meta->source_frame_width = 640;
frame_meta->source_frame_height = 480;
frame_meta->num_surfaces_per_frame = 1;
But this does not work. It crashes immediately, with this stack trace:
Error: signal 11:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7ff1d18d4980]
/lib/x86_64-linux-gnu/libc.so.6(+0x18ea5f)[0x7ff1d1447a5f]
/usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0(+0x42401)[0x7ff1cfaea401]
/usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0(+0x41b84)[0x7ff1cfae9b84]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0(+0x7688b)[0x7ff1d20c288b]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0(gst_pad_push+0x103)[0x7ff1d20cabb3]
/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so(+0x63120)[0x7ff1bdb47120]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0(+0xab269)[0x7ff1d20f7269]
It looks like something in libnvdsgst_multistream.so is crashing. Also, I have an element that saves the input image, and this element comes just after nvinfer. It gets triggered, but the input image which it saves looks corrupted. So I have no idea whats going on.
I would appreciate some guidance on this issue.