I’m implementing a pipeline structure as follows:
modelA(face detection)->(postprocess for A+ preprocess for B)face alignment-> modelB(face recognition)
Among them, the landmarks of model A are saved in nvdsusermeta, and the main function of face alignment is to affine transform the detected region.
Question:
How can I modify the gst-nvinfer plug-in to achieve face alignment?
Depends on what you want to do and what you want to output with face detection and face alignment. There may be a lot of details. Nvinfer is totally open source. You can investigate the work flow and understand the functions to make your decision.
How to read GST-nvinfer plug-in source code, whether there are related reference samples?
The code is there, investigate with your knowledge for c/c++, cuda, gstreamer, … There is just a simple introduction DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums.
I checked the gstnvinfer.cpp.
It is found that the preprocessing of the image is in the function get_ converted_ buffer, but I can’t find the logic of resize. How can I use opencv for image processing?
static GstFlowReturn
get_converted_buffer (GstNvInfer * nvinfer, NvBufSurface * src_surf,
NvBufSurfaceParams * src_frame, NvOSD_RectParams * crop_rect_params,
NvBufSurface * dest_surf, NvBufSurfaceParams * dest_frame,
gdouble & ratio_x, gdouble & ratio_y, void *destCudaPtr)
{
guint src_left = GST_ROUND_UP_2 ((unsigned int)crop_rect_params->left);
guint src_top = GST_ROUND_UP_2 ((unsigned int)crop_rect_params->top);
guint src_width = GST_ROUND_DOWN_2 ((unsigned int)crop_rect_params->width);
guint src_height = GST_ROUND_DOWN_2 ((unsigned int)crop_rect_params->height);
guint dest_width, dest_height;
if (nvinfer->maintain_aspect_ratio) {
/* Calculate the destination width and height required to maintain
* the aspect ratio. */
double hdest = dest_frame->width * src_height / (double) src_width;
double wdest = dest_frame->height * src_width / (double) src_height;
int pixel_size;
cudaError_t cudaReturn;
if (hdest <= dest_frame->height) {
dest_width = dest_frame->width;
dest_height = hdest;
} else {
dest_width = wdest;
dest_height = dest_frame->height;
}
switch (dest_frame->colorFormat) {
case NVBUF_COLOR_FORMAT_RGBA:
pixel_size = 4;
break;
case NVBUF_COLOR_FORMAT_RGB: //RGB
pixel_size = 3;////////////////////////////
break;
case NVBUF_COLOR_FORMAT_GRAY8:
case NVBUF_COLOR_FORMAT_NV12:
pixel_size = 1;
break;
default:
g_assert_not_reached ();
break;
}
/* Pad the scaled image with black color. */
cudaReturn =
cudaMemset2DAsync ((uint8_t *) destCudaPtr + pixel_size * dest_width,
dest_frame->planeParams.pitch[0], 0,
pixel_size * (dest_frame->width - dest_width), dest_frame->height,
nvinfer->convertStream);
if (cudaReturn != cudaSuccess) {
GST_ERROR_OBJECT (nvinfer,
"cudaMemset2DAsync failed with error %s while converting buffer",
cudaGetErrorName (cudaReturn));
return GST_FLOW_ERROR;
}
cudaReturn =
cudaMemset2DAsync ((uint8_t *) destCudaPtr +
dest_frame->planeParams.pitch[0] * dest_height,
dest_frame->planeParams.pitch[0], 0, pixel_size * dest_width,
dest_frame->height - dest_height, nvinfer->convertStream);
if (cudaReturn != cudaSuccess) {
GST_ERROR_OBJECT (nvinfer,
"cudaMemset2DAsync failed with error %s while converting buffer",
cudaGetErrorName (cudaReturn));
return GST_FLOW_ERROR;
}
} else {
dest_width = nvinfer->network_width;
dest_height = nvinfer->network_height;
}
/* Calculate the scaling ratio of the frame / object crop. This will be
* required later for rescaling the detector output boxes to input resolution.
*/
ratio_x = (double) dest_width / src_width;
ratio_y = (double) dest_height / src_height;
/* Create temporary src and dest surfaces for NvBufSurfTransform API. */
nvinfer->tmp_surf.surfaceList[nvinfer->tmp_surf.numFilled] = *src_frame;
/* Set the source ROI. Could be entire frame or an object. */
nvinfer->transform_params.src_rect[nvinfer->tmp_surf.numFilled] =
{src_top, src_left, src_width, src_height};
/* Set the dest ROI. Could be the entire destination frame or part of it to
* maintain aspect ratio. */
nvinfer->transform_params.dst_rect[nvinfer->tmp_surf.numFilled] =
{0, 0, dest_width, dest_height};
nvinfer->tmp_surf.numFilled++;
return GST_FLOW_OK;
}
The resize tramsformation is done in convert_batch_and_push_to_input_thread() with NvBufSurfTransform(). The transformed result is in batch. The batch will be handled in another thread gst_nvinfer_input_queue_loop().
Can you provide the source code of NvBufSurfTransform, I hope to refer to,thanks!
Do you mean the implementation inside NvBufSurfTransform?
yes
What is the relationship between nvinfer->tmp_surf , mem->surf and batch ?
What should I do if I want to change the functionality of the NvBufSurfTransform function?
I just want to replace the original Transform result
- NvBufSurfTransform implementation is Nvidia proprietary. I can not should you the code. But in /opt/nvidia/deepstream/deepstream/sources/includes/nvbufsurftransform.h, the comments has explained the purpose and usage of this function.
It is clear in the code. NvBufSurfTransform can do image conversion on batched images. the nvinfer->tmp_surf is the src of the conversion, mem->surf is the dest of the conversion. batch is the container of mem->surf.
If I want to modify the conversion result, how can I get and replace the original content from batch or men?
I tried to convert mem->surf to nvbufsurf, but failed
mem->surf is already nvbufsurface, what do you mean by convert mem->surf to nvbufsurf?
NvBufSurface *surface = memory->surf;
NvBufSurfaceMap (surface, -1, -1, NVBUF_MAP_READ);
NvBufSurfaceSyncForCpu (surface, 0, 0);
std::vectorcv::Mat frames_temp;
for (size_t i = 0; i < batch_size; i++)
{
// NvBufSurfaceParams frameHandle = temp_surface->surfaceList[i];
cv::Mat tmp;
int height = surface->surfaceList[i].height;
int width = surface->surfaceList[i].width;
bool color_format_supported = true;
switch(surface->surfaceList[i].colorFormat)
{
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_BGRA:
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_BGRx:
tmp = cv::Mat(height, width, CV_8UC4, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_BGRA2BGR);
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_RGBA:
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_RGBx:
tmp = cv::Mat(height, width, CV_8UC4, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_RGBA2BGR);
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_RGB:
tmp = cv::Mat(height, width, CV_8UC3, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_RGB2BGR);
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_BGR:
tmp = cv::Mat(height, width, CV_8UC3, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
frames_temp[i] = tmp.clone();
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_GRAY8:
tmp = cv::Mat(height, width, CV_8UC1, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_GRAY2BGR);
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_NV12:
tmp = cv::Mat(height * 3 / 2, width, CV_8UC1, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_YUV2BGR_NV12);
break;
default:
g_print(“unsupport color format tosave :%d in source:%d frame:%d”, surface->surfaceList[i].colorFormat, i, i);
frames_temp.clear();
// return false;
}
// cv::Mat outMat, alignedFace, alignedFaceRGBA;
// cv::cvtColor(inMat, outMat, cv::COLOR_RGBA2RGB);
}
for (size_t i = 0; i < frames_temp.size(); i++)
{
cv::imwrite(std::to_string(i) + “.jpg”, frames_temp[i]);
}
When I execute this code to extract the image from memory, I will report an error
nvbufsurface: mapping of memory type (0) not supported
terminate called after throwing an instance of ‘cv::Exception’
what(): OpenCV(3.4.12) /usr/local/include/opencv2/core/mat.inl.hpp:591: error: (-215:Assertion failed) total() == 0 || data != NULL in function ‘Mat’
Aborted (core dumped)
How can I retrieve image data from memory
I have wrote g_object_set(G_OBJECT(nvvidconv), “nvbuf-memory-type”, 3, NULL);
Can I copy images from GPU to CPU cv::Mat by cudaMemcpy2DAsync, but how can I copy them?
Just like this code:
/* Pad the scaled image with black color. */
cudaReturn =
cudaMemset2DAsync ((uint8_t *) destCudaPtr + pixel_size * dest_width,
dest_frame->planeParams.pitch[0], 0,
pixel_size * (dest_frame->width - dest_width), dest_frame->height,
nvinfer->convertStream);