Sgie custom preprocessing

I’m implementing a pipeline structure as follows:
modelA(face detection)->(postprocess for A+ preprocess for B)face alignment-> modelB(face recognition)

Among them, the landmarks of model A are saved in nvdsusermeta, and the main function of face alignment is to affine transform the detected region.

Question:
How can I modify the gst-nvinfer plug-in to achieve face alignment?

Depends on what you want to do and what you want to output with face detection and face alignment. There may be a lot of details. Nvinfer is totally open source. You can investigate the work flow and understand the functions to make your decision.

How to read GST-nvinfer plug-in source code, whether there are related reference samples?

The code is there, investigate with your knowledge for c/c++, cuda, gstreamer, … There is just a simple introduction DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums.

I checked the gstnvinfer.cpp.
It is found that the preprocessing of the image is in the function get_ converted_ buffer, but I can’t find the logic of resize. How can I use opencv for image processing?

static GstFlowReturn
get_converted_buffer (GstNvInfer * nvinfer, NvBufSurface * src_surf,
    NvBufSurfaceParams * src_frame, NvOSD_RectParams * crop_rect_params,
    NvBufSurface * dest_surf, NvBufSurfaceParams * dest_frame,
    gdouble & ratio_x, gdouble & ratio_y, void *destCudaPtr)
{
  guint src_left = GST_ROUND_UP_2 ((unsigned int)crop_rect_params->left);
  guint src_top = GST_ROUND_UP_2 ((unsigned int)crop_rect_params->top);
  guint src_width = GST_ROUND_DOWN_2 ((unsigned int)crop_rect_params->width);
  guint src_height = GST_ROUND_DOWN_2 ((unsigned int)crop_rect_params->height);
  guint dest_width, dest_height;

  if (nvinfer->maintain_aspect_ratio) { 
    /* Calculate the destination width and height required to maintain
     * the aspect ratio. */ 
    double hdest = dest_frame->width * src_height / (double) src_width;
    double wdest = dest_frame->height * src_width / (double) src_height;
    int pixel_size;
    cudaError_t cudaReturn;
    if (hdest <= dest_frame->height) {
      dest_width = dest_frame->width;
      dest_height = hdest;
    } else {
      dest_width = wdest;
      dest_height = dest_frame->height;
    }

    switch (dest_frame->colorFormat) {
      case NVBUF_COLOR_FORMAT_RGBA:
        pixel_size = 4;
        break;
      case NVBUF_COLOR_FORMAT_RGB: //RGB
        pixel_size = 3;////////////////////////////
        break;
      case NVBUF_COLOR_FORMAT_GRAY8:
      case NVBUF_COLOR_FORMAT_NV12:
        pixel_size = 1;
        break;
      default:
        g_assert_not_reached ();
        break;
    }

    /* Pad the scaled image with black color. */ 
    cudaReturn =
        cudaMemset2DAsync ((uint8_t *) destCudaPtr + pixel_size * dest_width,
        dest_frame->planeParams.pitch[0], 0,
        pixel_size * (dest_frame->width - dest_width), dest_frame->height,
        nvinfer->convertStream);
    if (cudaReturn != cudaSuccess) {
      GST_ERROR_OBJECT (nvinfer,
          "cudaMemset2DAsync failed with error %s while converting buffer",
          cudaGetErrorName (cudaReturn));
      return GST_FLOW_ERROR;
    }
    cudaReturn =
        cudaMemset2DAsync ((uint8_t *) destCudaPtr +
        dest_frame->planeParams.pitch[0] * dest_height,
        dest_frame->planeParams.pitch[0], 0, pixel_size * dest_width,
        dest_frame->height - dest_height, nvinfer->convertStream);
    if (cudaReturn != cudaSuccess) {
      GST_ERROR_OBJECT (nvinfer,
          "cudaMemset2DAsync failed with error %s while converting buffer",
          cudaGetErrorName (cudaReturn));
      return GST_FLOW_ERROR;
    }
  } else {
    dest_width = nvinfer->network_width;
    dest_height = nvinfer->network_height;
  }
  /* Calculate the scaling ratio of the frame / object crop. This will be
   * required later for rescaling the detector output boxes to input resolution.
   */
  ratio_x = (double) dest_width / src_width;
  ratio_y = (double) dest_height / src_height;

  /* Create temporary src and dest surfaces for NvBufSurfTransform API. */
  nvinfer->tmp_surf.surfaceList[nvinfer->tmp_surf.numFilled] = *src_frame;

  /* Set the source ROI. Could be entire frame or an object. */
  nvinfer->transform_params.src_rect[nvinfer->tmp_surf.numFilled] =
      {src_top, src_left, src_width, src_height};
  /* Set the dest ROI. Could be the entire destination frame or part of it to
   * maintain aspect ratio. */
  nvinfer->transform_params.dst_rect[nvinfer->tmp_surf.numFilled] =
      {0, 0, dest_width, dest_height};

  nvinfer->tmp_surf.numFilled++;

  return GST_FLOW_OK;
}

The resize tramsformation is done in convert_batch_and_push_to_input_thread() with NvBufSurfTransform(). The transformed result is in batch. The batch will be handled in another thread gst_nvinfer_input_queue_loop().

Can you provide the source code of NvBufSurfTransform, I hope to refer to,thanks!

Do you mean the implementation inside NvBufSurfTransform?

yes
What is the relationship between nvinfer->tmp_surf , mem->surf and batch ?
What should I do if I want to change the functionality of the NvBufSurfTransform function?
I just want to replace the original Transform result

  1. NvBufSurfTransform implementation is Nvidia proprietary. I can not should you the code. But in /opt/nvidia/deepstream/deepstream/sources/includes/nvbufsurftransform.h, the comments has explained the purpose and usage of this function.

It is clear in the code. NvBufSurfTransform can do image conversion on batched images. the nvinfer->tmp_surf is the src of the conversion, mem->surf is the dest of the conversion. batch is the container of mem->surf.

If I want to modify the conversion result, how can I get and replace the original content from batch or men?
I tried to convert mem->surf to nvbufsurf, but failed

mem->surf is already nvbufsurface, what do you mean by convert mem->surf to nvbufsurf?

NvBufSurface *surface = memory->surf;
NvBufSurfaceMap (surface, -1, -1, NVBUF_MAP_READ);
NvBufSurfaceSyncForCpu (surface, 0, 0);
std::vectorcv::Mat frames_temp;
for (size_t i = 0; i < batch_size; i++)
{
// NvBufSurfaceParams frameHandle = temp_surface->surfaceList[i];
cv::Mat tmp;
int height = surface->surfaceList[i].height;
int width = surface->surfaceList[i].width;
bool color_format_supported = true;
switch(surface->surfaceList[i].colorFormat)
{
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_BGRA:
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_BGRx:
tmp = cv::Mat(height, width, CV_8UC4, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_BGRA2BGR);
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_RGBA:
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_RGBx:
tmp = cv::Mat(height, width, CV_8UC4, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_RGBA2BGR);
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_RGB:
tmp = cv::Mat(height, width, CV_8UC3, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_RGB2BGR);
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_BGR:
tmp = cv::Mat(height, width, CV_8UC3, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
frames_temp[i] = tmp.clone();
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_GRAY8:
tmp = cv::Mat(height, width, CV_8UC1, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_GRAY2BGR);
break;
case NvBufSurfaceColorFormat::NVBUF_COLOR_FORMAT_NV12:
tmp = cv::Mat(height * 3 / 2, width, CV_8UC1, surface->surfaceList[i].mappedAddr.addr[0], surface->surfaceList[i].pitch);
cv::cvtColor(tmp, frames_temp[i], CV_YUV2BGR_NV12);
break;
default:
g_print(“unsupport color format tosave :%d in source:%d frame:%d”, surface->surfaceList[i].colorFormat, i, i);
frames_temp.clear();
// return false;
}
// cv::Mat outMat, alignedFace, alignedFaceRGBA;
// cv::cvtColor(inMat, outMat, cv::COLOR_RGBA2RGB);
}
for (size_t i = 0; i < frames_temp.size(); i++)
{
cv::imwrite(std::to_string(i) + “.jpg”, frames_temp[i]);
}

When I execute this code to extract the image from memory, I will report an error

nvbufsurface: mapping of memory type (0) not supported
terminate called after throwing an instance of ‘cv::Exception’
what(): OpenCV(3.4.12) /usr/local/include/opencv2/core/mat.inl.hpp:591: error: (-215:Assertion failed) total() == 0 || data != NULL in function ‘Mat’

Aborted (core dumped)

How can I retrieve image data from memory
I have wrote g_object_set(G_OBJECT(nvvidconv), “nvbuf-memory-type”, 3, NULL);

Can I copy images from GPU to CPU cv::Mat by cudaMemcpy2DAsync, but how can I copy them?
Just like this code:

/* Pad the scaled image with black color. */ 
cudaReturn =
    cudaMemset2DAsync ((uint8_t *) destCudaPtr + pixel_size * dest_width,
    dest_frame->planeParams.pitch[0], 0,
    pixel_size * (dest_frame->width - dest_width), dest_frame->height,
    nvinfer->convertStream);

There is a sample of how to read NvBufSurface with CPU, please refer to the sample DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums