Why I use NvBufSurfTransformCompositeBlend function got VIC operation is not supported error?

[iedusicong]

Hi, I am using NvBufSurfTransformCompositeBlend function, but got the error: "VIC operation is not supported
". So I want to verify is the function NvBufSurfTransformCompositeBlend not supported on VIC ? or other reasons.
https://docs.nvidia.com/jetson/archives/r35.4.1/ApiReference/group__ds__bbb.html#ga30737c04cf7ebf543ec4c10f9cb3b3ff

Now my NvBufSurface type is:
src0 : NVBUF_COLOR_FORMAT_RGBA
src1 : NVBUF_COLOR_FORMAT_RGBA
alpha: NVBUF_COLOR_FORMAT_GRAY8

My jetpack is 5.1.2. Thanks for your help, and my codes are shown as below:

void CameraStitcher::CompositeImage() {
   // get NvBufSurface from cv::Mat
   NvBufSurface** composite_imput = new NvBufSurface*[2];
   for (int i = 0; i < 2; i++) {
      NvBufSurfaceCreateParams create_params;
      create_params.gpuId = 0;
      create_params.width = crop_info_.width;
      create_params.height = crop_info_.height;
      create_params.size = 0;
      create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
      create_params.layout = NVBUF_LAYOUT_PITCH;
      create_params.memType = NVBUF_MEM_DEFAULT;

      if (NvBufSurfaceCreate(&composite_imput[i], 1, &create_params) != 0) {
         std::cout<<"Nv surface create error"<<std::endl;
      }
      composite_imput[i]->numFilled = 1;

      if (NvBufSurfaceMap(composite_imput[i], 0, 0, NVBUF_MAP_READ_WRITE) != 0){
         std::cout<<"Nv surface map error"<<std::endl;
      }

      cv::Mat remap_image = cv::imread("test_" + std::to_string(i) + ".png");
      cv::cvtColor(remap_image, remap_image, cv::COLOR_BGR2RGBA);
      memcpy(composite_imput[i]->surfaceList[0].mappedAddr.addr[0], remap_image.ptr(),
         remap_image.total() * remap_image.elemSize());
   }

   // get alpha from CV_8UC1 cv::Mat
   NvBufSurface* alpha; 
   NvBufSurfaceCreateParams mask_params;
   mask_params.gpuId = 0;
   mask_params.width = crop_info_.width;
   mask_params.height = crop_info_.height;
   mask_params.size = 0;
   mask_params.colorFormat = NVBUF_COLOR_FORMAT_GRAY8;
   mask_params.layout = NVBUF_LAYOUT_PITCH;
   mask_params.memType = NVBUF_MEM_DEFAULT;

   if (NvBufSurfaceCreate(&alpha, 1, &mask_params) != 0) {
      std::cout<<"Nv surface create error"<<std::endl;
   }
   alpha->numFilled = 1;

   if (NvBufSurfaceMap(alpha, 0, 0, NVBUF_MAP_READ_WRITE) != 0){
      std::cout<<"Nv surface map error"<<std::endl;
   }
   cv::Mat mask = camera_masks_[0].clone();  // CV_8UC3 type
   memcpy(alpha->surfaceList[0].mappedAddr.addr[0], mask.ptr(),
      mask.total() * mask.elemSize());

   // initialize output
   NvBufSurf::NvCommonAllocateParams input_params = {0};
   input_params.width = crop_info_.width;
   input_params.height = crop_info_.height;
   input_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
   input_params.layout = NVBUF_LAYOUT_PITCH;
   input_params.memType = NVBUF_MEM_DEFAULT;

   int composited_frame = 0;
   if(-1 == NvBufSurf::NvAllocate(&input_params, 1, &composited_frame)) {
      std::cout<<"NvAllocate error"<<std::endl;
   }
   if (!composited_frame) {
      std::cout<<"Failed to allocate composited buffer"<<std::endl;
   }

   NvBufSurface *output;
   if (-1 == NvBufSurfaceFromFd(composited_frame, (void**)(&output))) {
      std::cout<<"Failed to get output"<<std::endl;
   }

   // get params
   NvBufSurfTransformCompositeBlendParams composite_param;
   memset(&composite_param, 0, sizeof(composite_param));
   composite_param.composite_blend_flag = NVBUFSURF_TRANSFORM_BLEND;
   composite_param.input_buf_count = 2;
   composite_param.composite_blend_filter = NvBufSurfTransformInter_Bilinear;

   int comp_value =NvBufSurfTransformCompositeBlend(composite_imput[0], composite_imput[1], alpha, output, &composite_param);
   std::cout<<"comp_value "<<comp_value<<std::endl;
   return;
}

[DaneLLL]

Hi,
Please share your use-case so that we can check and suggest next. The error you hit is because this NvBufSurface has different format:

alpha: NVBUF_COLOR_FORMAT_GRAY8

[iedusicong]

Hi DaneLLL, thanks for your reply.
My use-case is that I want to stitch two images A and B, and get a new image C. The blending weights for each pixel in image A are stored in matrix D, which is alpha. Matrix A/B/C/D have same width and height. The value for output matrix C is calculated by:
C_value = (A_value * weight + B_value * (255-alpha)) / 255

In my code, matrix A and B are stored in “composite_imput”, matrix D is stored in “alpha”.

[iedusicong]

Hi DaneLLL, I tried to use NVBUF_COLOR_FORMAT_RGBA for alpha, but got the same error: VIC operation is not supported.
The way I construct alpha is that:

 // get alpha from cv::Mat
 NvBufSurface* alpha; 
 NvBufSurfaceCreateParams mask_params;
 mask_params.gpuId = 0;
 mask_params.width = width;
 mask_params.height = height;
 mask_params.size = 0;
 mask_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
 mask_params.layout = NVBUF_LAYOUT_PITCH;
 mask_params.memType = NVBUF_MEM_DEFAULT;

 if (NvBufSurfaceCreate(&alpha, 1, &mask_params) != 0) {
    std::cout<<"Nv surface create error"<<std::endl;
 }
 alpha->numFilled = 1;

 if (NvBufSurfaceMap(alpha, 0, 0, NVBUF_MAP_READ_WRITE) != 0){
    std::cout<<"Nv surface map error"<<std::endl;
 }

 cv::Mat mask1 = camera_masks_[0].clone();  // CV_8UC1 type
 cv::Mat mask2 = camera_masks_[0].clone();  // CV_8UC1 type
 cv::Mat mask3 = camera_masks_[0].clone();  // CV_8UC1 type
 std::vector<cv::Mat> camera_mask_array{mask1, mask2, mask3};
 cv::Mat mask;
 cv::merge(camera_mask_array, mask);           // CV_8UC3 type
 cv::cvtColor(mask, mask, cv::COLOR_BGR2RGBA); 
 memcpy(alpha->surfaceList[0].mappedAddr.addr[0], mask.ptr(),
    mask.total() * mask.elemSize());

[DaneLLL]

Hi,
The function NvBufSurfTransformCompositeBlend() does not support applying different alpha value to each pixel. You would need to implement CUDA code for this step. A possible solution is:

1. Call NvBufSurfTransformCompositeBlend() to stitch A and B NvBufsurface into C NvBufsurface
2. Call NvBufsurface and CUDA APIs to get CUDA pointer of C NvBufsurface
3. Implement CUDA code to apply alpha value to R, B, B channels of C NvBufsurface

Here is a patch for using NvBuffer APIs:
Resources for custom nvivafilter for Jetpack 5.1.1 Jetson NX - #11 by DaneLLL

NvBuffer APIs are replaced with NvBufsurface APIs on Jetpack 5. Although the patch is based on NvBuffer, you can find similar APIs in NvBufsurface.

NvBufSurfTransformCompositeBlend() is demonstrated in 13_argus_multi_camera sample.

[iedusicong]

Hi, Thanks for your reply!
I really want to use VIC to do this thing, because the limited computing power of GPU and CPU.
I want to know if thera a method for matrix multiplication with NvBufSurf on VIC device, like A.mul(B) method in OpenCV. If yes, I can also implement this function on VIC.

[DaneLLL]

Hi,
VIC does not support the function. Please use the OpenCV API, or implement CUDA code.

[iedusicong]

Got it! Thank you very much!