VisionWorks and masking images

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I have the output of a stereo process, and I’d like to mask a rectangular region with all 0’s. The code I’m using to do this is below. However, the mask doesn’t seem to be applied to the image. When I view it in GIMP, the rectangle is not marked. Note that the input image is 1280x960.

vx_uint8 maskedOffPixelValue = 0;
  void* basePtr = NULL;
  vx_imagepatch_addressing_t addr;
  vx_uint32 plane = 0;
  vx_rectangle_t maskRect;
  maskRect.start_x = 0;
  maskRect.start_y = 0;
  maskRect.end_x = 128;
  maskRect.end_y = 960;
  vx_status status = vxAccessImagePatch(inputImage, &maskRect, plane,
                                        &addr, &basePtr,
                                        VX_READ_AND_WRITE);
  for( vx_uint32 y = 0; y < addr.dim_y; y += addr.step_y )
  {
    for( vx_uint32 x = 0; x < addr.dim_x; x += addr.step_x )
    {
      vx_uint8* ptr2 = static_cast<vx_uint8*>(vxFormatImagePatchAddress2d(basePtr, x, y, &addr));
      *ptr2 = maskedOffPixelValue;
    }
  } //end: for( ... )
  status = vxCommitImagePatch(inputImage, &maskRect, plane, &addr, basePtr);

Is there a better way to accomplish this masking (I see that this API is listed as deprecated)? Anything jump out that I’m doing wrong?

2

Hi,

You can pre-define a mask and apply bitwise AND operation to your image.

Please find our document for more information:
[i]------------------------------------------------------

VisionWorks API
NVX CUDA API
NVX CUDA Primitives API[/i]

nvxcu_error_status_e nvxcuBitwiseAnd ( const nvxcu_image_t *       in1,
                                       const nvxcu_image_t *  	   in2,
                                       const nvxcu_image_t *       out,
                                       const nvxcu_exec_target_t * exec_target)

Thanks.

3

Understood, and I could also use the OpenVX version of the call I assume (vxAndNode). But what if I don’t know my input image dimensions until runtime? For example, I know I want to mask off 64 bits from the left of the image, but I don’t know the image height yet. Or, I don’t know exactly how many bits I want to mask off until the loop begins.

I could just use OpenCV calls and do it in CPU, but I was hoping to fit this into the graph somehow and make it part of the CUDA image processing pipeline.

LATE EDIT: I had bugs in my static_cast call, and I switched to the non-deprecated API, but still no luck. Code below. I notice this isn’t part of the graph…how does the pipeline handle non-Node object functions being called?

vx_int16 maskedOffPixelValue = 0;
  void* basePtr = NULL;
  vx_imagepatch_addressing_t addr;
  vx_map_id mapID;
  vx_uint32 plane = 0;
  vx_rectangle_t maskRect;
  maskRect.start_x = 0;
  maskRect.start_y = 0;
  maskRect.end_x = 128;
  maskRect.end_y = 960;
  vx_status status = vxMapImagePatch(image, &maskRect, plane,
                                     &mapID, &addr, &basePtr,
                                     VX_WRITE_ONLY, VX_MEMORY_TYPE_HOST, 0);
  for( vx_uint32 y = 0; y < addr.dim_y; y += addr.step_y )
  {
    for( vx_uint32 x = 0; x < addr.dim_x; x += addr.step_x )
    {
      vx_int16* ptr2 = static_cast<vx_int16*>(vxFormatImagePatchAddress2d(basePtr, x, y, &addr));
      *ptr2 = maskedOffPixelValue;
    }
  } //end: for( ... )
  status = vxUnmapImagePatch(image, mapID);

Here’s the OpenVX example I followed, from their 1.1 Reference Card:

vx_status status = VX_SUCCESS;
  void *base_ptr = NULL;
  vx_uint32 width = 640, height = 480, plane = 0;
  vx_image image = vxCreateImage(context, width, height, VX_DF_IMAGE_U8);
  vx_rectangle_t rect;
  vx_imagepatch_addressing_t addr;
  rect.start_x = rect.start_y = 0;
  rect.end_x = rect.end_y = PATCH_DIM;
  status = vxAccessImagePatch(image, &rect, plane,&addr, &base_ptr, VX_READ_AND_WRITE);
  if (status == VX_SUCCESS)
  {
    vx_uint32 x,y,i,j;
    vx_uint8 pixel = 0;
    /* addressing options */
    /* use linear addressing function/macro */
    for (i = 0; i < addr.dim_x*addr.dim_y; i++) {
      vx_uint8 *ptr2 = vxFormatImagePatchAddress1d(base_ptr, i, &addr);
      *ptr2 = pixel;
    }
     /* 2d addressing option */
    for (y = 0; y < addr.dim_y; y+=addr.step_y) {
      for (x = 0; x < addr.dim_x; x+=addr.step_x) {
        vx_uint8 *ptr2 = vxFormatImagePatchAddress2d(base_ptr, x, y, &addr);
        *ptr2 = pixel;
      }
    }
    /* direct addressing by client. for subsampled planes, scale will change. */
    for (y = 0; y < addr.dim_y; y+=addr.step_y) {
      j = (addr.stride_y*y*addr.scale_y)/VX_SCALE_UNITY;
      for (x = 0; x < addr.dim_x; x+=addr.step_x) {
        vx_uint8 *tmp = (vx_uint8 *)base_ptr;
        i = j + (addr.stride_x*x*addr.scale_x) / VX_SCALE_UNITY;
        tmp[i] = pixel;
      }
    }
    /* commits the data back to the image. If rect were 0 or empty, it would just decrement
     *  the reference (used when reading an image only).
     */
     status = vxCommitImagePatch(image, &rect, plane, &addr, base_ptr);
    }
  vxReleaseImage(&image);
4

Issue #2: Bitwise AND requires U8 images (makes sense), but I’m working on the output of the SGBM algorithm which is in S16 format. I’d rather not down-convert and lose precision, then up-convert again.

I’m trying a different approach, of bitwise multiply on a mask, which is supposed to work with S16 images. The output image is all 0s, however, so clearly not working as I intended.

Here’s the code. Note that maskImage is a U8 image, which I’ve written out and verified is correct. sgbmDisparity is the S16 output of the SGBM disparity node.

double scaleFactor = 1.0;
  vx_scalar scale = vxCreateScalar(_context, VX_TYPE_FLOAT32, &scaleFactor);
  maskNode_ = vxMultiplyNode(
    mainGraph_,
    sgbmDisparity,
    maskImage,
    scale,
    VX_CONVERT_POLICY_SATURATE,
    VX_ROUND_POLICY_TO_NEAREST_EVEN,
    outputImage);
  vxReleaseScalar(&scale);
  NVXIO_CHECK_REFERENCE(maskNode_);
5

OK, got it working. Posting my solution below for posterity, in case anybody else comes across this.

The code that ended up working was the MaskImagePatch code. However, since I’m using the graph and not just direct calls, I had to write a custom node and add it to the graph.

I won’t post all of that here, but the VisionWorks Tutorials for User Custom Nodes are good.

One note is that I’m not sure if this is the most efficient way to do this. It seems to hit the CPU a good amount. But since I’m operating on S16 images, and not U8, the bitwise AND isn’t an option.

Masking code:

vx_int16 maskedOffPixelValue = 0;
      void* basePtr = NULL;
      vx_imagepatch_addressing_t addr;
      vx_map_id mapID;
      vx_uint32 plane = 0;
      vx_rectangle_t maskRect;
      maskRect.start_x = 0;
      maskRect.start_y = 0;
      maskRect.end_x = 128;
      maskRect.end_y = 960;
      vx_status status = vxMapImagePatch(image, &maskRect, plane,
                                         &mapID, &addr, &basePtr,
                                         VX_WRITE_ONLY, VX_MEMORY_TYPE_HOST, 0);
      for( vx_uint32 y = 0; y < addr.dim_y; y += addr.step_y )
      {
        for( vx_uint32 x = 0; x < addr.dim_x; x += addr.step_x )
        {
          vx_int16* ptr2 = static_cast<vx_int16*>(vxFormatImagePatchAddress2d(basePtr, x, y, &addr));
          *ptr2 = maskedOffPixelValue;
        }
      } //end: for( ... )
      status = vxUnmapImagePatch(image, mapID);
6

Thanks for updating with us : )