Offsetting NVMM RGBA Image using VPI and DeepStream

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.0
• VPI Version 1.2

I did some changes on dsexample element to apply perspective warp on specific parts of the input frames using VPI. I got inspiration from this post.

Here’s the pipeline I am running:

gst-launch-1.0 filesrc location=/sample_video.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! muxer.sink_0 nvstreammux name=muxer width=1280 height=720 batch-size=1 !  nvvideoconvert !  "video/x-raw(memory:NVMM), format=RGBA"  ! dsexample ! nvvideoconvert ! nvdsosd ! nveglglessink sync=1

And here’s a snippet from transform_ip function in dsexample:

  GstDsExample *dsexample = GST_DSEXAMPLE (btrans);
  GstMapInfo in_map_info;
  GstFlowReturn flow_ret = GST_FLOW_ERROR;

  NvBufSurface *surface = NULL;
  int idx = 0;

  VPIImageData img_data_src;
  int width, height;
  cv::Mat outFrame;
  std::string vpi_status_str;

  VPIPerspectiveTransform h1 = {{1.0, 0.0, 0.0}, {0.0, 1.0, 0.0}, {0.0005, 0.0003, 1.0}};
  dsexample->frame_num++;
  CHECK_CUDA_STATUS(cudaSetDevice(dsexample->gpu_id), "Unable to set cuda device");

  memset(&in_map_info, 0, sizeof(in_map_info));
  if (!gst_buffer_map(inbuf, &in_map_info, GST_MAP_READ)) {
    g_print("Error: Failed to map gst buffer\n");
    goto error;
  }

  nvds_set_input_system_timestamp(inbuf, GST_ELEMENT_NAME(dsexample));
  surface = (NvBufSurface *)in_map_info.data;

  width = surface->surfaceList[0].width;
  height = surface->surfaceList[0].height;
  if (!dsexample->vpi_rgba8_out)
    CHECK_STATUS(
        vpiImageCreate(width / 2, height / 2, VPI_IMAGE_FORMAT_RGBA8, VPI_BACKEND_CUDA, &dsexample->vpi_rgba8_out));

  if (!dsexample->vpi_bgr8_out)
    CHECK_STATUS(vpiImageCreate(width / 2, height / 2, VPI_IMAGE_FORMAT_BGRA8, 0, &dsexample->vpi_bgr8_out));

  memset(&img_data_src, 0, sizeof(img_data_src));

  img_data_src.format = VPI_IMAGE_FORMAT_RGBA8;
  img_data_src.numPlanes = surface->surfaceList[idx].planeParams.num_planes;

  for (int32_t j = 0; j < img_data_src.numPlanes; j++) {
    img_data_src.planes[j].width = width / 2;
    img_data_src.planes[j].height = height / 2;
    img_data_src.planes[j].pitchBytes = width * 4;  // state the full stride of image from which you are cropping
    // set arbitrary offset
    int offset = img_data_src.planes[j].pitchBytes * height / 3 + (width / 2) * 4;
    img_data_src.planes[j].data = (uint8_t *)surface->surfaceList[0].dataPtr + offset;
  }

  if (!dsexample->vpi_rgba8_in || !dsexample->vpi_rgba8_out) {
    CHECK_STATUS(vpiImageCreateCUDAMemWrapper(&img_data_src, 0, &dsexample->vpi_rgba8_in));
  } else {
    CHECK_STATUS(vpiImageSetWrappedCUDAMem(dsexample->vpi_rgba8_in, &img_data_src));
  }
  CHECK_STATUS(vpiSubmitPerspectiveWarp(dsexample->stream, 0, dsexample->warp, dsexample->vpi_rgba8_in, h1,
                                        dsexample->vpi_rgba8_out, VPI_INTERP_LINEAR, VPI_BORDER_ZERO, 0));
  CHECK_STATUS(vpiStreamSync(dsexample->stream));

  // Convert output back to BGR using CUDA
  CHECK_STATUS(vpiSubmitConvertImageFormat(dsexample->stream, VPI_BACKEND_CUDA, dsexample->vpi_rgba8_out,
                                           dsexample->vpi_bgr8_out, NULL));
  CHECK_STATUS(vpiStreamSync(dsexample->stream));

  // Write it on disk
  VPIImageData imgdata;
  CHECK_STATUS(vpiImageLock(dsexample->vpi_bgr8_out, VPI_LOCK_READ, &imgdata));
  CHECK_STATUS(vpiImageDataExportOpenCVMat(imgdata, &outFrame));
  cv::imwrite("test.jpg", outFrame);
  CHECK_STATUS(vpiImageUnlock(dsexample->vpi_bgr8_out));

Here I am trying to apply perspective warp on only part of the input frame (a crop) by adding offset to the data pointer and write it on the disk separately. I made sure that input to dsexample is in rgba format so that the offset I am adding would make sense.

The thing is, it only works when I set muxer resolution to (1280 x 720) but when I switch to (1920 x 1080) I get
VPI_ERROR_INTERNAL: cudaErrorInvalidValue: cudaCreateTextureObject(...).

The exception is thrown from vpiStreamSync interface. I should also mention that not all offset values work with (1280 x 720), some offset values (still inside of frame bounds) break it too.

Any idea why I am getting this exception?

Any feedback here?

Sorry for the late response, our team will do the investigation and provide suggestions soon. Thanks

There is already “offset” and “psize” in NvBufSurfacePlaneParams (NVIDIA DeepStream SDK API Reference: NvBufSurfacePlaneParams Struct Reference), you don’t need to calculate

Hi @Fiona.Chen,

I do not wish to apply the offset to the NvBufSurface, I would like to apply this offset only to the VPIImageData which does not have an offset parameter. That’s why I am adding it to the data pointer myself.

Actually you are using deepstream components, so you must use deepstream APIs.

What do you mean “you must”? are you saying that I can’t use VPI APIs in a deepstream plugin?

can you suggest a deepstream API that implements perspective warp? because as far as I can tell VPI is my only option.

You can use both APIs. NvBufSurface has provided the correct offset. You don’t need to calculate by yourself.

As to perspective warp, DeepStream has a plugin for it. Gst-nvdewarper — DeepStream 6.1.1 Release documentation

Perhaps let me try and clarify what I would like to achieve once again. I would like to apply perspective warp on only part/crop of the input frame and write it to the disk separately. I wish to keep the NvBufSurface untouched. I would like to apply the offset to the VPIImageData which does not have an offset param. That is why I am calculating it myself and I am getting an exception for it. Any idea why I am getting this exception?

I have already came across Gst-nvdewarper before, as far as I was able to see from the documentation, there is no option to neither work on crops of the input frame nor write warped frames to the disk.

NvBufSurface is the only interface provided by deepstream to store the image data. If it not compatible to VPI data layout, you need to convert by yourself.

Yeah, thanks. This I what I am already trying to accomplish here, convert it by myself. Back to square one. Any idea why it is not working?

VPI_ERROR_INTERNAL: cudaErrorInvalidValue: cudaCreateTextureObject(...).