Deepstream plugin to crop frame

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

• Hardware Platform (Jetson / GPU) Jetson Nano
• DeepStream Version 6.0
• JetPack Version (valid for Jetson only) 4.6.1 (Linux 32.7.1)
• TensorRT Version 8.2.1.9
• Issue Type( questions, new requirements, bugs) Error

I am developing a plugin that crops a frame (i know i can use the nvvideoconvert plugin).

I am trying to perform NvBufSurfTransform to crop the input frame, the return value of the operation is 0 so as it says in the documentation the result is correct, however now i am facing another deal, ¿ this new surface (output_surface in my case) is the one that is going through the pipeline or i have to copy to a output buffer? .

I first started using the transform_ip which only has input buffer function of gstbasetransform_class but i saw a topic that recommend to use the transform function that has both input and output buffer. I am trying to memcopy this output_surface on the out_map_info.data that has been maped to the buffer.

The code goes like this:

if (NvBufSurfTransform(input_surface, output_surface, &transform_params) != 0) {
        g_print("Error: NvBufSurfTransform failed.");
        NvBufSurfaceDestroy(output_surface);
        gst_buffer_unmap(inbuf, &in_map_info);
        return GST_FLOW_ERROR;
    }else{
        GST_DEBUG_OBJECT (dscustomplugin, "\n-----TRANSFORM PERFORMED PERFECTLY-----");
    }

    // Mapear el buffer de salida
    if (!gst_buffer_map(outbuf, &out_map_info, GST_MAP_WRITE)) {
        GST_ERROR_OBJECT(dscustomplugin, "\nError: Failed to map output buffer");
        NvBufSurfaceDestroy(output_surface);
        gst_buffer_unmap(inbuf, &in_map_info);
        return GST_FLOW_ERROR;
    }else{
        GST_DEBUG_OBJECT (dscustomplugin, "\n-----OUTPUT BUFFER MAPPED-----");
    }

    memcpy(out_map_info.data, output_surface, sizeof(NvBufSurface));

When i run this pipeline i have the following error: 0x55826ea680 ERROR nvvideoconvert gstnvvideoconvert.c:3488:gst_nvvideoconvert_transform: buffer transform failed

gst-launch-1.0 nvarguscamerasrc bufapi-version=true ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)1920, height=(int)1080, framerate=(fraction)30/1" ! dscustomplugin ! nvvideoconvert ! nvv4l2h264enc bitrate=4000000 ! h264parse ! rtph264pay ! udpsink host=172.25.61.8 port=5000

What i am supposed to do if i want the plugin to perform a crop that is a region of the input frame?

3

Hi,

You’re on the right track with NvBufSurfTransform for cropping, but there are a few points to clarify and adjust in your implementation:

  • Handling NvBufSurfTransform Output: The NvBufSurfTransform output is a structure on the GPU, and you need to map it properly. You can either download the data from the GPU into a CPU buffer or use prepare_output_buffer to correctly allocate an NVMM buffer, get the surface from it, and pass it to NvBufSurfTransform as an argument.
  • Mapping NvBufSurface to GStreamer Buffers: When using NvBufSurface, you can directly map and use the memory if both buffers are in NVMM. To access and copy the data, you can use NvBufSurfaceMap to map the output buffer’s data.
  • transform_ip vs transform: It’s recommended to use transform when you need both input and output buffers, while transform_ip is for in-place modifications where the input and output buffers have the same caps.
4

You can refer to the above comment. But why don’t you consider using the nvvideoconvert plugin?

5

Thank you for the answers, however the following questions came up:

  • From the first point, do I have to call NvBufSurfTransform twice as you mentioned? First to get an output from the input surface and then allocate a buffer and pass the first output to NvBufSurfTransform as an argument?
  • Both buffers should be on NVMM as those are the caps for the plugin and i am using now transform function which has input buffer and output buffer, is this output buffer allocated anyway?

Thankyou yuweiw for your answer as well, i am not considering nvvideconvert as i want to perform the crop in the plugin

This is what i am trying now:

// Apply Nvbufsurftransform for cropping
    if (NvBufSurfTransform(input_surface, output_surface, &transform_params) != 0) {
        GST_DEBUG_OBJECT (dscustomplugin, "-----TRANSFORM FAILED-----");
        g_print("Error: NvBufSurfTransform failed.");
        NvBufSurfaceDestroy(output_surface);
        gst_buffer_unmap(inbuf, &in_map_info);
        return GST_FLOW_ERROR;
    }else{
        GST_DEBUG_OBJECT (dscustomplugin, "\n-----TRANSFORM PERFORMED PERFECTLY-----");
    }
 
    // Map output buffer
    if (!gst_buffer_map(outbuf, &out_map_info, GST_MAP_WRITE)) {
        GST_ERROR_OBJECT(dscustomplugin, "\nError: Failed to map output buffer");
        NvBufSurfaceDestroy(output_surface);
        gst_buffer_unmap(inbuf, &in_map_info);
        return GST_FLOW_ERROR;
    }else{
        GST_DEBUG_OBJECT (dscustomplugin, "\n-----OUTPUT BUFFER MAPPED-----");
    }
 
    if (NvBufSurfaceMap (output_surface, 0, 0, NVBUF_MAP_WRITE) != 0){
        GST_ERROR_OBJECT(dscustomplugin, "\nError: Failed to AAAAAA map output buffer");
        NvBufSurfaceDestroy(output_surface);
        gst_buffer_unmap(inbuf, &in_map_info);
        return GST_FLOW_ERROR;
    }
6

From the first point, do I have to call NvBufSurfTransform twice as you mentioned? First to get an output from the input surface and then allocate a buffer and pass the first output to NvBufSurfTransform as an argument?

You only need to call NvBufSurfTransform once, but you need to set the right parameters and pass the correct NvBufSurface as parameters (obtained by mapping the NVMM buffer data):

NvBufSurfTransform (NvBufSurface *src, NvBufSurface *dst, NvBufSurfTransformParams *transform_params)

Both buffers should be on NVMM as those are the caps for the plugin and i am using now transform function which has input buffer and output buffer, is this output buffer allocated anyway?

GStreamer doesn’t know how to allocate NVMM. Even if you set it on the caps, the output buffer will not be NVMM. If the element is calling process_ip, this isn’t an issue because it just reuses the input buffer as the output. However, if the element is calling process and you need NVMM as the output, you must implement prepare_output_buffer to allocate the NVMM correctly and pass the allocated pointer to process. check these for the allocation:

7

Would you mind posting a boilerplate code that does all of this, i am very lost with so much functions.

Thank you

8

Unfortunately, I can’t share the code as it is part of RidgeRun’s GstCUDA project, which is not open-source. However, I’d be happy to assist with any specific questions or guide you through particular aspects of the process.

Additionally, I noticed you’re using JetPack 4.6.1… I think the NvBufSurface API was introduced in JetPack 5. In older JetPack versions, we used ExtractFdFromNvBuffer instead of the NvBufSurface structure.