Problems converting/saving cropped NvBufSurface to JPEG file using ffmpeg

[prominence_ai]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Jetson
• DeepStream Version
6.0.1
• JetPack Version (valid for Jetson only)
4.6.3
• TensorRT Version
8.2.1-1+cuda10.2
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)

I’m having trouble converting a single mapped surface (NvBufSurface) to an encoded JPEG image using ffmpeg.

I have a working solution using opencv that I’ve been using for quite some time, however, having opencv as a build dependency is becoming more trouble than it’s worth. Therefore, I’ve been trying to update my solution to use ffmeg.

My current implementation is very similar to the ds-example, where a single frame or object within the frame is copied to a new NvBufSurface structure with a single surface. This include the steps of

• Setting up the surface-create-params
• Creating the surface
• Setting up the transform-params (left, top, width, height)
• Transforming the surface
• Mapping the new surface for read
• Syncing the surface for cpu access (if mem-type if unified)

Once the surface is ready, my previous opencv implementation would do the following

        // New background Mat for our image
        cv::Mat* pbgrFrame = new cv::Mat(cv::Size(width, height), CV_8UC3);

        // new forground Mat using the first (and only) bufer in the batch
        cv::Mat in_mat = cv::Mat(height, width, CV_8UC4, 
            pBufSurface->surfaceList[0].mappedAddr.addr[0],
            pBufSurface->surfaceList[0].pitch);

        // Convert the RGBA buffer to BGR
        cv::cvtColor(in_mat, *pbgrFrame, CV_RGBA2BGR);

        // save the mat to file

Moving to ffmpeg I’m now trying to convert pBufSurface->surfaceList[0].mappedAddr.addr[0] to a YUVJ420P frame using libswscale and then encode to jpeg using the mjpeg decoder from libcodec.

I’m starting out with the standard ffmpeg install… i.e. I’m not trying to use any hardware acceleration at this point.

So far I’m able to get full frames to work., which gives me some hope.

However, when I try converting a cropped frame - an object as defined by its bbox - I get the following resulting image.

Here’s the same cropped buffer encoded using the opencv code above.

My new implementation creates a frame to work with the buffer as follows

        AVFrame* pSrcFrame = av_frame_alloc();
        if (!pSrcFrame)
        {
            LOG_ERROR("Failed to allocate frame-buffers");
            return false;
        }
        pSrcFrame->format = AV_PIX_FMT_RGBA;
        pSrcFrame->width = width;
        pSrcFrame->height = height;
        pSrcFrame->pts = 1;
        pSrcFrame->linesize[0] = width*4;
        pSrcFrame->data[0] = (uint8_t*)pBufSurface->surfaceList[0].mappedAddr.addr[0];

I’m completely new to ffmpeg, so everything I’m doing is a bit of guess work. I’m hoping that someone might recognize the image pattern, and with the fact that full frames work, can point me in the right direction.

Thanks,
Robert.

[Fiona.Chen]

Please refer to the detailed description of the buffer memory alignment in NVIDIA DeepStream SDK API Reference: NvBufSurface Struct Reference | NVIDIA Docs

There is also a sample of how to get the YUV data from NvBufSurface. DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

[prominence_ai]

@Fiona.Chen I’ve reviewed the documentation, but forgive me as I’m unable to find any information on buffer-memory-alignment.

There is memType, layout, and pitch, but I’ve found nothing on alignment. Please advise.

Thanks again,
Robert.

[Fiona.Chen]

In I NvBufSurface Struct , there is NvBufSurfaceParams list, every NvBufSurfaceParams describe an aligned memory for containing the image raw data (YUV, RGB or other formats). In NvBufSurfaceParams , there is NvBufSurfacePlaneParams Struct to describe the detailed layout of every plane( Please google the plane for different color formats such as I420, YUYV, RGB planar, NV12, …). The parameters in NvBufSurfacePlaneParams describes the memory alignment of every plane. Such as start address offset, plane pitch(stride), data width, …

Image Stride - Win32 apps | Microsoft Learn

[prominence_ai]

@Fiona.Chen I’ve been able to get this working thanks.