CUDA mapping EGL frames from encoder returns incorrect pitch for UV planes

[kevin37]

Hi,

I am writing a video pipeline which uses CUDA NPP to process images and then encodes them with the multimedia API. The input format for the encoder is multiplanar YUV420 and I have been trying to use NPP to work with YUV images directly (rather than converting them to RGB), however the encoded frames appears to have pitch issues which I believe to be caused by an issue with the cudaEglFrame returned when mapping the buffer to CUDA.

I have created a testing pipeline which does the following;

1. dequeues an output plane from the encoder (the plane is MMAP but exported as a DMABUF)
2. calls NvEGLImageFromFd, cudaGraphicsEGLRegisterImage, and cudaGraphicsResourceGetMappedEglFrame to map the buffer as an EGL frame in CUDA
3. synchronize the CUDA device
4. uses nppiSet_8u_C1R to set each of the planes to128 using the ptr, pitch, xsize, and ysize obtained from the cudaPitchedPtr array of the cudaEglFrame
5. synchronize the CUDA device
6. unmaps the buffer with cudaGraphicsUnregisterResource and NvDestroyEGLImage
7. enqueue the buffer to the encoder

This pipeline is supposed to return a constant grey image, however there are staggered green bars in the image indicating that there is a pitch problem in the call to nppiSet_8u_C1R. The green bars are light green indicating that the Y plane was correctly set, but the UV planes had an issue with pitch.

My encoder is configured to encode 2160x1080 images, so I printed the xsize, ysize, and pitch for each plane from the cudaPitchedPtr and got 2160x1088(2304) 1080x544(1152) 1080x544(1152) (note that UV pitch of 1152 is exactly half of the Y pitch of 2304). I modified the test a little bit to draw a box in the corner and experimentally discovered that the image was correct when the UV planes had a pitch of 1280 instead of the 1152 in the cudaPitchedPtr. For the next test, I halved the dimensions to 1080x540 and got the values from cudaPitchedPtr of 1088x544(1280) 544x272(640) 544x272(640). Note that the pitch of the Y plane is 1280 and not 1152 even though the Y plane has dimensions 1088x544 which is the same as the UV planes from the previous test.

Based on this investigation, my best guess is that there is a bug where the pitch for UV is being reported as half the Y pitch rather than the actual pitch used in the allocation of the buffer. This test was run on a Jetson Nano with L4T r32.5.1.

Please let me know if there any more information required or if there is an issue with my test.

Thanks,

Kevin

[kevin37]

With further investigation, I have discovered that NvBufferGetParams does return the correct pitch.
width:[2160 1080 1080 0] height:[1088 544 544 0] pitch:[2304 1280 1280 0]

[AastaLLL]

Hi,

Would you mind attaching a sample image with us?
It will also help if you can provide a sample reproducible source with us.

Thanks.

[kevin37]

I have created a reproduction that just creates an NvBuffer, maps it to a CUDA frame, and prints the dimensions and pitches.

#include <stdio.h>

#include <cuda_runtime_api.h>
#include <cuda_egl_interop.h>
#include <nvbuf_utils.h>

int main()
{
    int width = 1080 * 2;
    int height = 1088;

    int fd;
    NvBufferCreateParams params = {
        .width = width,
        .height = height,
        .payloadType = NvBufferPayload_SurfArray,
        .layout = NvBufferLayout_Pitch,
        .colorFormat = NvBufferColorFormat_YUV420,
    };
    if (NvBufferCreateEx(&fd, &params) != 0)
    {
        fprintf(stderr, "failed to create buffer");
        return -1;
    }

    EGLImageKHR image = NvEGLImageFromFd(EGL_NO_DISPLAY, fd);

    cudaGraphicsResource_t resource;
    if (cudaGraphicsEGLRegisterImage(&resource, image, 0) != CUDA_SUCCESS)
    {
        fprintf(stderr, "failed to register egl image");
        return -1;
    }

    cudaEglFrame frame;
    if (cudaGraphicsResourceGetMappedEglFrame(&frame, resource, 0, 0) != CUDA_SUCCESS)
    {
        fprintf(stderr, "failed to register egl image");
        return -1;
    }

    NvBufferParams bufparams;
    if (NvBufferGetParams(fd, &bufparams) != 0)
    {
        fprintf(stderr, "failed to get nvbuf params");
        return -1;
    }

    printf("NvBufferParams: %dx%d(%d) %dx%d(%d) %dx%d(%d)\n",
           bufparams.width[0], bufparams.height[0], bufparams.pitch[0],
           bufparams.width[1], bufparams.height[1], bufparams.pitch[1],
           bufparams.width[2], bufparams.height[2], bufparams.pitch[2]);

    printf("cudaEglFrame: %zux%zu(%zu) %zux%zu(%zu) %zux%zu(%zu)\n",
        frame.frame.pPitch[0].xsize, frame.frame.pPitch[0].ysize, frame.frame.pPitch[0].pitch,
        frame.frame.pPitch[1].xsize, frame.frame.pPitch[1].ysize, frame.frame.pPitch[1].pitch,
        frame.frame.pPitch[2].xsize, frame.frame.pPitch[2].ysize, frame.frame.pPitch[2].pitch);

    if (bufparams.pitch[1] != frame.frame.pPitch[1].pitch || bufparams.pitch[2] != frame.frame.pPitch[2].pitch) {
        fprintf(stderr, "UV plane pitches don't match!\n");
    }

    return 0;
}

When I run this I get

NvBufferParams: 2160x1088(2304) 1080x544(1280) 1080x544(1280)
cudaEglFrame: 2160x1088(2304) 1080x544(1152) 1080x544(1152)
UV plane pitches don't match!

From my experiment from before, the pitch returned in NvBufferParams is the correct and the one from cudaEglFrame is incorrect.

[AastaLLL]

Hi,

Thanks for the useful sample.
Confirm that we can see the same issue in our environment.

We need to check with our internal team to get more information about this.
Will keep you updated.

[AastaLLL]

Hi,

Just want to confirm again.
Do you get the correct image with the NvBufferParams pitch value?

Thanks.

[kevin37]

Yes, the NvBufferParams pitch values appear to be correct. The striping pattern that I was seeing in my original question was due to me using the pitch values from cudaEglFrame and it was fixed by using the values from NvBufferParams.

[AastaLLL]

Hi,

This is a known issue and has been added to the document.

In the EGL frame, only the pitch value from the first plane can be used.
Please ignore the pitch values of other planes as they cannot be relied on.

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__EGL.html#group__CUDART__EGL_1gdd6215655a241c047d6d4939e242202a

Note:

Note that in case of multiplanar *eglFrame, pitch of only first plane (unsigned int cudaEglPlaneDesc::pitch) is to be considered by the application.

Thanks.

[kevin37]

Does this apply to the driver API as well?

[AastaLLL]

Hi,

Actually, the root cause of this issue is that the EGLFrame driver structure does not have a pitch for each plane.

https://docs.nvidia.com/cuda/cuda-driver-api/structCUeglFrame__v1.html#structCUeglFrame__v1

Thanks.

[kevin37]

Thanks for the clarification.

I know this is probably not the correct forum, but I think it would be better in these cases for the runtime API to return 0 for the pitches of the UV planes rather than dividing the pitch of the Y plane as it would make the error a lot more obvious.

[AastaLLL]

Hi,

Thanks for your suggestion.
I will share this with our internal team.

Thanks.