Tegra Multimedia API overlay plane does not overlay nvoverlaysink

I used the MMAPI to write images into all but one of the overlay planes and GStreamer to display a video stream into the remaining plane using nvoverlaysink. The nvoverlaysink output always appears over all the other planes.

My MMAPI test program is derived from tegra_multimedia_api/samples/08_video_dec_drm/video_dec_drm.cpp.

NvDrmFB ui_fb;
 uint32_t plane_count = ctx->drm_renderer->getPlaneCount();

 // Render a static JPEG image on the first plane
    ctx->drm_renderer->createDumbFB(image_w, image_h,
    for (uint32_t y = 0; y < image_h; ++y)
        for (uint32_t x = 0; x < image_w; ++x)
            uint32_t off = ui_fb.bo[0].pitch * y + x * 4;
            ui_fb.bo[0].data[off] = *p++;
            ui_fb.bo[0].data[off + 1] = *p++;
            ui_fb.bo[0].data[off + 2] = *p++;
            ui_fb.bo[0].data[off + 3] = *p++;

    for(int i=0;i<plane_count;i++)
       if(i != 2)
          ctx->drm_renderer->setPlane(i, ui_fb.fb_id,
            i * 200, i * 200, image_w, image_h,
            0, 0, image_w << 16, image_h << 16);

The above code renders the JPEG as expected with each image overlaying the other, with transparency applied.

The GStreamer pipeline is

gst-launch-1.0 udpsrc  port=7891! tsdemux name=demux demux. ! queue max-size-time=0 max-size-bytes=0 max-size-buffers=0 ! h264parse ! omxh264dec !  nvoverlaysink overlay=3 overlay-x=10 overlay-y=10 overlay-w=800 overlay-h=800

The overlay has to be either 0 or 3.
All other values fail, as expected: NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552.

The overlay configuration code is

cd /sys/class/graphics/fb0
echo 4 > blank
echo 0x0 > device/win_mask
echo 0x3f > device/win_mask
echo 0 > blank
systemctl stop lightdm

The DRM overlay planes correctly overlay each other.
The nvoverlaysinks correctly overlay each other, w/o transparency.

The combination does not work as expected.

As far as I can tell, the documentation does not describe this interaction.

You may refer to below sample.

It demonstrates on how to get NvBuffer in appsink.

Thanks, I am still working through the code.

I noted this command

export DISPLAY=:0

My goal is to run w/o X.

Also, I was looking for an explanation as why the nvoverlaysink overlays everything else.
My code displays everything correctly, except for that problem.

You could add a virtual X server (DISPLAY would still be set, but there wouldn’t be a real hardware display). Much of the CUDA-related software requires X as an interface to the GPU driver.