Zero-copy EGLImage texture upload?

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
1

please look at the code snippet

  GstSample *videosample =
        gst_app_sink_try_pull_sample(GST_APP_SINK(ImPipe.appsink), 10 * GST_MSECOND);
    if (videosample)
    {
        nvtxRangePush("Video Buffer Processing");
        GstBuffer *videobuf = gst_sample_get_buffer(videosample);
        GstMapInfo map;

        gst_buffer_map(videobuf, &map, GST_MAP_READ);

        glBindTexture(GL_TEXTURE_2D, videotex);
        glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 1280, 720, 0, GL_RGBA,
                     GL_UNSIGNED_BYTE, map.data);

        gst_buffer_unmap(videobuf, &map);
        gst_sample_unref(videosample);
        nvtxRangePop();
    }
    nvtxRangePop();

ImPipe.appsink is a appsink in a gstreamer pipeline.
Today it is using CPU memory

GstCaps *caps = gst_caps_new_simple("video/x-raw",
                                      "format", G_TYPE_STRING, "RGBA",
                                      "width", G_TYPE_INT, 1280,
                                      "height", G_TYPE_INT, 720,
                                      // "framerate", GST_TYPE_FRACTION, 10, 1,
                                      NULL);

Today the flow is
1 - pull frame from Gstreamer appsink
2 - Get the buffer from the sample
3 - Map the buffer to CPU pointer
4 - Upload raw pixels to a OpenGL texture
5 - Unmap and free Gstreamer object
How to make it work with NVMM and preve

5

Hi,
Would like to get more information. So you get CPU buffer in appsink, and would like to fill data into NVMM buffer. Is this understanding correct?

6

Yes, in the pipeline i use nvvidconv before the appsink to copy the from NVMM to CPU memory and then glTexImage2D also make a copy, I like to keep the pipeline in NVMM , remove the nvidconv then render with out copying.

(the goal in to render the “Gst Buffer” in an app)

Thks

7

Hi,
You can access NVMM buffer in appink through NvBufSurface APIs. The buffer can be mapped to cv::Mat or cv::cuda::GpuMat. Please refer to the code:
RTSP camera access frame issue - #19 by DaneLLL
How to create opencv gpumat from nvstream? - #18 by DaneLLL

8

Thank you for the reference. I’m know that NvBuSurface APIs is 'your" way to manipulate the “surfaces”, but he has a lot of interaction, mostly undocumented and quite unpredictable with DS/Gstreamer plugin

This is not answering the question, it seems reasonable to have a well define API or validated code snippet as i provided.

I’m looking for the equivalent of :

GLuint tex;
glGenTextures(1, &tex);
glBindTexture(GL_TEXTURE_2D, tex);
glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, eglImage);

from other dicussions:

4. glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, eglImage);

  • Normally, you would upload CPU memory into the texture using glTexImage2D(...) — which copies data from CPU → GPU.
  • But here, eglImage is already GPU memory (from NVMM via DMA-BUF/EGL).
  • glEGLImageTargetTexture2DOES tells OpenGL:

“Use this EGLImage as the texture’s storage. Don’t copy anything. Just point to it.”

  • Now, OpenGL reads from the exact same memory GStreamer wrote into — zero-copy, zero latency.

Thank you in advance

9

Hi,

I review the links you provide but they are somehow “off topic”. They show how to access the NVMM data to be able to post process the frame data in your example to do some blurring or so.

If comeback to the goal here:

I have a gstreamer pipeline and I want to mix the video frame with GUI, quite main stream.
I initially try to use nvcompositor and “push” the GUI frame to the sink for the nvcompositor to mix, alphablend and provide to the display sink. This is not working and the Nvidia support team trying to explain that i need to use a tiler , which is NOT the same thing at all.

Now I’m taking the route to create a appsink and, create the GL context and now I have to copy byte by byte the video frame the even so it is physically the same memory , it consume BW and CPU, and it is really acceptable in 2025 (it was not in 2000 either)!

Thank you in advance

10

Hi,
The NvBufSurface can be mapped to EglImage. And then you can access it through EGL functions. Please search for related topics:
Search results for 'NvBufSurface EglImage ' - NVIDIA Developer Forums

11

Can you provide me with the few line of code, in replacement of the code snippet I have provided, instead of guessing game.

gLEGLImageTargetTexture2DOES is NOT supported by Nvidia YES or NO ?

if no can you please provide the equivalent.
Thks

Michel

12

Hi,
Please check
There is no "glEGLImageTargetTexture2DOES" API defined in "libGLESv2_nvidia.so" - #3 by DaneLLL

NvBufSurface can be mapped to CPU pointer, CUDA pointer, EglImage. Certain EGL functions may not be demonstrated in the samples. Please give it a try.

13

Hi, thank you, but you are still running into circle I’m NOT doing this for fun , this is a real work for me.

Are you referring to NvEglRenderer class ?

NvEglRenderer is a helper class for rendering using EGL and OpenGL ES 2.0.

The renderer requires the file descriptor (FD) of a buffer as an input. The rendering rate, in frames per second (fps), is configurable.

The renderer creates an X Window of its own. The width, height, horizontal offset, and vertical offset of the window are configurable.

This is not what I’m looking for.
If it is that simple provided me with a code snippet that substitute glELLImageTargetTexture2DOES.

14

Hi,
We are not able to give firmed answer to the query since there is no existing demonstration. Most EGL functions are supposed to work but it’s possible certain functions do not work properly. Would suggest you base on existing samples to try the EGL functions.

There are also samples in

/usr/src/nvidia/graphics_demos

Please take a look.

15

Can you please escalate to the next level.
My question is quite clear.
Regards

Michel

16

Hi,
We checked it and your proposal may not work. The tested use-case is as demonstrated in NvEglRenderer. Without X11, it may not work as expected.

17

Thank you for your work but the communication is difficult.

My use case is:

an GL window is created by the app, the pipeline app sink get the frame and display the frame inside the existing window/GL context.


 processing -> pipeline -> appsink
                             !
                             !
                             !
gui_app  <-------------- create texture                               
(create window,        (without copy of frame data)
create GL context)                                                 !
swap buffers

This impossible that it is not working this a the basic of graphic engine (ask for support)

Michel

18

Hi,
Your use-case is not supported. In gstreamer, the memory copy looks required.

19

Hi,
Here is a sample demonstrating gstreamer + NvEglRenderer:

#include <cstdlib>
#include <gst/gst.h>
#include <gst/gstinfo.h>
#include <gst/app/gstappsink.h>
#include <glib-unix.h>
#include <sstream>

#include "NvEglRenderer.h"
#include "nvbufsurface.h"

using namespace std;

#define USE(x) ((void)(x))

static GstPipeline *gst_pipeline = nullptr;
static string launch_string;
static int frame_count = 0;
static int sleep_count = 0;
static int eos = 0;
static NvEglRenderer *renderer;

static void appsink_eos(GstAppSink * appsink, gpointer user_data)
{
    printf("app sink receive eos\n");
    eos = 1;
}

static GstFlowReturn new_buffer(GstAppSink *appsink, gpointer user_data)
{
    GstSample *sample = NULL;

    g_signal_emit_by_name (appsink, "pull-sample", &sample,NULL);
    if (sample)
    {
        GstBuffer *buffer = NULL;
        GstCaps   *caps   = NULL;
        GstMapInfo map    = {0};
        int dmabuf_fd = 0;

        caps = gst_sample_get_caps (sample);
        if (!caps)
        {
            printf("could not get snapshot format\n");
        }
        gst_caps_get_structure (caps, 0);
        buffer = gst_sample_get_buffer (sample);
        gst_buffer_map (buffer, &map, GST_MAP_READ);

        
        NvBufSurface *surface = NULL;
        surface = (NvBufSurface *) map.data;
        dmabuf_fd = surface->surfaceList[0].bufferDesc;
        renderer->render(dmabuf_fd);
        frame_count++;

        gst_buffer_unmap(buffer, &map);

        gst_sample_unref (sample);
    }
    else
    {
        g_print ("could not make snapshot\n");
    }

    return GST_FLOW_OK;
}

int main(int argc, char** argv) {
    USE(argc);
    USE(argv);

    gst_init (&argc, &argv);

    GMainLoop *main_loop;
    main_loop = g_main_loop_new (NULL, FALSE);
    ostringstream launch_stream;
    int w = 1920;
    int h = 1080;
    GstAppSinkCallbacks callbacks = {appsink_eos, NULL, new_buffer};

    launch_stream
    << "filesrc location=/usr/src/jetson_multimedia_api/data/Video/sample_outdoor_car_1080p_10fps.h264 "
    << "! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! "
    << "appsink name=mysink sync=false ";

    launch_string = launch_stream.str();

    g_print("Using launch string: %s\n", launch_string.c_str());

    GError *error = nullptr;
    gst_pipeline  = (GstPipeline*) gst_parse_launch(launch_string.c_str(), &error);

    if (gst_pipeline == nullptr) {
        g_print( "Failed to parse launch: %s\n", error->message);
        return -1;
    }
    if(error) g_error_free(error);

    GstElement *appsink_ = gst_bin_get_by_name(GST_BIN(gst_pipeline), "mysink");
    gst_app_sink_set_callbacks (GST_APP_SINK(appsink_), &callbacks, NULL, NULL);

    renderer = NvEglRenderer::createEglRenderer("renderer0",
                w, h, 0, 0);
    renderer->setFPS(30);

    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_PLAYING); 

    while (eos == 0) {
        sleep(1);
        sleep_count++;
    }

    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_NULL);
    gst_object_unref(GST_OBJECT(gst_pipeline));
    g_main_loop_unref(main_loop);

    delete renderer;

    return 0;
}

(name above code to decode_nvbufsurf.cpp)

$ export MMAPI_INCLUDE=/usr/src/jetson_multimedia_api/include
$ export MMAPI_CLASS=/usr/src/jetson_multimedia_api/samples/common/classes
$ export USR_LIB=/usr/lib/aarch64-linux-gnu
$ g++ -Wall -std=c++11  decode_nvbufsurf.cpp -o decode $(pkg-config --cflags --libs gstreamer-app-1.0) -I$MMAPI_INCLUDE $MMAPI_CLASS/NvEglRenderer.o $MMAPI_CLASS/NvElement.o $MMAPI_CLASS/NvElementProfiler.o $MMAPI_CLASS/NvLogging.o $USR_LIB/libEGL.so $USR_LIB/libGLESv2.so $USR_LIB/libX11.so $USR_LIB/tegra/libnvbufsurface.so $USR_LIB/tegra/libnvbufsurftransform.so
$ export DISPLAY=:0(or 1)
$ ./decode

We get NvBufSurface in RGBA in appsink. The buffer can be rendered through NvEglRenderer. It may still be different from your use-case but for your reference.

20

OK, I already look at that sample code in the MM API , thank you.

But as you stated it is not the problem.

As per your spec:

Detailed Description

NvEglRenderer is a helper class for rendering using EGL and OpenGL ES 2.0.

The renderer requires the file descriptor (FD) of a buffer as an input. The rendering rate, in frames per second (fps), is configurable.

The renderer creates an X Window of its own. The width, height, horizontal offset, and vertical offset of the window are configurable.

The question is how to you how to you share an EGL surface in OpenGL existing context already created by a GUI ?
Can we have dev looking into that ?

Regards

Michel

21

Hi,
The use-case is not supported. Existing EglImage cannot be mapped to NvBufSurface. So memory copy is required. Supported use-case is existing NvBufSurface can be mapped to EglImage.

22

This is the other way around
I want to ceate a texture with a NVMM buffer.

Best regards

Michel C.

23

Hi,
You may share this sample:
Zero-copy EGLImage texture upload? - #17 by michelcalifornia

We can set up developer kit and check. glEGLImageTargetTexture2DOES() can work with X window as demonstrated in NvEglRenderer. Not sure whether it works the same with the gui app and we have to try on developer kit.