Video frame mapping using gstreamer and EGL / OpenGL ES 3.0

Richardds · May 13, 2021, 9:56pm

Hello,
I have a simple EGL / OpenGL ES 3.0 program that renders texture on a full-screen quad.
I need to map a video from a file to that quad (in realtime) and apply my transformations / shaders.
I do not want to use the glshader plugin - I would like to have complete control of what is being rendered, so the only solution is to convert the video decoder’s output to OpenGL ES texture.

I have read many discussions that it is possible to get video texture without copying the actual pixels from GStreamer and uploading them to my program using glTexImage2D.

I spent several days searching for examples and documentation. However, I did not get even close.

I figured out that I will have to use nveglglessink or nv3dsink. However, I could not find any example in C++ or other languages at all.

I have the following pipeline written in C++:

filesrc location=<h265_video_path> ! qtdemux ! h265parse ! omxh265dec ! nvvidconv ! 'video/x-raw(memory:NVMM),format=(string)NV12' ! nvegltransform ! 'video/x-raw(memory:EGLImage),format=(string)RGBA' ! nveglglessink -e

When I run this pipeline (without initializing EGL / OpenGL ES 3.0 contexts), I get the same result as running the pipeline using gst-launch-1.0.

I run two threads - one for the GStreamer loop and the other for my quad rendering.

How do I tell GStreamer to use my EGL / OpenGL ES 3.0 context, so the texture is available in my program?
I tried the following in pipeline bus callback:

case GST_MESSAGE_NEED_CONTEXT:
{
    const gchar* contextType;
    gst_message_parse_context_type(message, &contextType);

    if (std::string(contextType) == "gst.egl.EGLDisplay") {
            // ???
    }
    break;
}

… but I cannot find any function accepting EGLContext pointer.

How do I get that texture ID? How to solve cross-threading?

If it is not possible, how do I get pixel buffer with current video frame?

Any tips/links are appreciated.

Thanks in advance!

DaneLLL · May 14, 2021, 2:57am

Hi,
You may check jetson_multimedia_api. The reference sample for video decoding is

/usr/src/jetson_multimedia_api/samples/00_video_decode/

May refer to NvEglRenderer:

/usr/src/jetson_multimedia_api/samples/common/classes/NvEglRenderer.cpp

Document: Jetson Linux API Reference: Main Page | NVIDIA Docs

Richardds · May 14, 2021, 12:58pm

Unfortunatelly, I am getting SIGSEGV on the 00_video_decode sample.
I have downloaded Tegra Multimedia API from https://developer.nvidia.com/embedded/dlc/Tegra_Multimedia_Nano
The video I am using was converted from h264 to h265 using ffmpeg:
ffmpeg -y -loglevel info -i .\bbb_sunflower_2160p_60fps_normal.mp4 -map 0:0 -c:v libx265 -map_metadata -1 -map_chapters -1 sample.h265

Here is output with --dbg-level 3:

root@jetson:/opt/nvidia/tegra_multimedia_api/samples/00_video_decode# DISPLAY=:0 ./video_decode H265 --dbg-level 3 /opt/jetson_test/Resources/Videos/sample.h265
Set governor to performance before enabling profiler
Creating decoder in blocking mode
Opening in BLOCKING MODE
Opening in BLOCKING MODE
[DEBUG] (NvV4l2Element.cpp:70) <dec0> :Opened, fd = 11
[DEBUG] (NvV4l2Element.cpp:228) <dec0> :Successfully subscribed to event 5
NvMMLiteOpen : Block : BlockType = 279
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 279
[DEBUG] (NvV4l2ElementPlane.cpp:376) <dec0> :Output Plane:VIDIOC_S_FMT at capture plane successful
Setting frame input mode to 1
[DEBUG] (NvV4l2Element.cpp:190) <dec0> :Set controls
[DEBUG] (NvVideoDecoder.cpp:207) <dec0> :Setting decoder frame input mode to 1: success
[DEBUG] (NvV4l2ElementPlane.cpp:522) <dec0> :Output Plane:Reqbuf returned 2 buffers
[DEBUG] (NvV4l2ElementPlane.cpp:627) <dec0> :Output Plane:QueryBuf for 0th buffer successful
[DEBUG] (NvV4l2ElementPlane.cpp:665) <dec0> :Output Plane:ExportBuf successful for Buffer 0, Plane 0, fd = 1047
[DEBUG] (NvBuffer.cpp:175) <Buffer> Mapped buffer 0, plane 0 to
[DEBUG] (NvV4l2ElementPlane.cpp:627) <dec0> :Output Plane:QueryBuf for 1th buffer successful
[DEBUG] (NvV4l2ElementPlane.cpp:665) <dec0> :Output Plane:ExportBuf successful for Buffer 1, Plane 0, fd = 1048
[DEBUG] (NvBuffer.cpp:175) <Buffer> Mapped buffer 1, plane 0 to
[DEBUG] (NvV4l2ElementPlane.cpp:556) <dec0> :Output Plane:STREAMON successful
Starting decoder capture loop thread
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Output Plane:Qed buffer 0
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Output Plane:Qed buffer 1
[DEBUG] (NvV4l2Element.cpp:110) <dec0> :DQed event 5
[DEBUG] (NvV4l2ElementPlane.cpp:358) <dec0> :Capture Plane:Getting format: success
[DEBUG] (NvV4l2ElementPlane.cpp:394) <dec0> :Capture Plane:Getting crop params: success
Video Resolution: 3840x2160
[INFO] (NvEglRenderer.cpp:110) <renderer0> Setting Screen width 3840 height 2160
[DEBUG] (NvEglRenderer.cpp:112) <renderer0> :Display opened successfully 547876772064
[DEBUG] (NvEglRenderer.cpp:201) <renderer0> :Egl Got display 547742554672
[DEBUG] (NvEglRenderer.cpp:218) <renderer0> :Got numconfigs as 1
[DEBUG] (NvEglRenderer.cpp:669) <renderer0> :Shaders intialized
[DEBUG] (NvEglRenderer.cpp:255) <renderer0> :Starting render thread
[DEBUG] (NvEglRenderer.cpp:150) <renderer0> :Renderer started successfully
[DEBUG] (NvV4l2ElementPlane.cpp:535) <dec0> :Capture Plane:Already in STREAMOFF
[DEBUG] (NvV4l2ElementPlane.cpp:916) <dec0> :Capture Plane:Stopped DQ Thread
[DEBUG] (NvV4l2ElementPlane.cpp:522) <dec0> :Capture Plane:Reqbuf returned 0 buffers
[DEBUG] (NvV4l2ElementPlane.cpp:458) <dec0> :Capture Plane:deinit successful
[DEBUG] (NvV4l2ElementPlane.cpp:376) <dec0> :Capture Plane:VIDIOC_S_FMT at capture plane successful
[DEBUG] (NvV4l2Element.cpp:171) <dec0> :Got value 10 for control 9963815
[DEBUG] (NvVideoDecoder.cpp:215) <dec0> :Getting decoder minimum capture plane buffers (10): success
Decoder colorspace ITU-R BT.601 with standard range luma (16-235)
[DEBUG] (NvV4l2ElementPlane.cpp:522) <dec0> :Capture Plane:Reqbuf returned 11 buffers
[DEBUG] (NvV4l2ElementPlane.cpp:556) <dec0> :Capture Plane:STREAMON successful
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 0
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 1
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 2
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 3
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 4
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 5
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 6
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 7
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 8
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 9
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 10
Query and set capture successful
Segmentation fault

Sometimes last lines of the output also contain:

Query and set capture successful
[WARN] (NvV4l2Element.cpp:119) <dec0> :Error while DQing event: Resource temporarily unavailable
[WARN] (NvV4l2ElementPlane.cpp:167) <dec0> :Capture Plane:Error while DQing buffer: Resource temporarily unavailable
Segmentation fault

Output from valgrind:

root@jetson:/opt/nvidia/tegra_multimedia_api/samples/00_video_decode# DISPLAY=:0 valgrind --leak-check=full ./video_decode H265 --dbg-level 3 /opt/jetson_test/Resources/Videos/sample.h265
==10524== Memcheck, a memory error detector
==10524== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==10524== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==10524== Command: ./video_decode H265 --dbg-level 3 /opt/jetson_test/Resources/Videos/sample.h265
==10524==
Set governor to performance before enabling profiler
Creating decoder in blocking mode
Opening in BLOCKING MODE
==10524== Mismatched free() / delete / delete []
==10524==    at 0x4846258: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==  Address 0x54e6850 is 0 bytes inside a block of size 40 alloc'd
==10524==    at 0x4844BFC: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==
==10524== Warning: noted but unhandled ioctl 0x4e04 with no size/direction hints.
==10524==    This could cause spurious value errors to appear.
==10524==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
Opening in BLOCKING MODE
==10524== Source and destination overlap in memcpy(0x48ba180, 0x48ba180, 208)
==10524==    at 0x484A080: __GI_memcpy (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==
[DEBUG] (NvV4l2Element.cpp:70) <dec0> :Opened, fd = 11
[DEBUG] (NvV4l2Element.cpp:228) <dec0> :Successfully subscribed to event 5
NvMMLiteOpen : Block : BlockType = 279
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 279
[DEBUG] (NvV4l2ElementPlane.cpp:376) <dec0> :Output Plane:VIDIOC_S_FMT at capture plane successful
Setting frame input mode to 1
[DEBUG] (NvV4l2Element.cpp:190) <dec0> :Set controls
[DEBUG] (NvVideoDecoder.cpp:207) <dec0> :Setting decoder frame input mode to 1: success
[DEBUG] (NvV4l2ElementPlane.cpp:522) <dec0> :Output Plane:Reqbuf returned 2 buffers
[DEBUG] (NvV4l2ElementPlane.cpp:627) <dec0> :Output Plane:QueryBuf for 0th buffer successful
[DEBUG] (NvV4l2ElementPlane.cpp:665) <dec0> :Output Plane:ExportBuf successful for Buffer 0, Plane 0, fd = 1054
[DEBUG] (NvBuffer.cpp:175) <Buffer> Mapped buffer 0, plane 0 to
[DEBUG] (NvV4l2ElementPlane.cpp:627) <dec0> :Output Plane:QueryBuf for 1th buffer successful
[DEBUG] (NvV4l2ElementPlane.cpp:665) <dec0> :Output Plane:ExportBuf successful for Buffer 1, Plane 0, fd = 1055
[DEBUG] (NvBuffer.cpp:175) <Buffer> Mapped buffer 1, plane 0 to
[DEBUG] (NvV4l2ElementPlane.cpp:556) <dec0> :Output Plane:STREAMON successful
Starting decoder capture loop thread
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Output Plane:Qed buffer 0
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Output Plane:Qed buffer 1
[DEBUG] (NvV4l2Element.cpp:110) <dec0> :DQed event 5
[DEBUG] (NvV4l2ElementPlane.cpp:358) <dec0> :Capture Plane:Getting format: success
[DEBUG] (NvV4l2ElementPlane.cpp:394) <dec0> :Capture Plane:Getting crop params: success
Video Resolution: 3840x2160
[INFO] (NvEglRenderer.cpp:110) <renderer0> Setting Screen width 3840 height 2160
[DEBUG] (NvEglRenderer.cpp:112) <renderer0> :Display opened successfully 90369920
[DEBUG] (NvEglRenderer.cpp:201) <renderer0> :Egl Got display 90435904
[DEBUG] (NvEglRenderer.cpp:218) <renderer0> :Got numconfigs as 1
[DEBUG] (NvEglRenderer.cpp:669) <renderer0> :Shaders intialized
[DEBUG] (NvEglRenderer.cpp:255) <renderer0> :Starting render thread
[DEBUG] (NvEglRenderer.cpp:150) <renderer0> :Renderer started successfully
[DEBUG] (NvV4l2ElementPlane.cpp:535) <dec0> :Capture Plane:Already in STREAMOFF
[DEBUG] (NvV4l2ElementPlane.cpp:916) <dec0> :Capture Plane:Stopped DQ Thread
[DEBUG] (NvV4l2ElementPlane.cpp:522) <dec0> :Capture Plane:Reqbuf returned 0 buffers
[DEBUG] (NvV4l2ElementPlane.cpp:458) <dec0> :Capture Plane:deinit successful
[DEBUG] (NvV4l2ElementPlane.cpp:376) <dec0> :Capture Plane:VIDIOC_S_FMT at capture plane successful
[DEBUG] (NvV4l2Element.cpp:171) <dec0> :Got value 10 for control 9963815
[DEBUG] (NvVideoDecoder.cpp:215) <dec0> :Getting decoder minimum capture plane buffers (10): success
Decoder colorspace ITU-R BT.601 with standard range luma (16-235)
[DEBUG] (NvV4l2ElementPlane.cpp:522) <dec0> :Capture Plane:Reqbuf returned 11 buffers
[DEBUG] (NvV4l2ElementPlane.cpp:556) <dec0> :Capture Plane:STREAMON successful
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 0
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 1
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 2
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 3
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 4
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 5
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 6
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 7
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 8
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 9
[DEBUG] (NvV4l2ElementPlane.cpp:260) <dec0> :Capture Plane:Qed buffer 10
Query and set capture successful
[WARN] (NvV4l2Element.cpp:119) <dec0> :Error while DQing event: Resource temporarily unavailable
[WARN] (NvV4l2ElementPlane.cpp:167) <dec0> :Capture Plane:Error while DQing buffer: Resource temporarily unavailable
[WARN] (NvV4l2Element.cpp:119) <dec0> :Error while DQing event: Resource temporarily unavailable
==10524== Thread 2:
==10524== Invalid write of size 4
==10524==    at 0x7289A7C: NvxUpdateNvMMBufferToNvMediaVideoSurface (in /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so)
==10524==    by 0x71E1BFB: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so)
==10524==    by 0x72A67AF: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so)
==10524==    by 0x72EFA67: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so)
==10524==    by 0x730108F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so)
==10524==    by 0x72F1FA3: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so)
==10524==    by 0x72F2B27: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so)
==10524==    by 0x72A724F: NvMediaParserParse (in /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so)
==10524==    by 0x71DD77B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so)
==10524==    by 0x523D627: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvos.so)
==10524==    by 0x487B087: start_thread (pthread_create.c:463)
==10524==    by 0x4E624EB: thread_start (clone.S:78)
==10524==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==10524==
==10524==
==10524== Process terminating with default action of signal 11 (SIGSEGV)
==10524==  Access not within mapped region at address 0x0
==10524==    at 0x7289A7C: NvxUpdateNvMMBufferToNvMediaVideoSurface (in /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so)
==10524==    by 0x71E1BFB: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so)
==10524==    by 0x72A67AF: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so)
==10524==    by 0x72EFA67: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so)
==10524==    by 0x730108F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so)
==10524==    by 0x72F1FA3: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so)
==10524==    by 0x72F2B27: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so)
==10524==    by 0x72A724F: NvMediaParserParse (in /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so)
==10524==    by 0x71DD77B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so)
==10524==    by 0x523D627: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libnvos.so)
==10524==    by 0x487B087: start_thread (pthread_create.c:463)
==10524==    by 0x4E624EB: thread_start (clone.S:78)
==10524==  If you believe this happened as a result of a stack
==10524==  overflow in your program's main thread (unlikely but
==10524==  possible), you can try to increase the size of the
==10524==  main thread stack using the --main-stacksize= flag.
==10524==  The main thread stack size used in this run was 8388608.
==10524==
==10524== HEAP SUMMARY:
==10524==     in use at exit: 11,263,014 bytes in 2,593 blocks
==10524==   total heap usage: 3,768 allocs, 1,175 frees, 13,959,925 bytes allocated
==10524==
==10524== Thread 1 DecOutPlane:
==10524== 64 bytes in 4 blocks are possibly lost in loss record 1 of 14
==10524==    at 0x4844B3C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==
==10524== 184 bytes in 1 blocks are possibly lost in loss record 2 of 14
==10524==    at 0x4846D10: realloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==
==10524== 6,192 (2,064 direct, 4,128 indirect) bytes in 1 blocks are definitely lost in loss record 7 of 14
==10524==    at 0x4846B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==
==10524== 12,824 (10,816 direct, 2,008 indirect) bytes in 184 blocks are definitely lost in loss record 9 of 14
==10524==    at 0x484522C: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==
==10524== 20,911 bytes in 112 blocks are possibly lost in loss record 11 of 14
==10524==    at 0x4844BFC: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==
==10524== 332,506 bytes in 42 blocks are possibly lost in loss record 12 of 14
==10524==    at 0x4846B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==10524==
==10524== LEAK SUMMARY:
==10524==    definitely lost: 12,880 bytes in 185 blocks
==10524==    indirectly lost: 6,136 bytes in 25 blocks
==10524==      possibly lost: 353,665 bytes in 159 blocks
==10524==    still reachable: 10,890,333 bytes in 2,224 blocks
==10524==                       of which reachable via heuristic:
==10524==                         newarray           : 7,128 bytes in 37 blocks
==10524==                         multipleinheritance: 1,288 bytes in 2 blocks
==10524==         suppressed: 0 bytes in 0 blocks
==10524== Reachable blocks (those to which a pointer was found) are not shown.
==10524== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==10524==
==10524== For counts of detected and suppressed errors, rerun with: -v
==10524== ERROR SUMMARY: 11 errors from 9 contexts (suppressed: 0 from 0)
Segmentation fault

DaneLLL · May 15, 2021, 1:38am

Hi,
It looks like the versions do not match. Please try to install the samples through

sudo apt install nvidia-l4t-jetson-multimedia-api

Richardds · May 15, 2021, 11:54am

That is exactly what I was looking for. Thank you @DaneLLL !

Topic		Replies	Views
'Tegra_multimedia_api / samples / 00_video_decode' problem Jetson TX1	10	3468	October 18, 2021
[DEEPSTREAM] nvds_obj_enc_process Segmentation Fault on Yolo and Jetson nano DeepStream SDK camera , gstreamer , encoder , jetson , deepstream	11	61	October 9, 2024
Issue with Cropping and Saving a Specific Frame Area in .jpg Format DeepStream SDK jetson , deepstream	18	75	March 10, 2025
No EGL device available Jetson TX1	28	2738	October 18, 2021
Jetson goes curling (or, simultaneously viewing multiple IP-cams) Jetson TK1	33	12072	November 28, 2014
I want to decode H264NAL unit data to still image data Jetson AGX Orin decoder , mmapi	40	1416	December 13, 2023
EGL display error and execute: 751 Failed to create CaptureSession Jetson Nano camera , gstreamer	6	1723	July 27, 2022
OpenCV CUDA processing from gstreamer pipeline [JP4, JP5] Jetson AGX Orin opencv , gstreamer	3	1048	November 30, 2023
Program shows black screen after creating 1280x720 Texture (GL_RGB8 format, 2764800 bytes) Jetson Nano jetson-inference	4	580	January 5, 2022
Error generated while running the code after connecting the camera Jetson Xavier NX gstreamer , nvbugs	45	1259	January 2, 2024

Video frame mapping using gstreamer and EGL / OpenGL ES 3.0

Related topics