zgjja
October 13, 2024, 10:47am
1
• Hardware Platform (GPU) x86_64 • DeepStream Version: 7.0 • TensorRT Version (From DS-7.0 docker) • NVIDIA GPU Driver Version (From DS-7.0 docker) • Issue Type( questions about nveglglessink) • Requirement details
I want to make a simple demo that integrate deepstream pipeline with ImGui but did not make it, the issue is described here:
opened 04:42AM - 11 Oct 24 UTC
backends
opengl
### Version/Branch of Dear ImGui:
docking
### Back-ends:
imgui_impl_glf… w.cpp + imgui_impl_opengl3.cpp
### Compiler, OS:
Linux + GCC11
### Full config/build information:
#### CMake
```CMAKE
cmake_minimum_required(VERSION 3.27)
project(imgui)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
find_package(PkgConfig REQUIRED)
pkg_check_modules(glib REQUIRED IMPORTED_TARGET glib-2.0)
pkg_check_modules(
gst
REQUIRED
IMPORTED_TARGET
gstreamer-1.0
gstreamer-rtsp-server-1.0
gstreamer-video-1.0
gstreamer-base-1.0
gstreamer-app-1.0
gstreamer-gl-egl-1.0)
pkg_check_modules(glfw REQUIRED IMPORTED_TARGET glfw3)
pkg_check_modules(opengl REQUIRED IMPORTED_TARGET glesv2 egl)
set(imgui_src
imgui/imgui.cpp
imgui/imgui_draw.cpp
imgui/imgui_tables.cpp
imgui/imgui_widgets.cpp
imgui/imgui_demo.cpp
imgui/backends/imgui_impl_glfw.cpp
imgui/backends/imgui_impl_opengl3.cpp)
set(src main.cpp)
add_executable(${PROJECT_NAME} ${src} ${imgui_src})
target_include_directories(
${PROJECT_NAME}
PUBLIC "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/imgui>"
"$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/imgui/backends>"
"$<INSTALL_INTERFACE:${CMAKE_BINARY_DIR}/include>"
"$<INSTALL_INTERFACE:${CMAKE_BINARY_DIR}/include/backends>")
target_link_libraries(${PROJECT_NAME} PUBLIC PkgConfig::opengl PkgConfig::glfw
PkgConfig::gst)
target_compile_definitions(
${PROJECT_NAME}
PUBLIC IMGUI_IMPL_OPENGL_ES2
PUBLIC GLFW_EXPOSE_NATIVE_X11)
```
### Details:
I want to integrate ImGui with deepstream(it uses gstreamer -- a video processing library as backbone). I will summarize some details for better understanding:
* a "sink" element from gstreamer can output a video to a window(X11 for my case)
* a sink element window can be redirected to your own window ID (use [GstVideoOverlay](https://gstreamer.freedesktop.org/documentation/video/gstvideooverlay.html?gi-language=c))
* glfw has ability to get window ID created by ImGui with `glfwGetX11Window`
From the info above, i want to use ImGui to show this video content, so my demo code would be similar to `examples/example_glfw_opengl3`
#### Result:
1. If my code do `glfwMakeContextCurrent(gl_window);`, the gstreamer will complain:
```
ERROR egladaption ext/eglgles/gstegladaptation_egl.c:210:gst_egl_adaptation_context_make_current:<sink> Couldn't bind context
```
in this situation, the ImGui elements can be rendered but my video cannot. I checked the code, here is the link: [gstegladaptation_egl.c](https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/gst-plugins-bad-1.1.2/ext/eglgles/gstegladaptation_egl.c#L213)(this may not be the exact version of my library use)
2. If I remove that line of code, the video can be rendered but ImGui elements cannot, and ImGui complains:
```
ERROR: ImGui_ImplOpenGL3_CreateDeviceObjects: failed to compile fragment shader! With GLSL: #version 100
ERROR: ImGui_ImplOpenGL3_CreateDeviceObjects: failed to link shader program! With GLSL #version 100
```
#### Analyse
* I check the code of `gstegladaptation_egl.c`, it use `eglMakeCurrent()` to use its internal context, maybe `glfwMakeContextCurrent` conflict with it (NOT sure)??
* In github, there are a few trying to do this kind of integration, but they grab the buffer from gstreamer directly and sent to ImGui rather than using `GstVideoOverlay` concept, which require more work to do.
* X11 and GTK+ can create window directly and pass the window handle, this works fine with gstreamer, but they are not easy to use like ImGui
* if I want to do something like this, how can i make it??
### Screenshots/Video:
* Case 1
Can render video, but no ImGui elements
* Case 2
Can render ImGui elements, but no video
### Minimal, Complete and Verifiable Example code:
#### Generate the test video
```C++
auto *pipeline =
gst_parse_launch("nvvideotestsrc ! nvvideoconvert ! nveglglessink name=sink", nullptr);
auto *bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
gst_bus_set_sync_handler(bus, (GstBusSyncHandler)dispatch_window_handle, (gpointer)pipeline, NULL);
gst_object_unref(bus);
gst_element_set_state(GST_ELEMENT(pipeline), GST_STATE_PLAYING);
```
#### Dispatch ImGui window
```C++
static GstBusSyncReply dispatch_window_handle(GstBus *bus, GstMessage *message, GstPipeline *pipeline) {
// ignore anything but 'prepare-window-handle' element messages
if (!gst_is_video_overlay_prepare_window_handle_message(message)) {
return GST_BUS_PASS;
}
auto *sink = GST_ELEMENT(gst_bin_get_by_name(GST_BIN(pipeline), "sink"));
auto *overlay = GST_VIDEO_OVERLAY(sink);
glfwSetErrorCallback(glfw_error_callback);
if (!glfwInit()) {
raise(SIGINT);
}
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 2);
glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 0);
glfwWindowHint(GLFW_CLIENT_API, GLFW_OPENGL_ES_API);
IMGUI_CHECKVERSION();
gl_window = glfwCreateWindow(1280, 720, "Hello, world!", nullptr, nullptr);
if (!gl_window) {
glfwTerminate();
raise(SIGINT);
}
// glfwMakeContextCurrent(gl_window); <-- This code cause my problem!
// glfwSwapInterval(1);
auto a = gst_gl_context_get_current_gl_context(GST_GL_PLATFORM_ANY);
auto x11_window = glfwGetX11Window(gl_window);
// These 2 lines are not necessary
// auto x11_display = glfwGetX11Display();
// g_object_set(G_OBJECT(sink), "display", (gpointer)x11_display, NULL);
auto imgui_ctx = ImGui::CreateContext();
auto &io = ImGui::GetIO(); (void)io;
io.ConfigFlags |= ImGuiConfigFlags_NavEnableKeyboard;
io.ConfigFlags |= ImGuiConfigFlags_NavEnableSetMousePos;
io.ConfigFlags |= ImGuiConfigFlags_ViewportsEnable;
ImGui::SetCurrentContext(imgui_ctx);
ImGui::StyleColorsDark();
ImGui_ImplGlfw_InitForOpenGL(gl_window, true);
ImGui_ImplOpenGL3_Init(nullptr);
gst_video_overlay_set_window_handle(overlay, (guintptr)x11_window);
gst_video_overlay_set_render_rectangle(overlay, 0, 0, 512, 256);
gst_video_overlay_expose(overlay);
gst_message_unref(message);
return GST_BUS_DROP;
}
```
#### ImGui render loop
while (!glfwWindowShouldClose(gl_window)) {
glfwPollEvents();
ImGui_ImplOpenGL3_NewFrame();
ImGui_ImplGlfw_NewFrame();
ImGui::NewFrame();
if (!ImGui::Begin("DearImGuiDemo", nullptr, 0)) {
ImGui::End();
return 1;
}
ImGui::Text("Hello, world!");
ImGui::End();
ImGui::Render();
glClear(GL_COLOR_BUFFER_BIT);
ImGui_ImplOpenGL3_RenderDrawData(ImGui::GetDrawData());
auto &io = ImGui::GetIO();
if (io.ConfigFlags & ImGuiConfigFlags_ViewportsEnable) {
ImGui::UpdatePlatformWindows();
ImGui::RenderPlatformWindowsDefault();
}
glfwSwapBuffers(gl_window);
}
Now the core issue i assume is that i can not make the OpenGL context used by nveglglessink and the one in ImGui to “work together” inside one window , is there any hint that i can make this? From github, i found someone use appsink to feed the data into OpenGL’s processing logic, which may add more codes, which seems sub-optimal to me.
when using
gl_window = glfwCreateWindow(1280, 720, "Hello, world!", nullptr, nullptr);
glfwMakeContextCurrent(gl_window);
to “make the context”, eglgles would complain:
0:00:00.613931114 480684 0x55555711ef80 ERROR egladaption ext/eglgles/gstegladaptation_egl.c:210:gst_egl_adaptation_context_make_current:<sink> Couldn't bind context
0:00:00.613948601 480684 0x55555711ef80 ERROR egladaption ext/eglgles/gstegladaptation.c:654:gst_egl_adaptation_init_surface:<sink> Couldn't setup EGL surface
0:00:00.613952761 480684 0x55555711ef80 ERROR nveglglessink ext/eglgles/gsteglglessink.c:3272:gst_eglglessink_configure_caps:<sink> Couldn't init EGL surface from window
0:00:00.613955733 480684 0x55555711ef80 ERROR nveglglessink ext/eglgles/gsteglglessink.c:3293:gst_eglglessink_configure_caps:<sink> Configuring caps failed
0:00:00.613971758 480684 0x7fff98000b70 ERROR nveglglessink ext/eglgles/gsteglglessink.c:3338:gst_eglglessink_setcaps:<sink> Failed to configure caps
0:00:00.614020571 480684 0x7fff98000b70 ERROR nveglglessink ext/eglgles/gsteglglessink.c:3338:gst_eglglessink_setcaps:<sink> Failed to configure caps
0:00:00.614792979 480684 0x7fff98000b70 ERROR nveglglessink ext/eglgles/gsteglglessink.c:3338:gst_eglglessink_setcaps:<sink> Failed to configure caps
I have checked the gstreamer’s OpenGL doc here: OpenGL
From this post, the nveglglessink is open source, i did not find it, where can i get it? Memory leak information is observed in case nveglglessink plugin is used
From another post, the eglglessink may be deprecated?
Are you working with Orin or other Jetson devices?
zgjja
October 14, 2024, 2:46am
4
No, from x86_64 Ubuntu22.04 docker env (DS7.0)
zgjja:
when using
gl_window = glfwCreateWindow(1280, 720, "Hello, world!", nullptr, nullptr);
glfwMakeContextCurrent(gl_window);
to “make the context”
What do you mean by this description?
nveglglessink is based on EGL.
The Jetson nveglglessink is open source while nveglglessink for dGPU is not open source. But you can refer to the Jetson nveglglessink plugin source code for the basic plugin parts. Working With Sources — NVIDIA Jetson Linux Developer Guide 1 documentation
The nveglglessink is already deprecated from Orin platforms. But it still exists in dGPU platform.
yingliu
November 15, 2024, 7:31am
6
There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks
system
November 29, 2024, 7:32am
7
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.