Black screen when running OpenGL application in OpenXR runtime

We’re currently using our Jetson Xavier NX to run OpenXR applications. We were able to build OpenXR (release-1.0.17) together with Monado as runtime environment and run the the hello_xr sample application with vulkan as well as OpenGL as the underlying graphics API.
However when we ran StereoKit’s (GitHub - maluoi/StereoKit: An easy-to-use mixed reality library for building HoloLens and VR applications with C# and OpenXR!) sample application (StereoKitCTest) via OpenXR/Monado all we seem to get is an empty black frame, as if no frame is passed to Monado.
When rendering directly in an OpenGL window (flatscreen mode in StereoKit) this problem doesn’t occur and the frame gets rendered as expected.
Similarly, when running StereoKitCTest via Monado on x86, this issue does not occur at all, and the correct frame is being rendered inside Monado.
As another test, we were also able to run a Godot application inside Monado without issue on the jetson NX.

We have already tried combinations of multiple JetPack, OpenXR SDK and Monado versions with the same end result.

It seems something inside the Nvidia tegra driver in conjunction with OpenGL is causing this issue. We are quite desperate and don’t know what to do next, any help would be appreciated!

Hi,
We don’t have much experience about this use-case. Would need other users to share experience.

In each release, we have demo samples in

 /usr/src/nvidia/graphics_demos 

If you suspect it is something wrong in our release, we would need your help to check if you can reproduce it by applying a patch to either sample, and share us the patch and steps. So that we can try to reproduce it and do investigation.

Update:
We may have come closer to the cause of our rendering problem. After running multiple XR sample applications (Demos) we observed that the issue is specific to the projection layer used (in OpenGL). For example simple demos like hello_xr and xr-simple-playgrounds seem to work without any issues. A more “advanced” demo like xr-simple-example is only showing the background layer and drops the complicated layers (rotating cube). However no warning or error is thrown → we could only identify this error by running the same application side by side on x86.
With that in mind we were also able to load a custom background color in StereoKit which confirms our findings.

Since the demos inside /usr/src/nvidia/graphics_demos do not represent OpenXR applications its not possible to reproduce this error with it. However with the following setup:

JetPack 4.5.1
OpenXR release1.0.17
Monado v21.0.0
xr_simple_example master

we were able to reproduce the error on several other Jetson devices (Xavier AGX, Nano).

After talking to the developers of Stereokit, OpenXR and Monado, the suspicion that it has something to do with the L4T drivers intensifies, since neither errors nor warnings are thrown at the application level.

1 Like

I have experimented with my opengl openxr example app and found that commenting out the glFramebufferTexture2D call with the GL_DEPTH_ATTACHMENT makes the cubes appear main.c · 8fb1576a42313449e8a3a966df1cc474145d7d5c · Monado / Demos / openxr-simple-example · GitLab

I have also tried creating the depth textures in GL. When attaching those depth textures, rendering works.

GLuint d[swapchain_lengths[0]];
glGenTextures(swapchain_lengths[0], d);
glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT16, w, h, 0, GL_DEPTH_COMPONENT, GL_UNSIGNED_INT, 0);

When the depth textures are created by the Monado OpenXR runtime, they are created in Vulkan, exported with VK_KHR_external_memory_fd and imported into OpenGL with GL_EXT_memory_object_fd.

I haven’t checked in detail whether there are other possible issues with the opengl depth textures Monado creates, but the suspicion so far is that the issue is attaching opengl depth textures that were imported from vulkan.

Unrelated to nvidia, but this issue gave me the idea that the exported/imported depth textures might be the issue: i965: support only color formats with memory objects (!10646) · Merge requests · Mesa / mesa

2 Likes

Here is a standalone example that demonstrates the issue, compilation instructions at the top. Excuse the quick and dirty code, not exactly my favorite pastime.

https://gist.githubusercontent.com/ChristophHaag/0920abf7a2ff7fbe1c981e38b83892ff/raw/482ad0394337f487696f0bb6eb3a6e2b86bbc8eb/nvidia_depth.c

Actual result vs expected result:

I’m also not exactly a graphics expert, so if there is anything wrong with the code, please feel free to point that out. I can only say that it works on every driver combination I know except tegra. There are no vulkan validation warnings and the only gl debug output seems like it would not cause something like this

GL CALLBACK:  type = 0x8251, severity = 0x826b, message = Buffer detailed info: Buffer object 1 (bound to GL_ARRAY_BUFFER_ARB, usage hint is GL_STATIC_DRAW) will use VIDEO memory as the source for buffer object operations.
GL CALLBACK:  type = 0x8251, severity = 0x826b, message = Buffer detailed info: Buffer object 1 (bound to GL_VERTEX_ATTRIB_ARRAY_BUFFER_BINDING_ARB (0), and GL_ARRAY_BUFFER_ARB, usage hint is GL_DYNAMIC_DRAW) will use VIDEO memory as the source for buffer object operations.

edit: To clarify, the depth textures aren’t actually used by the application. They are only attached to the framebuffer to enable the gl depth test (it seems this is a requirement when rendering to a texture). The only interaction of the application with the depth texture is clearing it before rendering: glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);. In the OpenXR case it will also be given to the OpenXR runtime where it can be read to help with reprojection, but in this example code, it’s completely unused other than for the gl depth test.
See also the #if 0 block in the code to toggle between exporting textures from vulkan and simply creating them in gl (which makes it render fine on tegra).

2 Likes

Hi @timongentzsch ,

Have you tried @christoph.haag’s suggestion here? Is it working for you?

Hi @WayneWWW,
Yes I was able to reproduce the standalone example from Christoph Haag with the same results. I have also been able to test it with other devices of the jetson family (Xavier AGX, Xavier NX): same results.
The path Christoph Haag identified seems to point pretty much to the root of the problem (exporting depth textures from Vulkan in OpenGL), since one is able circumvent this bug by simple opengl rendering (if=0block).
Could the problem already be tracked down at Nvidia or could a solution be worked out in this regard?

No, we have no solution yet. This generally takes time.

If possible, please directly share any simple code and steps to reproduce your issue.

A simple example that reproduces the error has been supplied by christoph Haag. Unfortunately, it is not possible to further simplify the issue. Is the Nvidia team having problems running the provided code?

Hi,
We will try the sample in
Black screen when running OpenGL application in OpenXR runtime - #6 by christoph.haag

Thanks for the sharing.

Hi timongentzsch,

We tried the sample on Xavier-NX, but got below errors:

$ VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ./a.out
dbus[26115]: arguments to dbus_message_new_method_call() were incorrect, assertion "path != NULL" failed in file ../../../dbus/dbus-message.c line 1362.
This is normally a bug in some application using the D-Bus library.
  D-Bus not built with -rdynamic so unable to print a backtrace
Aborted (core dumped)

Any idea about this errors?
Thanks!

Hi @carolyuu,
Yes we ran into similar issues. The problem seems to be a bug in the d-bus library. A solution/ temporary fix to this would be to supress any d-bus warnings by export DBUS_FATAL_WARNINGS=0 . Hope this helps!

Never seen that error. SDL2 DBus error when calling SDL_Init(SDL_INIT_VIDEO); Looks like this is a bug between old dbus and old SDL and nvidia updating the tegra distributions would fix this.

FWIW I tested this with a relatively fresh install Jetson Nano 2GB, updated with just apt-get upgrade and it worked fine.

Hi timongentzsch,

We fixed this issue.
Please try attached “nvidia_depth_fixed.c” one.
nvidia_depth_fixed.c (37.2 KB)

1 Like

Just a few notes about the fix:

  • Please do not assume that when importing/exporting an image, it should be dedicated. Please check with a memory query, as this is done in the fix.
  • Make sure to use the correct image usage on image creation.
  • If were to need to make any layout transition or image upload/download, the most efficient way would be to create the Vulkan queue once, then record all of the operations for all the images and then either wait for completion or use a fence(s) with the GL context before rendering. Creating a queue, submitting work, and waiting for each image resource is expensive (slow and inefficient). In this particular case, no layout transition was necessary.
1 Like

Many thanks for figuring this out. Narrowing it down, this is the minimal change to make the cubes appear.

--- nvidia_depth2.c     2021-08-05 19:33:14.664191162 +0200
+++ nvidia_depth.c      2021-08-05 19:32:26.919155732 +0200
@@ -538,7 +538,7 @@
        GLuint gl_texture;
        GLuint gl_mem_object;

-       GLint gl_dedicated_mem = GL_TRUE;
+       GLint gl_dedicated_mem = GL_FALSE;

        glCreateMemoryObjectsEXT (1, &gl_mem_object);
        gl_check_error ("glCreateMemoryObjectsEXT");

I will go ahead and fix it properly in Monado (including dedicated/non dedicated allocation on the vulkan side).

The provided example code was really just a quick and dirty copy&paste job to reproduce the issue but thanks for looking at it anyway.

Do you think this is something the GL driver could warn about in glEnable(GL_DEBUG_OUTPUT);?

1 Like

The above statement is contrary to what I have said above - not to make assumptions about dedicated memory imports/exports. Specifically, for each image, determine the memory requirements, including the requirement for the image to be dedicated on export/import. So, for the exported memory:

	// Create an image
	res = vkCreateImage (vk->device, &image_info, NULL, &vk_image);
	CHECK_VK_RESULT(res, "vkCreateImage", false);

	VkImageMemoryRequirementsInfo2 imageMemReqInfo2 = { VK_STRUCTURE_TYPE_IMAGE_MEMORY_REQUIREMENTS_INFO_2 };
	imageMemReqInfo2.image = vk_image;
	
	VkMemoryRequirements2 memReq2 = { VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2 };

	VkMemoryDedicatedRequirements memDedicatedReq = { VK_STRUCTURE_TYPE_MEMORY_DEDICATED_REQUIREMENTS };
	memReq2.pNext = &memDedicatedReq;

	// Get the extended memory requirements. Determine if the image's memory must be dedicated or not.
	vkGetImageMemoryRequirements2(vk->device, &imageMemReqInfo2, &memReq2);

	VkBool32  dedicatedAllocation =  (memDedicatedReq.requiresDedicatedAllocation != VK_FALSE) ||
					(memDedicatedReq.prefersDedicatedAllocation != VK_FALSE);

	// Use the VkMemoryDedicatedAllocateInfo only if the dedicatedAllocation indicates that dedicated memeory is required 
	VkMemoryDedicatedAllocateInfoKHR dedicated_memory_info =
	{
		.sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO_KHR,
		.pNext = NULL,
		.image = vk_image,
		.buffer = VK_NULL_HANDLE
	};


	VkMemoryAllocateInfo memory_info =
	{
		.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO,
		.pNext = dedicatedAllocation ? &dedicated_memory_info : NULL,
		.allocationSize = memReq2.memoryRequirements.size,
		/* filled in later */
		.memoryTypeIndex = 0
	};

Then allocate and export the memory.

Create the memory object for the GL texture with the correct attributes:

	//  Determine if the memory for the texture must be dedicated or not, based on the 
        //  "dedicatedAllocation " flag above
	GLint gl_dedicated_mem = dedicatedAllocation ? GL_TRUE : GL_FALSE;

	GLuint gl_mem_object;
	glCreateMemoryObjectsEXT (1, &gl_mem_object);

	glMemoryObjectParameterivEXT (gl_mem_object,
								  GL_DEDICATED_MEMORY_OBJECT_EXT,
							   &gl_dedicated_mem);

Please, pay specific attention to the “dedicatedAllocation” flag. If this protocol from the Vulkan specification is to be strictly followed, then there would be no issues with image/texture exports/imports.

It would be very difficult to create a validation layer to track memory exports/imports because those can be done cross-process, and the API would not have a full context on the intent and the type of the memory being imported.

Yea I just wanted to point out the main thing it boiled down to - telling the GL driver to use the wrong memory model for the depth texture will cause the depth test not to work. In Monado I’m fixing it properly as you suggested.

It was a question because I have no idea how the drivers are implemented in the background. If the fd has some metadata associated with it that can be queried somehow, there could be some validation attempted. If it’s really “just” an fd, then I see why you can’t really do anything to validate.

Unfortunately, our Vulkan (kernel) driver does not export any metadata along with the device memory/video surfaces. It relies on the API (middleware) to do the right thing.

It turns out this is not the only issue we suffer form on tegra.

This merge requests fixes our dedicated allocation issue and with it openxr-simple-example renders properly: comp: Only use dedicated allocation when supported/preferred (!867) · Merge requests · Monado / Monado · GitLab

The second issue was that hello_xr -G OpenGL and Stereokit only render black inside the Monado compositor. This too happens only on Tegra, not on desktop nvidia.

hello_xr -G OpenGL chooses the GL_RGBA16F OpenGL format. Which will be backed by the Monado compositor allocating a Vulkan texture with VK_FORMAT_R16G16B16A16_SFLOAT and importing it into OpenGL in a similar manner.

Switching out hello_xr’s selected color format with GL_SRGB8_ALPHA8_EXT fixes its rendering in the Monado compositor.

diff --git a/src/tests/hello_xr/openxr_program.cpp b/src/tests/hello_xr/openxr_program.cpp
index 294dff7..6456ced 100644
--- a/src/tests/hello_xr/openxr_program.cpp
+++ b/src/tests/hello_xr/openxr_program.cpp
@@ -638,7 +638,7 @@ struct OpenXrProgram : IOpenXrProgram {
             CHECK_XRCMD(xrEnumerateSwapchainFormats(m_session, (uint32_t)swapchainFormats.size(), &swapchainFormatCount,
                                                     swapchainFormats.data()));
             CHECK(swapchainFormatCount == swapchainFormats.size());
-            m_colorSwapchainFormat = m_graphicsPlugin->SelectColorSwapchainFormat(swapchainFormats);
+            m_colorSwapchainFormat = GL_SRGB8_ALPHA8_EXT; // m_graphicsPlugin->SelectColorSwapchainFormat(swapchainFormats);

             // Print swapchain formats and the selected one.
             {

Unfortunately hello_xr -G OpenGL only submits its rendered images to the Monado compositor and does not display them on the desktop, so it doesn’t tell us if the rendering in general is broken or if the Monado compositor just fails to display the textures.

This brings me to the reason why I post this here. If I modify my openxr-simple-example, which does blit the left eye texture to a window before submitting it to the monado compositor, to use the GL_RGBA16F format like this

diff --git a/main.c b/main.c
index 8fd440b..6ca650b 100644
--- a/main.c
+++ b/main.c
@@ -751,7 +751,7 @@ main(int argc, char** argv)

        // SRGB is usually a better choice than linear
        // a more sophisticated approach would iterate supported swapchain formats and choose from them
-       int64_t color_format = get_swapchain_format(instance, session, GL_SRGB8_ALPHA8_EXT, true);
+       int64_t color_format = get_swapchain_format(instance, session, GL_RGBA16F, true);

        // GL_DEPTH_COMPONENT16 is a good bet
        // SteamVR 1.16.4 supports GL_DEPTH_COMPONENT16, GL_DEPTH_COMPONENT24, GL_DEPTH_COMPONENT32

the Monado compositor runs into a time out

ERROR [vk_submit_cmd_buffer] vkWaitForFences: VK_TIMEOUT
ERROR [vk_has_error] vk_submit_cmd_buffer failed with VK_TIMEOUT in ../src/xrt/compositor/main/comp_layer_renderer.c:626

and the kernel logs some gpu errors

Aug 09 15:38:01 tegra kernel: nvgpu: 57000000.gpu   nvgpu_set_error_notifier_locked:137  [ERR]  error notifier set to 8 for ch 499
Aug 09 15:38:01 tegra kernel: nvgpu: 57000000.gpu     gk20a_fifo_handle_sched_error:2552 [ERR]  fifo sched ctxsw timeout error: engine=0, tsg=10, ms=3100

Full dmesg, relevant errors towards the end tegra_dmesg.txt (502.3 KB)

Unfortunately when I modify my standalone example to render to a shared VK_FORMAT_R16G16B16A16_SFLOAT/GL_RGBA16F texture, I have no issue. Rendering to such a shared texture and blitting it to the window works fine.

So I have no standalone reproduction for this issue at this time, It looks like this only happens when the Monado compositor on the Vulkan side is also using the shared texture.