If you want to develop real-time ray tracing for VR, which framework you would use and how?
-
Unity and unreal does not support ray-tracing for VR because dx11 attachment, until rtxgi plugin on unreal engine (I do not know how does that work)
-
APIs like OpenGL, DXR(Falcor), VKR, OptiX (OWL) might work. DXR, especially VKR is not beginner friendly (my personal opinion), and OpenGL could be a bit naive to maintain SBT. OptiX (OWL-CUDA) could be better choice for NVIDIA GPUs.
-
If rendering with an API above mentioned is okay, how to connect with the VR headset? I have HTC Vive Pro Eye, so it would be connected through SteamVR. Now both OpenVR (deprecated) and OpenXR might work.
-
What about VRWorks? Does it support Real-Time Ray-Tracing? The examples show DX11, DX12, VK11, VK12, OpenGL but no OptiX example. That could be easier to start with.
I am inexperienced in both ray tracing, and VR. I would like to ask, as an experience programmer, which framework you would chose and why? I am personally giving preference to OptiX-OWL-CUDA, but after practicing some sample codes, I cannot find way how to do it for the VR. Please help.
Perhaps Unreal Engine’s Lumen feature would be a good place for real-time ray tracing. They did some cool things with real-time global illumination, and their engine has VR support.
However, UE5 still in the beta. So you might have to work around some bugs here and there. For example, this crash is currently causing issues on the RTX 3060 Ti:
Fatal error: [File:D:\build++UE5\Sync\Engine\Source\Runtime\D3D12RHI\Private\D3D12Util.cpp] [Line: 859] pResource->Map(Subresource, pReadRange, reinterpret_cast<void**>(&pData)) failed at D:\build++UE5\Sync\Engine\Source\Runtime\D3D12RHI\Private\D3D12RHIPrivate.h:1387 with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG
UnrealEditor_D3D12RHI
UnrealEditor_D3D12RHI
UnrealEditor_D3D12RHI
UnrealEditor_Renderer
UnrealEditor_Renderer
UnrealEditor_Renderer
UnrealEditor_Renderer
UnrealEditor_Renderer
UnrealEditor_Renderer
UnrealEditor_Renderer
UnrealEditor_Renderer
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_RenderCore
UnrealEditor_RenderCore
UnrealEditor_Core
UnrealEditor_Core
kernel32
ntdll
Still, being cutting-edge is not a reason to avoid it. OptiX, like anything, may also have its share of bugs. You just have to weight the upsides and downsides of each ray tracing API.
1 Like
When I started, I actually spent a lot of time on this topic. Unreal Engine could be one of the best options, because you then have no need to think about scene and material handling and just focus on the rendering part.
However, to my best knowledge there are some serious downsides of unreal engine, as well as unity3D. Please correct me if I am wrong:
- Unreal engine ray/path tracing is based on DXR and still yet the VR does not support the ray tracing. The VR is still based on DX11 pipeline. I do not know whether there is any update on this issue recently, but I think as UE4 is commercial game engine, and as VR raytracing is far behind in commercial implementation, they won’t add this feature soon enough.
- Besides VR ray tracing, I am not sure whether UE supports multi-GPU ray tracing or not! But OptiX allow you to work on multi-GPU rendering.
1 Like
Very good points @_Bi2022 , VR and Raytracing in professional game engines still is a bit of a construction site.
The dependency on DX12 for some features like RTXGI does limit the possibilities of VR in that context.
That said, NVIDIA engineering is aware that there is an interest in the community for advanced Raytracing functionality and VR and they are looking into how that could be made to work. Unfortunately there is no official timeline available.
I think Richard also mentioned VR at some part of the Q&A in his latest Webinar about RTX features in Unreal 5. Well worth watching.
1 Like
Thanks @MarkusHoHo for joining in this discussion. As the rapid development of RTX, the next big thing could be ray-tracing and VR/AR together for the best immersion experience.
From NVDIA’s site, I know there is Falcor, that can be extended to VR, also OptiX, now I know how to extended it to VR, but still, a lot of additional work to just setup the test bed. On the other hand, if there is some base work like VRWorks, working on VR and raytracing would be even easier. It is just my personal opinion. :)
