I was playing with RELAX prepass shader adapting it for VR and mostly it was a breeze, just some shifts in coordinates. Got it fully working and then noticed that default gDiffuseBlurRadius is 0, turned it on and ran into an issue. Investigating it I noticed that if for debugging I start with 0 diffuseIllumination and weightSum, and only accumulate sampleDiffuseIllumination for display (disabling the rest of the denoiser to see it clearly) then the result is shifted in left eye to the left, in right eye to the right, by what appears like half IPD each. I suspect GetKernelSampleCoordinates must receive some correction for VR case, but projection matrix math is not my forte… Sad part is that without blurring the prepass is a passthrough and a waste of cycles… I am just shortcircuiting it for now, but hope you guys can give me a hint for a proper fix…
Hi @tangorn and welcome back, it has been a while since your last post.
Using NRD in a VR project sounds really interesting! I forwarded your question to some of our engineers. Since it is a bit cross-domain it might take a while to get you feedback though.
Hi @tangorn ,
Like the rest of the ReLAX denoising pipeline, which uses gWorldToClip matrix to get from world space to clip space to get UVs for fetching,
the pre-blur pass in ReLAX uses gWorldToClip matrix to get the UVs.
This is not the case for ReBLUR: it uses gViewToClip (pure projection, without world-to-view transform) matrix in the pre-blur pass to get the UVs.
Do you have similar issue with ReBLUR?
Can you describe the changes you made to the shaders to adapt the denoiser to VR?
Hi. Thank you for quick reply. I am working off the nvrtx fork of unreal engine, branch 4.27, and it only has RELAX DiffuseSpecular in it, version 2.10. So I went to NVIDIAGameWorks/RayTracingDenoiser repo and checked out 2.10, then added RELAX Diffuse to the unreal code using DiffuseSpecular boilerplate as an example. No ReBLUR yet, maybe later.
The first thing i changed was since unreal calls denoiser several times per frame, and gbuffer does not change between these calls, I pulled all the gbuffer derived precomputation into a separate common prepass that is called before the actual denoiser calls. No reason to recompute the same stuff in every call.
The second change is VR specific: instead of using ipos directly in the shaders, I compute (assuming raytracing is done at a fractional resolution of the gbuffer)
const uint2 pixelPosDownsized = ipos + View.ViewRectMin.xy / gUpscaleFactor;
const uint2 pixelPosFullScene = GetPixelCoord(pixelPosDownsized, gUpscaleFactor);
const float2 InUV = (pixelPosFullScene + 0.5) * Buffer_ExtentInverse;
const float2 samplingUVRatio = View.ViewSizeAndInvSize.xy * Buffer_ExtentInverse;
Here pixelPosDownsized is the pixel coord for the denoised signal, that is shifted by scaled view rect for the right eye, pixelPosFullScene is the corresponding pixel coordinate for the gbuffer full size textures. GetPixelCoord is built in unreal method that i think does the same as your checkerboard stuff, the latter being disabled in nvrtx boilerplate code anyway. gUpscaleFactor is the ratio between gbuffer and raytracing input, so if RT is done at 50% screen, gUpscaleFactor is 2. UVs are naturally the same for both resolutions. samplingUVRatio translates the result of
const float2 uv = GetKernelSampleCoordinates(gWorldToClip, offset, centerWorldPos,
TvBv[0], TvBv[1], rotator) *
samplingUVRatio + Buffer_UVViewportMin;
so that it is not displayed across both eyes twice.
Those are pretty much all the changes I made. My current thinking is that the world position computation is the problem. As it is mentioned in the comments, camera shall be the origin, and with the current code both eyes are their own respective origins that are shifted and rotated with respect to each other, so basically both eyes have different world coords for everything. I am thinking along the lines of replacing this with a single origin at headset transform for both eyes, and hope that this will ensure convergence in all operations that require world position. Do you think it makes sense?
At the end it looked like a bug in world position computation
const float2 clipSpaceXY = (((float2)InPixelPos + float2(0.5, 0.5)) * Buffer_ViewportSizeInverse) * 2.0 - 1.0;
float4 worldPos = mul(float4(clipSpaceXY, DeviceZ, 1), View.ClipToTranslatedWorld);
computes identical world position in both eyes and hopefully will remove all the issues downstream…
[edited] Oh, and adding rotated eye offset after that…