I was playing with RELAX prepass shader adapting it for VR and mostly it was a breeze, just some shifts in coordinates. Got it fully working and then noticed that default gDiffuseBlurRadius is 0, turned it on and ran into an issue. Investigating it I noticed that if for debugging I start with 0 diffuseIllumination and weightSum, and only accumulate sampleDiffuseIllumination for display (disabling the rest of the denoiser to see it clearly) then the result is shifted in left eye to the left, in right eye to the right, by what appears like half IPD each. I suspect GetKernelSampleCoordinates must receive some correction for VR case, but projection matrix math is not my forte… Sad part is that without blurring the prepass is a passthrough and a waste of cycles… I am just shortcircuiting it for now, but hope you guys can give me a hint for a proper fix…
Hi @tangorn and welcome back, it has been a while since your last post.
Using NRD in a VR project sounds really interesting! I forwarded your question to some of our engineers. Since it is a bit cross-domain it might take a while to get you feedback though.
Hi @tangorn ,
Like the rest of the ReLAX denoising pipeline, which uses gWorldToClip matrix to get from world space to clip space to get UVs for fetching,
the pre-blur pass in ReLAX uses gWorldToClip matrix to get the UVs.
This is not the case for ReBLUR: it uses gViewToClip (pure projection, without world-to-view transform) matrix in the pre-blur pass to get the UVs.
Do you have similar issue with ReBLUR?
Can you describe the changes you made to the shaders to adapt the denoiser to VR?
Hi. Thank you for quick reply. I am working off the nvrtx fork of unreal engine, branch 4.27, and it only has RELAX DiffuseSpecular in it, version 2.10. So I went to NVIDIAGameWorks/RayTracingDenoiser repo and checked out 2.10, then added RELAX Diffuse to the unreal code using DiffuseSpecular boilerplate as an example. No ReBLUR yet, maybe later.
The first thing i changed was since unreal calls denoiser several times per frame, and gbuffer does not change between these calls, I pulled all the gbuffer derived precomputation into a separate common prepass that is called before the actual denoiser calls. No reason to recompute the same stuff in every call.
The second change is VR specific: instead of using ipos directly in the shaders, I compute (assuming raytracing is done at a fractional resolution of the gbuffer)
const uint2 pixelPosDownsized = ipos + View.ViewRectMin.xy / gUpscaleFactor;
const uint2 pixelPosFullScene = GetPixelCoord(pixelPosDownsized, gUpscaleFactor);
const float2 InUV = (pixelPosFullScene + 0.5) * Buffer_ExtentInverse;
const float2 samplingUVRatio = View.ViewSizeAndInvSize.xy * Buffer_ExtentInverse;
Here pixelPosDownsized is the pixel coord for the denoised signal, that is shifted by scaled view rect for the right eye, pixelPosFullScene is the corresponding pixel coordinate for the gbuffer full size textures. GetPixelCoord is built in unreal method that i think does the same as your checkerboard stuff, the latter being disabled in nvrtx boilerplate code anyway. gUpscaleFactor is the ratio between gbuffer and raytracing input, so if RT is done at 50% screen, gUpscaleFactor is 2. UVs are naturally the same for both resolutions. samplingUVRatio translates the result of
const float2 uv = GetKernelSampleCoordinates(gWorldToClip, offset, centerWorldPos,
TvBv[0], TvBv[1], rotator) *
samplingUVRatio + Buffer_UVViewportMin;
so that it is not displayed across both eyes twice.
Those are pretty much all the changes I made. My current thinking is that the world position computation is the problem. As it is mentioned in the comments, camera shall be the origin, and with the current code both eyes are their own respective origins that are shifted and rotated with respect to each other, so basically both eyes have different world coords for everything. I am thinking along the lines of replacing this with a single origin at headset transform for both eyes, and hope that this will ensure convergence in all operations that require world position. Do you think it makes sense?
At the end it looked like a bug in world position computation
const float2 clipSpaceXY = (((float2)InPixelPos + float2(0.5, 0.5)) * Buffer_ViewportSizeInverse) * 2.0 - 1.0;
float4 worldPos = mul(float4(clipSpaceXY, DeviceZ, 1), View.ClipToTranslatedWorld);
computes identical world position in both eyes and hopefully will remove all the issues downstream…
[edited] Oh, and adding rotated eye offset after that…
Tangorn, did you ever release anything? I’m still working on path tracing in VR, using NRD or DLSS-RR as denoisers, and NRC/SHARC as radiance pre-caching helps a lot. For VR compatibility I generally just made everything dual-mono, so view/projections, and motion vectors’ current→prev positions are fully independently calculated, in the exact same way as they are for mono 2D apps. That seems to sort all issues but the main problem for me is just reducing the noise so that post-denoise, under heavy movement, the left and right eyes don’t get too much “stereo rivalry”.
Also, for reflections, our eyes DO in fact get stereo rivalry in real life, especially in glossy reflections (ex a bathroom tile wall reflecting a far off light, have significantly different positions and reflectances between left and right eyes, i.e. closing one or the other will show a massively different image). Anyway, for denoising the issue is just discomfort from stereo rivalry caused by denoisers having significantly different outputs. I.e. any part of the 2D image that would exhibit any “boiling” post-DLSS-RR, would have a different boiling pattern in VR, and while it may seem like a possible advantage to use our brains to “average it out”, if the left/right eyes differ TOO much it generates eye-strain. Or, as James Cameron put it recently “brain strain” (which is really what’s happening with low framerate 24p 3D choppiness during heavy action: the brain can’t make sense of what it’s capturing and has to work overtime to make heads or tails of it). This is why he increased to 270 degree shutter angle (from 24p’s default 180), and increased from 24 to 48 FPS. In VR we are taught zero motion blur is the goal but that’s zero persistence blur globally for uniform head rotations/translations, not per object which should be blurred. anyway, there’s lots of gotchas in VR and stereo rivalry caused by denoisers’ inferior performance for specular noise (which is much harder to denoise due to being concentrated, i.e. lower probability events, especially after a diffuse bounce. like DS***D paths paths. Diffuse…Specular…whatever…Diffuse)
Path tracing in VR is VR in hard mode due to these issues, and VR is already graphics in “hard mode” in some ways. 3D is way, way harder to pull off comfortably. Also remember that the projection matrices are often anti-symmetric in the left and right eyes (up/down FOV angles are the same left/right, but left and right FOV angles are usually flipped and negated, i.e. unless the lenses are perfectly round and centered to the display, the left/right FOV angles aren’t equal and oppositely signed, as they are for 2D viewports / projection matrices). And viewport jitter wouldn’t even have the same screenspace translation if the views aren’t parallel (lots of VR headsets have canted displays to sacrifice binocular overlap for greater outward horizontal field of view, so the views aren’t simply a +/- 1/2 IPD offset in X, relative to the coordinate system of the head).
If you (or anyone else), wants to discuss VR path tracing, I’ve been working at it for years now and nearly ready to ship something.