Denoiser and thin lens cameras

Hi,

How do you denoise images with shallow depth of field: how to calculate albedo and normal values for pixels which, in the final image, actually average spatially very distant hits?

Currently, I am naively using average values from a few accumulation frames collected before applying denoiser, but albedo and normal are obviously very noisy with this approach.

Is there any smarter idea?

Pre-processing on the noise before the OptiX denoiser is applied will not produce better results. It is not trained with any smoothed images. It requires the original noise.
It’s also not going to work very nicely with very few samples because there is often not enough information present inside the input image.
It’s also not guaranteed that adding albedo or albedo+normal buffers will always result in a better image.

Means simply keep providing the original noisy RGB image. The more samples that has accumulated already, the better.
Then try if adding albedo or albedo+normal inputs actually improve the result.
The albedo buffer is supposed to help most with high frequency color details in textures, for example.

I’ve been using the OptiX denoiser on motion blur data which is similar and it was not that bad.
https://github.com/nvpro-samples/optix_advanced_samples/tree/master/src/optixIntroduction/optixIntro_10
A depth of field camera for that application framework can be found in this post:
https://forums.developer.nvidia.com/t/dof-lens-shader/78089/2

OK, thanks. Sometimes the sharp part gets denoised best with the normal buffer added and then out of focus part of the same image suffers. I was thinking of a generic solution, so I don’t have to try all combinations. I have 3 ideas on my list to try:

  • two denoised images, one with raw rgb only, and one with albedo and normal buffers, blend using out-of-focus variable (somehow) sice I keep 3D hit info anyway; this would consume some more memory, double denoising time, and smells patchy;
  • train denoiser with the out-of-focus buffer at the input (less probable - I like ML work, but producing a data set of training images could be too much… - or maybe are you aware of a public dataset for such purposes?)
  • likely the first to try: collect variance of pixel values and shoot more rays in the noisy directions.