Will it be possible to train and use AI denoiser without setting up rendering (for example on noise from camera sensors for fast photo or video denoising)?
I am aware that training on different datasets is already possible, I am just interested in simple API to use this without having to setup a full 3D scene.
Hi Igor,
The OptiX denoiser network training needs a few inputs that will make it difficult to train using photo or video data.
The training data contains albedo and normal maps, which help the network identify surfaces that should appear smooth and noise-free. The OptiX API also allows optional albedo and normal maps to be used during inferencing.
The training data also needs noisy images (low samples per pixel) to be paired with resolved images (very high samples per pixel).
Both of those kinds of inputs will be difficult to attain in the case of photographic sensor data, but aside from that there’s no requirement to train or use the denoiser on 3d renderings specifically. It might be possible to fake the albedo & normal maps in the training, or to use empty inputs for them. I haven’t tried that, it might reduce the denoiser’s effectiveness, or it might not work at all. For the noisy/resolved image pairs, you’d need to make sure the images are aligned exactly or the network will encode more than noise properties and probably learn to warp your input image.
Hi dhart,
Hmm… what comes to mind is to:
It could be simulated for testing purposes by creating a blank 50% gray filled image in Photoshop then adding 3% Gaussian noise to it and feeding those two versions to the network.
Do you think something like that would work?
If you’re photographing solid color panels like an 18% gray calibration card, then I think it would work to feed the albedo map and the normal map the same solid color. And, yes, you could simulate the noise using Photoshop and use that to test your training.
I can’t say whether this would produce an effective photo denoiser, but if you try it we’d love to hear about it. Are you thinking of doing this with photos from multiple cameras in the training, to denoise photos in general, or just one camera? Personally, I don’t know how well this would work on a much smaller training set than the one we use, nor whether having correllated noise from the same sensor in multiple photos would help or hurt the situation. It might work… :)
The idea was a general photo/video denoiser.
It could be extended to compression artifacts as well – for example, do JPEG compression of source image at higher compression rates and then teach the network to remove blocking artifacts.
I am interested in trying to code something but I am at a loss at how to start. Any pointers in creating the simplest possible working scene setup for this would be of great help.
The easiest way to get started training your own data for the OptiX denoiser is to do the NVIDIA online course. It’s mentioned in the OptiX programming guide here: http://raytracing-docs.nvidia.com/optix/guide/index.html#post_processing_framework#deep-learning-based-denoiser
“You can also create a custom model by training the denoiser with your own set of images and use the resulting training data in OptiX, but this process is not part of OptiX itself. To learn how to generate your own training data based on your renderer’s images you can attend the course Rendered Image Denoising using Autoencoders, which is part of the NVIDIA Deep Learning Institute.”
Here is the course link:
https://courses.nvidia.com/courses/course-v1:DLI+L-GD-03+V1/about
Thanks!