Is there possible to do postprocessing in Optix?
I would like to apply some strided blur to the result of an ambient occlusion algorithm.
Do I have to do this within a cuda kernel?

Basically yes, but not in one pass.
With the single ray programming model of OptiX you have no access to the results of any of the other rays at the time of rendering. They could be produced in any order.

There are multiple ways:

  • Ray generation programs (RGP) don’t need to call rtTrace. You could just implement your blur function in another RGP entrypoint, call that in your application after you rendered the image with the main RGP, and produce the output on another buffer.
    That is basically a CUDA Kernel. (Image data remains on the GPU.)
  • You could also render to an OpenGL Pixel-BufferObject (PBO), upload that to an OpenGL texture and do your blur operation with GLSL shaders during the final display operation (texture blit).
  • Similar with HLSL under D3D if that’s your 3D API of choice.
  • Or you could do the same with CUDA interop and a separate CUDA kernel.

I’m doing post-processing with GLSL normally.