I want to use a flat shader in OptiX. My question would be, how could I customize the intersection program to return as surface normal the vector that is normal to the vertices of the plane of a particular plane of the mesh?
You mean the face normal of a triangle?
That’s already available when using the standard intersect_triangle routine in OptiX. You’ll find that when looking at the OptiX intersection program examples and can search for its implementation in the OptiX SDK headers.
It’s the “n” argument returned by that function and inside the OptiX sample programs that’s placed into the attribute “geometric_normal”, which is normally used to decide if a ray hit a front- or back-face of a triangle insaide the closest hit programs.
If you do lighting with that, you will get a facetted look. You would still have a per ray view vector and would capture specular highlights over the triangle surface nonetheless. That’s similar to doing lighting in a fragment shader in a rasterizer. It’s not the same as per vertex flat shading mode in a rasterizer which only calculates a single color during lighting with the normal at the provoking vertex.
many thaks for the info!