Check your coordinate spaces again! Your reflectD
calculation is using normalDir
and rayD
in object space but reflect the world space rayDirection
on it.
The reflect() function looks like this: https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/nvlink_shared/shaders/vector_math.h#L611
where i
is the incident direction pointing to the surface (positive ray.direction) and n
is the normal on the surface at which the incident vector is reflected, all in the same coordinate space. That function doesn’t care about to which side the normal points. The result is the reflected outgoing vector which has the same length as the incident vector. Means there is no need to normalize reflectD
when the incident direction (rayDirection) is a normalized vector.
Also your T and B vectors are not normalized in case you need them as ortho-normal basis.
I’m not sure about the handedness in those either. Have a look into my TBN helper struct which handle that in this function:
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/nvlink_shared/shaders/shader_common.h#L116
You could also save one of the expensive transform operations if you transform only the object space shading normal into world space and do the reflection in world space. But then the direction angle needs a dot product.